[Buildroot] building kernel modules

[Buildroot] building kernel modules

[Roman Chertov]

Hello,

I am using buildroot to build a distro for an ARM ep9xx system.
Everything works great except for one kernel module. I am trying to
debug it, but it is a pain to make the whole kernel tree to insert a
single printk.  Also, it looks like I have to do "touch
<...>/.configured" to get the kernel to detect the change the compile a
new version of the module.

Is there any documentation regarding building kernel modules by
themselves after having built the main kernel?  Something similar to how
userlevel apps get cross compiled.  I suppose that somehow I need to
access all of the flags that buildroot passes to the kernel when it
builds it.

Ideally, I just want to just compile the module, and then insmod it over
the NFS rootfs.

Many Thanks,

Roman

[Buildroot] building kernel modules

[Roman Chertov]

So I figured out how to get the external kernel module compilation
going.  The cross compilation works fine, but the linking fails as there
are two undefined symbols.  Also, there is a strange error regarding
/usr/local/src/mv-kernel/arm-none-linux-gnueabi/bin/arm-none-linux-gnueabi-gcc

However, the compilation proceeds even with that error.

[rchertov at number2 src]$ sh make.sh
make:
/usr/local/src/mv-kernel/arm-none-linux-gnueabi/bin/arm-none-linux-gnueabi-gcc:
Command not found
make: Entering directory
`/proj/tools/buildroot-2009.08/project_build_arm/uclibc/linux-2.6.29'
  LD [M]  /proj/tools/can_driver/musec_can/src/musec_can.o
  Building modules, stage 2.
  MODPOST 1 modules
WARNING: "__bad_udelay"
[/proj/tools/can_driver/musec_can/src/musec_can.ko] undefined!
WARNING: "__aeabi_uldivmod"
[/proj/tools/can_driver/musec_can/src/musec_can.ko] undefined!
  LD [M]  /proj/tools/can_driver/musec_can/src/musec_can.ko
make: Leaving directory
`/proj/tools/buildroot-2009.08/project_build_arm/uclibc/linux-2.6.29'


I believe that the linker uses libc instead of uclibc which was compiled
for ARM, and hence the linking step is failing.  I would appreciate it
if somebody could point out which linker setting I am missing.  Below, I
listed all of the variables that I redefined for the build.

ARCH=arm
CROSS_COMPILE=/<br_dir>/build_arm/staging_dir/usr/bin/arm-linux-uclibcgnueabi-
AS      =$(CROSS_COMPILE)as
LD      =$(CROSS_COMPILE)ld
CC      =$(CROSS_COMPILE)gcc
CPP     =$(CC) -E
AR      =$(CROSS_COMPILE)ar
NM      =$(CROSS_COMPILE)nm
STRIP   =$(CROSS_COMPILE)strip
OBJCOPY =$(CROSS_COMPILE)objcopy
OBJDUMP =$(CROSS_COMPILE)objdump


EXTRA_CFLAGS=-I /<br_dir>/project_build_arm/uclibc/linux-2.6.29/include\
-I<br_dir>/build_arm/staging_dir/usr/lib/gcc/arm-linux-uclibcgnueabi/4.3.3/include

LDFLAGS=-L/<br_dir>/build_arm/staging_dir/lib

...



Roman Chertov wrote:
> Hello,
> 
> I am using buildroot to build a distro for an ARM ep9xx system.
> Everything works great except for one kernel module. I am trying to
> debug it, but it is a pain to make the whole kernel tree to insert a
> single printk.  Also, it looks like I have to do "touch
> <...>/.configured" to get the kernel to detect the change the compile a
> new version of the module.
> 
> Is there any documentation regarding building kernel modules by
> themselves after having built the main kernel?  Something similar to how
> userlevel apps get cross compiled.  I suppose that somehow I need to
> access all of the flags that buildroot passes to the kernel when it
> builds it.
> 
> Ideally, I just want to just compile the module, and then insmod it over
> the NFS rootfs.
> 
> Many Thanks,
> 
> Roman
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
> 

[Buildroot] building kernel modules

[Daniel Mack]

On Sun, Nov 08, 2009 at 09:28:38PM -0800, Roman Chertov wrote:
> So I figured out how to get the external kernel module compilation
> going.  The cross compilation works fine, but the linking fails as there
> are two undefined symbols.  Also, there is a strange error regarding
> /usr/local/src/mv-kernel/arm-none-linux-gnueabi/bin/arm-none-linux-gnueabi-gcc
> 
> However, the compilation proceeds even with that error.
> 
> [rchertov at number2 src]$ sh make.sh
> make:
> /usr/local/src/mv-kernel/arm-none-linux-gnueabi/bin/arm-none-linux-gnueabi-gcc:
> Command not found
> make: Entering directory
> `/proj/tools/buildroot-2009.08/project_build_arm/uclibc/linux-2.6.29'
>   LD [M]  /proj/tools/can_driver/musec_can/src/musec_can.o
>   Building modules, stage 2.
>   MODPOST 1 modules
> WARNING: "__bad_udelay"

This error is due to an illegal usage of udelay() in the module code.
Calling this function at all is frowned upon, and calling it with too
big values fail[*] the kernel compilation for very good reasons. You
should rather use schedule_timeout() or timer functions to give CPU time
to do something useful while waiting for so long. But, without knowing
the sources, it's impossible to give any further hints.

([*] the way they implemented that depending on compile-time constraints
     is very nice, btw. Dig through the headers if your curious ;))

> [/proj/tools/can_driver/musec_can/src/musec_can.ko] undefined!
> WARNING: "__aeabi_uldivmod"
> [/proj/tools/can_driver/musec_can/src/musec_can.ko] undefined!
>   LD [M]  /proj/tools/can_driver/musec_can/src/musec_can.ko
> make: Leaving directory
> `/proj/tools/buildroot-2009.08/project_build_arm/uclibc/linux-2.6.29'
> 
> 
> I believe that the linker uses libc instead of uclibc which was compiled
> for ARM, and hence the linking step is failing.  I would appreciate it
> if somebody could point out which linker setting I am missing.  Below, I
> listed all of the variables that I redefined for the build.
> 
> ARCH=arm
> CROSS_COMPILE=/<br_dir>/build_arm/staging_dir/usr/bin/arm-linux-uclibcgnueabi-

These two look sane.

> AS      =$(CROSS_COMPILE)as
> LD      =$(CROSS_COMPILE)ld
> CC      =$(CROSS_COMPILE)gcc
> CPP     =$(CC) -E
> AR      =$(CROSS_COMPILE)ar
> NM      =$(CROSS_COMPILE)nm
> STRIP   =$(CROSS_COMPILE)strip
> OBJCOPY =$(CROSS_COMPILE)objcopy
> OBJDUMP =$(CROSS_COMPILE)objdump
> 
> 
> EXTRA_CFLAGS=-I /<br_dir>/project_build_arm/uclibc/linux-2.6.29/include\
> -I<br_dir>/build_arm/staging_dir/usr/lib/gcc/arm-linux-uclibcgnueabi/4.3.3/include
> 
> LDFLAGS=-L/<br_dir>/build_arm/staging_dir/lib
> 
> ...

But don't need all the rest. All you need to provide is ARCH=arm and
CROSS_COMPILE=..., the rest is done automatically. The kernel's build
system is very self-contained and the sources come with all kind of
headers and libraries. No need to point it to any external resources.

Following that should make the 2nd error go away.

Daniel

[Buildroot] building kernel modules

[Roman Chertov]

> This error is due to an illegal usage of udelay() in the module code.
> Calling this function at all is frowned upon, and calling it with too
> big values fail[*] the kernel compilation for very good reasons. You
> should rather use schedule_timeout() or timer functions to give CPU time
> to do something useful while waiting for so long. But, without knowing
> the sources, it's impossible to give any further hints.
> 

I have removed udelay calls in the code. (I am porting an old 2.4 driver
to 2.6.29) and the _bad_udelay error went away.

> 
> But don't need all the rest. All you need to provide is ARCH=arm and
> CROSS_COMPILE=..., the rest is done automatically. The kernel's build
> system is very self-contained and the sources come with all kind of
> headers and libraries. No need to point it to any external resources.
> 
> Following that should make the 2nd error go away.

I have changed my makefile to this

obj-m  := musec_can.o
musec_can-y := main.o hms30c7202_can.o c_can.o \
        irq.o open.o close.o read.o write.o ioctl.o

The makefile is then invoked via a shell script:

make ARCH=arm \
CROSS_COMPILE=/<br_dir>/build_arm/staging_dir/usr/bin/arm-linux-uclibcgnueabi-\
-C /proj/tools/buildroot-2009.08/project_build_arm/uclibc/linux-2.6.29 \
M=`pwd` modules


I have also configured buildroot to use gcc 4.4.1.  I also rebuilt
uclibc and the kernel with new compiler.  When I make the module, I get
the following during the linking phase.

make: Entering directory
`/proj/tools/buildroot-2009.08/project_build_arm/uclibc/linux-2.6.29'
  Building modules, stage 2.
  MODPOST 1 modules
WARNING: "__aeabi_uldivmod"
[/proj/tools/can_driver/musec_can/src/musec_can.ko] undefined!
make: Leaving directory
`/proj/tools/buildroot-2009.08/project_build_arm/uclibc/linux-2.6.29'

Naturally, this module is not loadable as there an undefined symbol in
the module.  insmod reports the following:

insmod: can't insert './musec_can.ko': unknown symbol in module, or
unknown parameter

I would appreciate some help in resolving this.

Thanks,

Roman

> 
> Daniel
> 

[Buildroot] building kernel modules

[Daniel Mack]

On Mon, Nov 09, 2009 at 02:38:00PM -0800, Roman Chertov wrote:
> > But don't need all the rest. All you need to provide is ARCH=arm and
> > CROSS_COMPILE=..., the rest is done automatically. The kernel's build
> > system is very self-contained and the sources come with all kind of
> > headers and libraries. No need to point it to any external resources.
> > 
> > Following that should make the 2nd error go away.
> 
> I have changed my makefile to this
> 
> obj-m  := musec_can.o
> musec_can-y := main.o hms30c7202_can.o c_can.o \
>         irq.o open.o close.o read.o write.o ioctl.o
> 
> The makefile is then invoked via a shell script:
> 
> make ARCH=arm \
> CROSS_COMPILE=/<br_dir>/build_arm/staging_dir/usr/bin/arm-linux-uclibcgnueabi-\
> -C /proj/tools/buildroot-2009.08/project_build_arm/uclibc/linux-2.6.29 \
> M=`pwd` modules
> 
> 
> I have also configured buildroot to use gcc 4.4.1.  I also rebuilt
> uclibc and the kernel with new compiler.  When I make the module, I get
> the following during the linking phase.
> 
> make: Entering directory
> `/proj/tools/buildroot-2009.08/project_build_arm/uclibc/linux-2.6.29'
>   Building modules, stage 2.
>   MODPOST 1 modules
> WARNING: "__aeabi_uldivmod"

Hmm. If you only get this error for your own module (and not the kernel
itself), you should have a look at the assembly output and see what gcc
does there and why. It shouldn't be using the __aeabi_uldivmod function
which is part of libgcc which the kernel does not seem to link on your
platform. According to the postings below, these errors are likely to be
caused by false gcc optimisations.

You should get around that by altering your sources, but again, without
the code, this is all speculating.

Those links could also help:

  http://www.spinics.net/lists/arm-kernel/msg48776.html
  http://lists.arm.linux.org.uk/lurker/message/20080227.081641.1580db5d.en.html

In case you consider bringing your driver mainline, you can also post it
to LKML and get some feedback there.

After all, what you're seeing is an effect that has to do with your
code, the kernel and gcc. And not with buildroot :)

Daniel

[Buildroot] building kernel modules

[Roman Chertov]

> 
> Hmm. If you only get this error for your own module (and not the kernel
> itself), you should have a look at the assembly output and see what gcc
> does there and why. It shouldn't be using the __aeabi_uldivmod function
> which is part of libgcc which the kernel does not seem to link on your
> platform. According to the postings below, these errors are likely to be
> caused by false gcc optimisations.
> 
> You should get around that by altering your sources, but again, without
> the code, this is all speculating.

Turns out it was a division of a long that was causing that.  After I
changed the long to a uint32_t then the problem went away.  Now, I have
to figure out why I cannot insert the module.

musec_can: unknown relocation: 40
insmod: can't insert
'/proj/tools/can_driver/musec_can/src/musec_can.ko': invalid module format

Modules built during the kernel built work fine.  I guess I have to
figure out what linker setting I am missing.

> 
> Those links could also help:
> 
>   http://www.spinics.net/lists/arm-kernel/msg48776.html
>   http://lists.arm.linux.org.uk/lurker/message/20080227.081641.1580db5d.en.html
> 
> In case you consider bringing your driver mainline, you can also post it
> to LKML and get some feedback there.

That would be down the road once I actually test the driver and iron out
any problems that might occur.

> 
> After all, what you're seeing is an effect that has to do with your
> code, the kernel and gcc. And not with buildroot :)

True.  I was just hoping that somebody would give me a starting point as
  I new to cross compiling :)  Thanks for the help.

Roman

> 
> Daniel
> 

[Buildroot] building kernel modules

[Daniel Mack]

On Tue, Nov 10, 2009 at 09:18:33AM -0800, Roman Chertov wrote:
> Turns out it was a division of a long that was causing that.  After I
> changed the long to a uint32_t then the problem went away.  Now, I have
> to figure out why I cannot insert the module.
> 
> musec_can: unknown relocation: 40
> insmod: can't insert
> '/proj/tools/can_driver/musec_can/src/musec_can.ko': invalid module format
> 
> Modules built during the kernel built work fine.  I guess I have to
> figure out what linker setting I am missing.

See dmesg for more of the reason. There's a comprehensive guideline I
can point you to, just in case that differs from what you did:

  http://lwn.net/Articles/21823/

> > After all, what you're seeing is an effect that has to do with your
> > code, the kernel and gcc. And not with buildroot :)
> 
> True.  I was just hoping that somebody would give me a starting point as
>   I new to cross compiling :)  Thanks for the help.

No problem :)

Daniel

[Buildroot] building kernel modules

[Roman Chertov]

Daniel Mack wrote:
> On Tue, Nov 10, 2009 at 09:18:33AM -0800, Roman Chertov wrote:
>> Turns out it was a division of a long that was causing that.  After I
>> changed the long to a uint32_t then the problem went away.  Now, I have
>> to figure out why I cannot insert the module.
>>
>> musec_can: unknown relocation: 40
>> insmod: can't insert
>> '/proj/tools/can_driver/musec_can/src/musec_can.ko': invalid module format

This was due to a kernel bug
http://lkml.indiana.edu/hypermail/linux/kernel/0903.3/02813.html
The patch provided in the link fixed the problem for me.


>>
>> Modules built during the kernel built work fine.  I guess I have to
>> figure out what linker setting I am missing.
> 
> See dmesg for more of the reason. There's a comprehensive guideline I
> can point you to, just in case that differs from what you did:
> 
>   http://lwn.net/Articles/21823/
> 
>>> After all, what you're seeing is an effect that has to do with your
>>> code, the kernel and gcc. And not with buildroot :)
>> True.  I was just hoping that somebody would give me a starting point as
>>   I new to cross compiling :)  Thanks for the help.
> 
> No problem :)
> 
> Daniel
> 

[Buildroot] building kernel modules

[Brian Beattie]

Does anybody here know how to build kernel modules for buildroot outside
the kernel tree?  Any documents or tutorials?  Everything I've found so
far covers self-hosted builds and does not seem to work for
cross-compiling.

I'm working inside the kernel tree, but that is a little clumsy.

Baring that, how about forcing a kernel recompile, "make linux26" does
not seem to do it.