pom without rebuilding kernel?

pom without rebuilding kernel?

[Axel Thimm]

[-- Attachment #1: Type: text/plain, Size: 1093 bytes --]

I am thinking how nice it would be (or already is?) to build only the
required kernel modules for a given kernel, without having to rebuild
the kernel itself, like a lot of kernel drivers do (e.g. bttv, lirc,
lm_sensors etc).

Imagine building only the kernel modules, putting them in a folder
like

/lib/modules/`uname -r`/netfilter

adding the following to the top of /etc/modules.conf:

path[toplevel]=/lib/modules/`uname -r`/netfilter
path[toplevel]=/lib/modules/`uname -r`

and being happy ever after. :)

E.g. are there changes in pom patches to the core of the kernel
sources, or can pom modules be compiled out of the kernel tree?

I suppose that for a certain kernel version and above no hooks/glue to
the core non-modular networking components are required and therefore
such a plan could be followed. But I found no hint in any docu to
netfilter, not even a similar request.

If there is way to do it, or a netfilter developer thinks it is worth
the trouble, I'd rpm-package some modules for RH kernels. :)

Thanks!
-- 
Axel.Thimm@physik.fu-berlin.de

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: pom without rebuilding kernel?

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 2384 bytes --]

On Fri, Oct 10, 2003 at 03:45:25PM +0200, Axel Thimm wrote:
> I am thinking how nice it would be (or already is?) to build only the
> required kernel modules for a given kernel, without having to rebuild
> the kernel itself, like a lot of kernel drivers do (e.g. bttv, lirc,
> lm_sensors etc).

yes, you can do that in 99% of all cases.   Just use patch-o-matic like
normal and then do a 


cd your_kernel_dir
make MODULE_SUBDIRS=net/ipv4/netfilter modules

then manually copy the modules into your module directory, or better 
test them via insmod first.

> E.g. are there changes in pom patches to the core of the kernel
> sources, or can pom modules be compiled out of the kernel tree?

in very rare cases the core network stack needs to be modified.  all
other patches don't really require to rebuild the kernel.  But make sure
that you always unload _ALL_ netfilter modules (conntrack, etc) before
you load any of your newly-compiled modules.

I suppose your kernel will crash very soon if you keep your old
ip_conntrack.o while loading a new ip_conntrack_pptp.o, as an example.

> If there is way to do it, or a netfilter developer thinks it is worth
> the trouble, I'd rpm-package some modules for RH kernels. :)

no, I am strictly opposing binary distribution.  At least not as an
official part of the project.  Feel free to provide them at a different
place, but I just think it's too dangerous (and not really necessarry).

patch-o-matic is for potentially unstable or not generally useful code.
If people want to use some of it, they should understand enough and be
able to compile them on their own.
 
Bugfixes shouldn't be in patch-o-matic anyway, we've had a long
discussion about that at the netfilter workshop.  This will change when
the new (2.6 compatible) patch-o-matic will be put in place.  I'm
working on that now, doing some perl coding for the first time in the
last couple of years ;)

> Thanks!
> -- 
> Axel.Thimm@physik.fu-berlin.de



-- 
- Harald Welte <laforge@netfilter.org>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: pom without rebuilding kernel?

[Henrik Nordstrom]

On Fri, 10 Oct 2003, Axel Thimm wrote:

> I am thinking how nice it would be (or already is?) to build only the
> required kernel modules for a given kernel, without having to rebuild
> the kernel itself, like a lot of kernel drivers do (e.g. bttv, lirc,
> lm_sensors etc).

Yes, and that you can do.. just patch the kernel sources and run "make 
modules", or extract the sources and compile the modules manually.

> E.g. are there changes in pom patches to the core of the kernel
> sources, or can pom modules be compiled out of the kernel tree?

If you extract the module sources from the pom patches you can in most
cases compile the modules out of the kernel tree if you like, assuming you
have the correct kernel headers in place for the exact kernel binary you
are using (this is a rather tricky part when trying to compile modules to
a distro binary kernel)

There is not really any difference between compiling netfilter kernel 
modules and other kinds of kernel modules, only the packaging is slightly 
different (patch files to the kernel tree rather than separate source 
files, but the patch command can extract the files for you)

Regards
Henrik

Re: pom without rebuilding kernel?

[Axel Thimm]

[-- Attachment #1: Type: text/plain, Size: 3365 bytes --]

On Sat, Oct 11, 2003 at 09:20:44PM +0200, Harald Welte wrote:
> On Fri, Oct 10, 2003 at 03:45:25PM +0200, Axel Thimm wrote:
> > E.g. are there changes in pom patches to the core of the kernel
> > sources, or can pom modules be compiled out of the kernel tree?
> 
> in very rare cases the core network stack needs to be modified.  all
> other patches don't really require to rebuild the kernel.

How can I identify those patches? Most of the patching happens in
net/<protocol>/netfilter and include/linux/netfilter_<protocol>. Are
those patches safe for building out of the tree (w/o full kernel
rebuilds)?

> > If there is way to do it, or a netfilter developer thinks it is worth
> > the trouble, I'd rpm-package some modules for RH kernels. :)
> 
> no, I am strictly opposing binary distribution.  At least not as an
> official part of the project.  Feel free to provide them at a
> different place, but I just think it's too dangerous (and not really
> necessarry).

I am already maintaining some kernel module rpms as well as some
non-modularizable patches to RH kernel on atrpms. I am having requests
to add some netfilter modules (notably the TARPIT and H323 patches),
and I was considering of packaging (parts of) pom as drop in kernel
module replacements or enchancements.

> patch-o-matic is for potentially unstable or not generally useful code.
> If people want to use some of it, they should understand enough and be
> able to compile them on their own.
>  
> Bugfixes shouldn't be in patch-o-matic anyway, we've had a long
> discussion about that at the netfilter workshop.  This will change when
> the new (2.6 compatible) patch-o-matic will be put in place.  I'm
> working on that now, doing some perl coding for the first time in the
> last couple of years ;)

On Sat, Oct 11, 2003 at 10:14:54PM +0200, Henrik Nordstrom wrote:
> If you extract the module sources from the pom patches you can in most
> cases compile the modules out of the kernel tree if you like, assuming you
> have the correct kernel headers in place for the exact kernel binary you
> are using (this is a rather tricky part when trying to compile modules to
> a distro binary kernel)

Here is what I did to get a working H232 kernel module, is this sane
(seems to work for these modules at least)? Thanks.

# Create copies of subparts from the unchanged kernel tree to ./linux
for x in Makefile Documentation/Configure.help include/linux/netfilter_ipv4 include/linux/netfilter_ipv6 net/ipv4/netfilter net/ipv6/netfilter; do
  mkdir -p linux/`dirname $x`
  cp -a %{kernelsrcdir}/$x linux/`dirname $x`
done

# Patch the copy
KERNEL_DIR=linux ./runme --batch extra/h323-conntrack-nat.patch

cd linux
%patch0 -p1 -b .lockhelp
cd ..

make -C %{kernelsrcdir} SUBDIRS=`pwd`/linux/net/ipv4/netfilter dep
make -C %{kernelsrcdir} SUBDIRS=`pwd`/linux/net/ipv4/netfilter modules CONFIG_IP_NF_H323=m CONFIG_IP_NF_NAT_H323=m "CPPFLAGS=-D__KERNEL__ -I`pwd`/linux/include -I\$(HPATH)"
make -C %{kernelsrcdir} SUBDIRS=`pwd`/linux/net/ipv4/netfilter INSTALL_MOD_PATH=$RPM_BUILD_ROOT MOD_DESTDIR=../netfilter/ipv4/netfilter modules_install CONFIG_IP_NF_H323=m CONFIG_IP_NF_NAT_H323=m

Unfortunately the make dep step is writing to the orginal kernel tree,
but i couldn't find a better solution.
-- 
Axel.Thimm@physik.fu-berlin.de

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: pom without rebuilding kernel?

[Henrik Nordstrom]

On Fri, 17 Oct 2003, Axel Thimm wrote:

> How can I identify those patches? Most of the patching happens in
> net/<protocol>/netfilter and include/linux/netfilter_<protocol>. Are
> those patches safe for building out of the tree (w/o full kernel
> rebuilds)?

Any patch which modifies only files within the netfilter directories 
should be safe building netfilter out of the kernel tree.

On patches which only adds files it is safe to just build that new added 
modules out of the tree.

Patches which modifies files within the netfilter directories requires 
that all netfilter modules which depends on the modified files are 
rebuilt. In case of modified C files this is trivial, but if header files 
are modified a more careful analysis is required to determine which 
netfilter modules are impacted by this header file modification.

> I am already maintaining some kernel module rpms as well as some
> non-modularizable patches to RH kernel on atrpms. I am having requests
> to add some netfilter modules (notably the TARPIT and H323 patches),
> and I was considering of packaging (parts of) pom as drop in kernel
> module replacements or enchancements.

On patches which add nat or conntrack helpers you need to verify that the
size of the conntrack structure is not modified. If it is then basically
every single netfilter module need to be rebuild. If the size of the 
structures remains the same then it is safe to build the new helpers 
without rebuilding the rest.

> Here is what I did to get a working H232 kernel module, is this sane
> (seems to work for these modules at least)? Thanks.
> 
> # Create copies of subparts from the unchanged kernel tree to ./linux
> for x in Makefile Documentation/Configure.help include/linux/netfilter_ipv4 include/linux/netfilter_ipv6 net/ipv4/netfilter net/ipv6/netfilter; do
>   mkdir -p linux/`dirname $x`
>   cp -a %{kernelsrcdir}/$x linux/`dirname $x`
> done

You could just

  rm -rf linux
  cp -rpl %{kernelsrcdir} linux

this quickly makes a linked copy of the tree. patch (and p-o-m) works fine 
with linked source trees.

> # Patch the copy
> KERNEL_DIR=linux ./runme --batch extra/h323-conntrack-nat.patch
> 
> cd linux
> %patch0 -p1 -b .lockhelp
> cd ..
> 
> make -C %{kernelsrcdir} SUBDIRS=`pwd`/linux/net/ipv4/netfilter dep
> make -C %{kernelsrcdir} SUBDIRS=`pwd`/linux/net/ipv4/netfilter modules CONFIG_IP_NF_H323=m CONFIG_IP_NF_NAT_H323=m "CPPFLAGS=-D__KERNEL__ -I`pwd`/linux/include -I\$(HPATH)"
> make -C %{kernelsrcdir} SUBDIRS=`pwd`/linux/net/ipv4/netfilter INSTALL_MOD_PATH=$RPM_BUILD_ROOT MOD_DESTDIR=../netfilter/ipv4/netfilter modules_install CONFIG_IP_NF_H323=m CONFIG_IP_NF_NAT_H323=m

It should be noted that this last step is only safe for installing into a 
secondary directory. Do not attempt this without specifying a secondary 
MOD_DESTDIR.. (it will delete all other modules..)

> Unfortunately the make dep step is writing to the orginal kernel tree,
> but i couldn't find a better solution.

for most (if not all) netfilter patches you do not really need to run
"make dep".

Regards
Henrik