[PATCH] MODULE_FIRMWARE for binary firmware(s)

[PATCH] MODULE_FIRMWARE for binary firmware(s)

[James Bottomley]

From: Jon Masters <jcm@redhat.com>

Right now, various kernel modules are being migrated over to use
request_firmware in order to pull in binary firmware blobs from userland
when the module is loaded. This makes sense.

However, there is right now little mechanism in place to automatically
determine which binary firmware blobs must be included with a kernel in
order to satisfy the prerequisites of these drivers. This affects
vendors, but also regular users to a certain extent too.

The attached patch introduces MODULE_FIRMWARE as a mechanism for
advertising that a particular firmware file is to be loaded - it will
then show up via modinfo and could be used e.g. when packaging a kernel.

Signed-off-by: Jon Masters <jcm@redhat.com>

Comments added in line with all the other MODULE_ tag

Signed-off-by: James Bottomley <James.Bottomley@SteelEye.com>
Index: BUILD-2.6/include/linux/module.h
===================================================================
--- BUILD-2.6.orig/include/linux/module.h	2006-08-28 13:16:22.000000000 -0500
+++ BUILD-2.6/include/linux/module.h	2006-08-28 13:17:46.000000000 -0500
@@ -156,6 +156,11 @@ extern struct module __this_module;
 */
 #define MODULE_VERSION(_version) MODULE_INFO(version, _version)
 
+/* Optional firmware file (or files) needed by the module
+ * format is simply firmware file name.  Multiple firmware
+ * files require multiple MODULE_FIRMWARE() specifiers */
+#define MODULE_FIRMWARE(_firmware) MODULE_INFO(firmware, _firmware)
+
 /* Given an address, look for it in the exception tables */
 const struct exception_table_entry *search_exception_tables(unsigned long add);
 

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[James Bottomley]

This is a reference implementation with the debian mkinitrd-tools
package.  It shows how to identify the firmware files necessary for
drivers in the initrd and also includes a primitive system for loading
them.

I've tested this with the aic94xx driver using the new MODULE_FIRMWARE()
tag.  Initramfs should be much easier because it already includes most
of the boot time loading; all it has to do is the piece identifying the
firmware for the selected modules.

James

---
Index: initrd-tools-0.1.84.1/mkinitrd
===================================================================
--- initrd-tools-0.1.84.1.orig/mkinitrd	2006-08-28 13:37:30.000000000 -0500
+++ initrd-tools-0.1.84.1/mkinitrd	2006-08-28 16:33:28.000000000 -0500
@@ -950,6 +950,7 @@ add_modules_dep() {
 		return
 	elif ! [ $oldstyle ]; then
 		add_modules_dep_2_5 $VERSION
+		add_firmware $VERSION
 		return
 	fi
 
@@ -1016,6 +1017,25 @@ add_modules_dep_2_5() {
 	fi
 }
 
+add_firmware() {
+	ver=$1
+	set -- $FSTYPES
+	unset IFS
+
+	cat modules.? |
+		while read junk mod junk; do
+			modpath=$(modprobe --set-version "$ver" --list $mod)
+			if [ -z "$modpath" ]; then
+				continue;
+			fi
+			p=$(modinfo -F firmware "$modpath" |sed 's/^/\/lib\/firmware\//')
+			if [ -n "$p" ]; then
+				echo $p
+				echo /usr/sbin/firmware_loader
+			fi
+		done
+}
+
 add_command() {
 	if [ -h initrd/"$1" ]; then
 		return
Index: initrd-tools-0.1.84.1/firmware_loader
===================================================================
--- /dev/null	1970-01-01 00:00:00.000000000 +0000
+++ initrd-tools-0.1.84.1/firmware_loader	2006-08-28 16:56:18.000000000 -0500
@@ -0,0 +1,29 @@
+#!/bin/sh -e
+#
+# firmware loader agent
+#
+FIRMWARE_DIRS="/lib/firmware"
+
+if [ "$SUBSYSTEM" != "firmware" ]; then
+    exit 0;
+fi
+
+if [ ! -e /sys/$DEVPATH/loading ]; then
+    echo "/sys/$DEVPATH/ does not exist"
+    exit 1
+fi
+
+for DIR in $FIRMWARE_DIRS; do
+    [ -e "$DIR/$FIRMWARE" ] || continue
+    echo 1 > /sys/$DEVPATH/loading
+    cat "$DIR/$FIRMWARE" > /sys/$DEVPATH/data
+    echo 0 > /sys/$DEVPATH/loading
+    exit 0
+done
+
+# the firmware was not found
+echo -1 > /sys/$DEVPATH/loading
+
+echo "Cannot find the $FIRMWARE firmware"
+exit 1
+
Index: initrd-tools-0.1.84.1/debian/rules
===================================================================
--- initrd-tools-0.1.84.1.orig/debian/rules	2006-08-28 16:07:52.000000000 -0500
+++ initrd-tools-0.1.84.1/debian/rules	2006-08-28 16:08:56.000000000 -0500
@@ -35,7 +35,7 @@ install: 
 	install -o root -g root -m 644 \
 		echo init linuxrc debian/initrd-tools/usr/share/initrd-tools/
 	install -o root -g root -m 755 \
-		mkinitrd debian/initrd-tools/usr/sbin/
+		mkinitrd firmware_loader debian/initrd-tools/usr/sbin/
 	install -o root -g root -m 644 \
 		mkinitrd.conf modules debian/initrd-tools/etc/mkinitrd/
 ifeq ($(DEB_HOST_ARCH),powerpc)
Index: initrd-tools-0.1.84.1/linuxrc
===================================================================
--- initrd-tools-0.1.84.1.orig/linuxrc	2006-08-28 16:30:30.000000000 -0500
+++ initrd-tools-0.1.84.1/linuxrc	2006-08-28 16:40:45.000000000 -0500
@@ -10,3 +10,7 @@ echo 256 > proc/sys/kernel/real-root-dev
 mount -nt tmpfs tmpfs bin ||
 	mount -nt ramfs ramfs bin
 echo $root > bin/root
+if [ -x /usr/sbin/firmware_loader ]; then
+	echo /usr/sbin/firmware_loader > /proc/sys/kernel/hotplug
+fi
+
Index: initrd-tools-0.1.84.1/init
===================================================================
--- initrd-tools-0.1.84.1.orig/init	2006-08-28 16:54:52.000000000 -0500
+++ initrd-tools-0.1.84.1/init	2006-08-28 16:55:01.000000000 -0500
@@ -366,6 +366,7 @@ get_cmdline
 [ -c /dev/.devfsd ] && DEVFS=yes
 
 mount -nt devfs devfs devfs
+mount -nt sysfs sysfs sys
 if [ $IDE_CORE != none ] && [ -n "$ide_options" ]; then
 	echo modprobe -k $IDE_CORE "options=\"$ide_options\""
 	modprobe -k $IDE_CORE options="$ide_options"

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Sven Luther]

On Mon, Aug 28, 2006 at 05:11:42PM -0500, James Bottomley wrote:
> This is a reference implementation with the debian mkinitrd-tools
> package.  It shows how to identify the firmware files necessary for
> drivers in the initrd and also includes a primitive system for loading
> them.
> 
> I've tested this with the aic94xx driver using the new MODULE_FIRMWARE()
> tag.  Initramfs should be much easier because it already includes most
> of the boot time loading; all it has to do is the piece identifying the
> firmware for the selected modules.

Notice that mkinitrd-tools is dead, and will probably be removed from etch.

mkinitramfs-tools and yaird are the two currently used tools.

Friendly,

Sven Luther

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[James Bottomley]

On Tue, 2006-08-29 at 01:04 +0200, Sven Luther wrote:
> Notice that mkinitrd-tools is dead, and will probably be removed from etch.
> 
> mkinitramfs-tools and yaird are the two currently used tools.

Yes ... I'm aware of that.  That's why this is a reference
implementation.  initramfs should be easier ... I just don't have any
initramfs systems at the moment, so I did what I had and could verify.

James

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Oleg Verych]

Sven Luther wrote:
> On Mon, Aug 28, 2006 at 05:11:42PM -0500, James Bottomley wrote:
 >
>>I've tested this with the aic94xx driver using the new MODULE_FIRMWARE()
>>tag.  Initramfs should be much easier because it already includes most
>>of the boot time loading; all it has to do is the piece identifying the
>>firmware for the selected modules.
...
> Notice that mkinitrd-tools is dead, and will probably be removed from etch.
> 
request_firmware() is dead also.
YMMV, but three years, and there are still big chunks of binary in kernel.
And please don't add new useless info _in_ it.

Nobody cares.
While this implementation exists, it wasn't well designed and hard to use.
As with in-kernel bootsplash and i18n, everything maybe done in userspace, only
with little help from the kernel:
<http://permalink.gmane.org/gmane.linux.kernel/435955>.

Thanks.

-- 
-o--=O`C  /. .\   (+)
  #oo'L O      o    |
<___=E M    ^--    |  (you're barking up the wrong tree)

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[James Bottomley]

On Tue, 2006-08-29 at 02:35 +0200, Oleg Verych wrote:
> request_firmware() is dead also.
> YMMV, but three years, and there are still big chunks of binary in kernel.
> And please don't add new useless info _in_ it.

I er don't think so.

We (as in the Kernel) are forcing drivers on to this path.  You should
have noticed it already with the qla2xxx.  The aic94xx is the first
driver that's likely found as the boot driver for a system, that's why I
get to propose the reference implementation.

James

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Oleg Verych]

James Bottomley wrote:
> On Tue, 2006-08-29 at 02:35 +0200, Oleg Verych wrote:
> 
>>request_firmware() is dead also.
>>YMMV, but three years, and there are still big chunks of binary in kernel.
>>And please don't add new useless info _in_ it.
> 
> 
> I er don't think so.
> 
Hell, what can be as easy as this:
,-
|modprobe $drv
|(dd </lib/firmware/$drv.bin>/dev/blobs && echo OK) || echo Error
`-
where /dev/blobs is similar to /dev/port or even /dev/null char device.
if synchronization is needed, add `echo $drv >/dev/blobs`, remove modprobe,

> 
> James
> 
?

-- 
--
-o--=O`C  /. .\   (+)
  #oo'L O      o    |
<___=E M    ^--    |  (you're barking up the wrong tree)

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Greg KH]

On Tue, Aug 29, 2006 at 05:14:30AM +0200, Oleg Verych wrote:
> On Mon, Aug 28, 2006 at 06:51:03PM -0700, Greg KH wrote:
> > On Tue, Aug 29, 2006 at 04:15:49AM +0200, Oleg Verych wrote:
> > > >>request_firmware() is dead also.
> > > >>YMMV, but three years, and there are still big chunks of binary in kernel.
> > > >>And please don't add new useless info _in_ it.
> > > Hell, what can be as easy as this:
> > > ,-
> > > |modprobe $drv
> > > |(dd </lib/firmware/$drv.bin>/dev/blobs && echo OK) || echo Error
> > > `-
> > > where /dev/blobs is similar to /dev/port or even /dev/null char device.
> > > if synchronization is needed, add `echo $drv >/dev/blobs`, remove modprobe,
> > 
> > I don't see such code in the kernel at this time.  So it's not a
> > solution, sorry.
> > 
> I know. return -ENOPATCH

Yes, and that's the only way to make changes in the kernel, sorry.

> I'm nether a CS nor software engineer, just wondering why simple thing isn't
> simple _in_ the Kernel. I'm reading list "just for fun (C)" and any good word
> about this (IMHO) unix-way design *may* lead professional programmers to do
> tiny worthy things (think about kevent discussion).
> If it's (i'm) stupid, please, say so (in the way Nicholas Miell did ;).

I don't think it's workable, and goes against the current way the kernel
does things.  But please, feel free to prove me wrong with a patch
otherwise.  I don't want to debate it otherwise.

I think the current way we handle firmware works quite well, especially
given the wide range of different devices that it works for (everything
from BIOS upgrades to different wireless driver stages).

thanks,

greg k-h

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Oleg Verych]

Greg KH wrote:
> On Tue, Aug 29, 2006 at 05:14:30AM +0200, Oleg Verych wrote:
> 
>>On Mon, Aug 28, 2006 at 06:51:03PM -0700, Greg KH wrote:
>>
>>>On Tue, Aug 29, 2006 at 04:15:49AM +0200, Oleg Verych wrote:
>>>
>>>>>>request_firmware() is dead also.
>>>>>>YMMV, but three years, and there are still big chunks of binary in kernel.
>>>>>>And please don't add new useless info _in_ it.
>>>>
>>>>Hell, what can be as easy as this:
>>>>,-
>>>>|modprobe $drv
>>>>|(dd </lib/firmware/$drv.bin>/dev/blobs && echo OK) || echo Error
>>>>`-
>>>>where /dev/blobs is similar to /dev/port or even /dev/null char device.
>>>>if synchronization is needed, add `echo $drv >/dev/blobs`, remove modprobe,
>>>
...
> 
>>I'm nether a CS nor software engineer, just wondering why simple thing isn't
>>simple _in_ the Kernel. I'm reading list "just for fun (C)" and any good word
>>about this (IMHO) unix-way design *may* lead professional programmers to do
>>tiny worthy things (think about kevent discussion).
>>If it's (i'm) stupid, please, say so (in the way Nicholas Miell did ;).
> 
> I don't think it's workable, and goes against the current way the kernel
> does things.  But please, feel free to prove me wrong with a patch
> otherwise.  I don't want to debate it otherwise.
> 
Thanks, and OK, this is my last reply on this.

> I think the current way we handle firmware works quite well, especially
> given the wide range of different devices that it works for (everything
> from BIOS upgrades to different wireless driver stages).
> 
Oh, come on, even skilled developers (not particular kernel's) having
difficulties with current hotplug-sysfs-modprobe thing;
in this case one couldn't easily figure out problems and way to solve them
<http://permalink.gmane.org/gmane.comp.hardware.texas-instruments.msp430.gcc.user/5444>

Goodbye.

--
-o--=O`C  /. .\   (+)
  #oo'L O      o    |
<___=E M    ^--    |  (you're barking up the wrong tree)

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[David Lang]

On Mon, 28 Aug 2006, Greg KH wrote:

> I think the current way we handle firmware works quite well, especially
> given the wide range of different devices that it works for (everything
> from BIOS upgrades to different wireless driver stages).

the current system works for many people yes, but not everyone.

I'm still waiting to find a way to get the iw2200 working without having to use 
modules.

there was a post a month or two ago from someone who had indicated they found a 
way to re-initialize it after the system came up, but after someone asked for 
details of how to do it there was no response.

David Lang

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Greg KH]

On Tue, Aug 29, 2006 at 08:46:45AM -0700, David Lang wrote:
> On Mon, 28 Aug 2006, Greg KH wrote:
> 
> >I think the current way we handle firmware works quite well, especially
> >given the wide range of different devices that it works for (everything
> >from BIOS upgrades to different wireless driver stages).
> 
> the current system works for many people yes, but not everyone.
> 
> I'm still waiting to find a way to get the iw2200 working without having to 
> use modules.

Sounds like a bug you need to pester the iw2200 developers about then.
I don't think it has much to do with the firmware subsystem though :)

thanks,

greg k-h

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Michael Buesch]

On Tuesday 29 August 2006 20:32, Greg KH wrote:
> On Tue, Aug 29, 2006 at 08:46:45AM -0700, David Lang wrote:
> > On Mon, 28 Aug 2006, Greg KH wrote:
> > 
> > >I think the current way we handle firmware works quite well, especially
> > >given the wide range of different devices that it works for (everything
> > >from BIOS upgrades to different wireless driver stages).
> > 
> > the current system works for many people yes, but not everyone.
> > 
> > I'm still waiting to find a way to get the iw2200 working without having to 
> > use modules.
> 
> Sounds like a bug you need to pester the iw2200 developers about then.
> I don't think it has much to do with the firmware subsystem though :)

Well, yes and no.
The ipw needs the firmware on insmod time (in contrast to bcm43xx
for example, which needs it on ifconfig up time).
So ipw needs to call request_firmware at insmod time. In case of
built-in, that is when the initcall happens. No userland is available
and request_firmware can not call the userspace helpers to upload
the firmware to sysfs.
Well, not really easy to find a sane solution for this. :)

-- 
Greetings Michael.

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Olaf Hering]

On Tue, Aug 29, Michael Buesch wrote:

> On Tuesday 29 August 2006 20:32, Greg KH wrote:
> > On Tue, Aug 29, 2006 at 08:46:45AM -0700, David Lang wrote:
> > > On Mon, 28 Aug 2006, Greg KH wrote:
> > > 
> > > >I think the current way we handle firmware works quite well, especially
> > > >given the wide range of different devices that it works for (everything
> > > >from BIOS upgrades to different wireless driver stages).
> > > 
> > > the current system works for many people yes, but not everyone.
> > > 
> > > I'm still waiting to find a way to get the iw2200 working without having to 
> > > use modules.
> > 
> > Sounds like a bug you need to pester the iw2200 developers about then.
> > I don't think it has much to do with the firmware subsystem though :)
> 
> Well, yes and no.
> The ipw needs the firmware on insmod time (in contrast to bcm43xx
> for example, which needs it on ifconfig up time).
> So ipw needs to call request_firmware at insmod time. In case of
> built-in, that is when the initcall happens. No userland is available
> and request_firmware can not call the userspace helpers to upload
> the firmware to sysfs.

I dont use nor do I have access ipw hardware, but:
If it is an initcall, the initramfs should be usable at that time.
A creative CONFIG_INITRAMFS_SOURCE= will help, add the firmware and a
small helper that does the cat(1).

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[David Lang]

On Tue, 29 Aug 2006, Olaf Hering wrote:

> On Tue, Aug 29, Michael Buesch wrote:
>
>> On Tuesday 29 August 2006 20:32, Greg KH wrote:
>>> On Tue, Aug 29, 2006 at 08:46:45AM -0700, David Lang wrote:
>>>> On Mon, 28 Aug 2006, Greg KH wrote:
>>>>
>>>>> I think the current way we handle firmware works quite well, especially
>>>>> given the wide range of different devices that it works for (everything
>>>>> from BIOS upgrades to different wireless driver stages).
>>>>
>>>> the current system works for many people yes, but not everyone.
>>>>
>>>> I'm still waiting to find a way to get the iw2200 working without having to
>>>> use modules.
>>>
>>> Sounds like a bug you need to pester the iw2200 developers about then.
>>> I don't think it has much to do with the firmware subsystem though :)
>>
>> Well, yes and no.
>> The ipw needs the firmware on insmod time (in contrast to bcm43xx
>> for example, which needs it on ifconfig up time).
>> So ipw needs to call request_firmware at insmod time. In case of
>> built-in, that is when the initcall happens. No userland is available
>> and request_firmware can not call the userspace helpers to upload
>> the firmware to sysfs.
>
> I dont use nor do I have access ipw hardware, but:
> If it is an initcall, the initramfs should be usable at that time.
> A creative CONFIG_INITRAMFS_SOURCE= will help, add the firmware and a
> small helper that does the cat(1).
>

you are assuming that

1. modules are enabled and ipw2200 is compiled as a module

2. initrd or initramfs are in use

besides, several kernel versions ago this used to work. the current requirement 
is a regression as far as the user is concerned.

David Lang

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Greg KH]

On Tue, Aug 29, 2006 at 01:42:56PM -0700, David Lang wrote:
> 
> besides, several kernel versions ago this used to work. the current 
> requirement is a regression as far as the user is concerned.

Then go bug the driver authors please :)

thanks,

greg k-h

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Olaf Hering]

On Tue, Aug 29, David Lang wrote:

> you are assuming that
> 
> 1. modules are enabled and ipw2200 is compiled as a module

No, why?

> 2. initrd or initramfs are in use

initramfs is always in use.

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[David Lang]

On Wed, 30 Aug 2006, Olaf Hering wrote:

>> you are assuming that
>>
>> 1. modules are enabled and ipw2200 is compiled as a module
>
> No, why?

becouse if the ipw isn't compiled as a module then it's initialized (without 
firmware) before the initramfs or initrd is run. if it could be initialized 
later without being a module then it could be initialized as part of the normal 
system

>> 2. initrd or initramfs are in use
>
> initramfs is always in use.

not on my machines.

what does it use for the initramfs?

I don't enable the options in the kernel for initrd, and I don't give it any 
source for an initramfs.

David Lang

P.S. there was a suggestion yesterday in this thread that I haven't tested yet. 
I plan to test that tonight. if it works then the card can be reinitialized 
after the system boots, still no initrd or initramfs needed.

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Sven Luther]

On Wed, Aug 30, 2006 at 10:52:02AM -0700, David Lang wrote:
> On Wed, 30 Aug 2006, Olaf Hering wrote:
> 
> >>you are assuming that
> >>
> >>1. modules are enabled and ipw2200 is compiled as a module
> >
> >No, why?
> 
> becouse if the ipw isn't compiled as a module then it's initialized 
> (without firmware) before the initramfs or initrd is run. if it could be 
> initialized later without being a module then it could be initialized as 
> part of the normal system

Euh, ...

I wonder why you need to initialize the ipw2200 module so early ? It was my
understanding that the initramfs thingy was run very early in the process, and
i had even thought of moving fbdev modules into it.

Do you really need to bring up ipw2200 so early ? It is some kind of wireless
device, right ? 

As for initramfs, you can just cat it behind the kernel, and it should work
just fine, or at least this is how it was supposed to work.

Friendly,

Sven Luther

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[David Lang]

On Wed, 30 Aug 2006, Sven Luther wrote:

> On Wed, Aug 30, 2006 at 10:52:02AM -0700, David Lang wrote:
>> On Wed, 30 Aug 2006, Olaf Hering wrote:
>>
>>>> you are assuming that
>>>>
>>>> 1. modules are enabled and ipw2200 is compiled as a module
>>>
>>> No, why?
>>
>> becouse if the ipw isn't compiled as a module then it's initialized
>> (without firmware) before the initramfs or initrd is run. if it could be
>> initialized later without being a module then it could be initialized as
>> part of the normal system
>
> Euh, ...
>
> I wonder why you need to initialize the ipw2200 module so early ? It was my
> understanding that the initramfs thingy was run very early in the process, and
> i had even thought of moving fbdev modules into it.
>
> Do you really need to bring up ipw2200 so early ? It is some kind of wireless
> device, right ?

if modules are not in use the device is initialized when the kernel starts up. 
this is before any userspace starts.

> As for initramfs, you can just cat it behind the kernel, and it should work
> just fine, or at least this is how it was supposed to work.

yes, if I want to set one up.

other then this new requirement to have the ipw2200 driver as a module I have no 
reason to use one. normal userspace is good enough for me.

David Lang

> Friendly,
>
> Sven Luther
>

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Sven Luther]

On Wed, Aug 30, 2006 at 11:20:53AM -0700, David Lang wrote:
> On Wed, 30 Aug 2006, Sven Luther wrote:
> 
> >On Wed, Aug 30, 2006 at 10:52:02AM -0700, David Lang wrote:
> >>On Wed, 30 Aug 2006, Olaf Hering wrote:
> >>
> >>>>you are assuming that
> >>>>
> >>>>1. modules are enabled and ipw2200 is compiled as a module
> >>>
> >>>No, why?
> >>
> >>becouse if the ipw isn't compiled as a module then it's initialized
> >>(without firmware) before the initramfs or initrd is run. if it could be
> >>initialized later without being a module then it could be initialized as
> >>part of the normal system
> >
> >Euh, ...
> >
> >I wonder why you need to initialize the ipw2200 module so early ? It was my
> >understanding that the initramfs thingy was run very early in the process, 
> >and
> >i had even thought of moving fbdev modules into it.
> >
> >Do you really need to bring up ipw2200 so early ? It is some kind of 
> >wireless
> >device, right ?
> 
> if modules are not in use the device is initialized when the kernel starts 
> up. this is before any userspace starts.

Well. but you could do the initialization at open time too, like the other
case that was mentioned here, no ? 

> >As for initramfs, you can just cat it behind the kernel, and it should work
> >just fine, or at least this is how it was supposed to work.
> 
> yes, if I want to set one up.
> 
> other then this new requirement to have the ipw2200 driver as a module I 
> have no reason to use one. normal userspace is good enough for me.

Well, ok.

The real question seems to be if we want to keep the firmware inside the
driver or not.

If we want to remove it, then we put, not the module, but the firmware itself
with some basic userspace to load it on demand in the initramfs, and this is
reason enough to create an initramfs. The fact that the builtin device is
initialized before the initramfs is loaded seems like a bug to me, since the
idea of the initramfs (well, one of them at least), was to initialize it early
enough for this kind of things.

If on the other side, it is more important to not have an initramfs (because
of security issues, or bootloader constraints or what not), then sure, there
is not much choice than putting the firmware in the driver or in the kernel
directly.

But again, the initramfs is just a memory space available at the end of the
kernel, and there is no hardware-related constraint to access it which are in
any way different from having the firmware linked in together with the kernel,
so it is only a matter of organisation of code, as well as taking a decision
on the above, and to act accordyingly.

Does using an initramfs really have some negative side, security related ?
Would some kind of signed or encrypted initramfs be preferable there ? 

Friendly,

Sven Luther

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[David Lang]

On Wed, 30 Aug 2006, Sven Luther wrote:

>>>
>>> Do you really need to bring up ipw2200 so early ? It is some kind of
>>> wireless
>>> device, right ?
>>
>> if modules are not in use the device is initialized when the kernel starts
>> up. this is before any userspace starts.
>
> Well. but you could do the initialization at open time too, like the other
> case that was mentioned here, no ?

no, at least not in the current kernel. as was mentioned earlier in this thread 
the ipw2200 needs the firmware at initialization, but some others don't need it 
until open. I don't know if it's even possible to re-write the driver to do 
this.

>>> As for initramfs, you can just cat it behind the kernel, and it should work
>>> just fine, or at least this is how it was supposed to work.
>>
>> yes, if I want to set one up.
>>
>> other then this new requirement to have the ipw2200 driver as a module I
>> have no reason to use one. normal userspace is good enough for me.
>
> Well, ok.
>
> The real question seems to be if we want to keep the firmware inside the
> driver or not.
>
> If we want to remove it, then we put, not the module, but the firmware itself
> with some basic userspace to load it on demand in the initramfs, and this is
> reason enough to create an initramfs. The fact that the builtin device is
> initialized before the initramfs is loaded seems like a bug to me, since the
> idea of the initramfs (well, one of them at least), was to initialize it early
> enough for this kind of things.

this isn't my understanding.

my understanding is that the kernel fully initializes all built-in drivers, then 
loads userspace and starts running it.

that userspace can be on a device that it knows how to read, or it can be 
userspace on initramfs so that you can load additional modules to give you 
access to the hardware that you want to run on.

this is needed if your root drive is a SCSI drive and you have it's driver 
compiled as a module for example.

this is needed if your root drive uses dm and you need to initialize the array 
(one advantage of md, from the user standpoint, is that it doesn't require this 
additional layer before use)

however this is not soon enough to supply the firmware for devices like this.

> If on the other side, it is more important to not have an initramfs (because
> of security issues, or bootloader constraints or what not), then sure, there
> is not much choice than putting the firmware in the driver or in the kernel
> directly.
>
> But again, the initramfs is just a memory space available at the end of the
> kernel, and there is no hardware-related constraint to access it which are in
> any way different from having the firmware linked in together with the kernel,
> so it is only a matter of organisation of code, as well as taking a decision
> on the above, and to act accordyingly.

if the firmware needed for any drivers compiled in was appended to the kernel 
the same way that initramfs is, without requireing the other things needed to 
make initrmfs useable I think that would be reasonable (bundling them togeather 
as opposed to embedding the firmware in the kernel). it may even be possible to 
have the firmware as files in a initramfs that contains nothing else, and the 
kernel knows how to read the data directly (without the hotplug firmware request 
userspace stuff)

> Does using an initramfs really have some negative side, security related ?
> Would some kind of signed or encrypted initramfs be preferable there ?

adding an initramfs to a system that doesn't need it otherwise adds 
complications to the configure and boot process.

requireing modules when they weren't required before adds complication, and 
if/when the patch that's floating around to eliminate access to /dev/kmem is 
ever accepted, there are security advantages of running a kernel that doesn't 
have any support for run-time modifications (i.e. module loading).

I realize that many people want to make initramfs mandatory (with things like 
partition detection moved to userspace), but unless there is a standard 
initramfs that is shipped and maintained with the kernel to implement things 
like this (see the klibc discussion a few weeks ago) you are adding 
complications without much of a benifit to the user.

David Lang

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Sven Luther]

On Wed, Aug 30, 2006 at 12:34:16PM -0700, David Lang wrote:
> On Wed, 30 Aug 2006, Sven Luther wrote:
> 
> >>>
> >>>Do you really need to bring up ipw2200 so early ? It is some kind of
> >>>wireless
> >>>device, right ?
> >>
> >>if modules are not in use the device is initialized when the kernel starts
> >>up. this is before any userspace starts.
> >
> >Well. but you could do the initialization at open time too, like the other
> >case that was mentioned here, no ?
> 
> no, at least not in the current kernel. as was mentioned earlier in this 
> thread the ipw2200 needs the firmware at initialization, but some others 
> don't need it until open. I don't know if it's even possible to re-write 
> the driver to do this.

Oh well, this doesn't explain why it is so, but i suppose you know what you
speak about.

> >>>As for initramfs, you can just cat it behind the kernel, and it should 
> >>>work
> >>>just fine, or at least this is how it was supposed to work.
> >>
> >>yes, if I want to set one up.
> >>
> >>other then this new requirement to have the ipw2200 driver as a module I
> >>have no reason to use one. normal userspace is good enough for me.
> >
> >Well, ok.
> >
> >The real question seems to be if we want to keep the firmware inside the
> >driver or not.
> >
> >If we want to remove it, then we put, not the module, but the firmware 
> >itself
> >with some basic userspace to load it on demand in the initramfs, and this 
> >is
> >reason enough to create an initramfs. The fact that the builtin device is
> >initialized before the initramfs is loaded seems like a bug to me, since 
> >the
> >idea of the initramfs (well, one of them at least), was to initialize it 
> >early
> >enough for this kind of things.
> 
> this isn't my understanding.

Indeed, in the initrd era, the ramdisk was initialized too late for this kind
of stuff, but it was one of the features of the initramfs ramdisks to
initialize it earlier, which made firmware loading possible.

> my understanding is that the kernel fully initializes all built-in drivers, 
> then loads userspace and starts running it.

Well, there is userspace and userspace.

> that userspace can be on a device that it knows how to read, or it can be 
> userspace on initramfs so that you can load additional modules to give you 
> access to the hardware that you want to run on.

Yep, but initramfs is initialized ways earlier than normal userspace.

> this is needed if your root drive is a SCSI drive and you have it's driver 
> compiled as a module for example.

Ok.

> this is needed if your root drive uses dm and you need to initialize the 
> array (one advantage of md, from the user standpoint, is that it doesn't 
> require this additional layer before use)

Ok.

> however this is not soon enough to supply the firmware for devices like 
> this.

Are you sure of this ? This is somewhat contrary to what i have heard, and it
sure would make sense to be able to access the initramfs ramdisk much earlier.

> >If on the other side, it is more important to not have an initramfs 
> >(because
> >of security issues, or bootloader constraints or what not), then sure, 
> >there
> >is not much choice than putting the firmware in the driver or in the kernel
> >directly.
> >
> >But again, the initramfs is just a memory space available at the end of the
> >kernel, and there is no hardware-related constraint to access it which are 
> >in
> >any way different from having the firmware linked in together with the 
> >kernel,
> >so it is only a matter of organisation of code, as well as taking a 
> >decision
> >on the above, and to act accordyingly.
> 
> if the firmware needed for any drivers compiled in was appended to the 
> kernel the same way that initramfs is, without requireing the other things 
> needed to make initrmfs useable I think that would be reasonable (bundling 
> them togeather as opposed to embedding the firmware in the kernel). it may 
> even be possible to have the firmware as files in a initramfs that contains 
> nothing else, and the kernel knows how to read the data directly (without 
> the hotplug firmware request userspace stuff)

Indeed, and it seems to me that exactly this kind of use was indeed considered
when the initramfs infrastructure was designed. Not sure about the latest bit
concerning hotplug though.

> >Does using an initramfs really have some negative side, security related ?
> >Would some kind of signed or encrypted initramfs be preferable there ?
> 
> adding an initramfs to a system that doesn't need it otherwise adds 
> complications to the configure and boot process.

Yeah, well, if it is just cated at the end of the kernel, it would be rather
minimal.

> requireing modules when they weren't required before adds complication, and 
> if/when the patch that's floating around to eliminate access to /dev/kmem 
> is ever accepted, there are security advantages of running a kernel that 
> doesn't have any support for run-time modifications (i.e. module loading).

Indeed.

> I realize that many people want to make initramfs mandatory (with things 
> like partition detection moved to userspace), but unless there is a 
> standard initramfs that is shipped and maintained with the kernel to 
> implement things like this (see the klibc discussion a few weeks ago) you 
> are adding complications without much of a benifit to the user.

Well, since debian moved to 2.6 kernels on powerpc, i have been using ramdisks,
and let me assure you that from a distribution point of view, it is way more
elegant and nice than the non-ramdisk solution.

Friendly,

Sven Luther

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[David Lang]

On Wed, 30 Aug 2006, Sven Luther wrote:

>> no, at least not in the current kernel. as was mentioned earlier in this
>> thread the ipw2200 needs the firmware at initialization, but some others
>> don't need it until open. I don't know if it's even possible to re-write
>> the driver to do this.
>
> Oh well, this doesn't explain why it is so, but i suppose you know what you
> speak about.

I'm a user, not a developer. I know what is, but not nessasarily why, and I have 
no idea how bad it would be to change.

>>> If we want to remove it, then we put, not the module, but the firmware
>>> itself
>>> with some basic userspace to load it on demand in the initramfs, and this
>>> is
>>> reason enough to create an initramfs. The fact that the builtin device is
>>> initialized before the initramfs is loaded seems like a bug to me, since
>>> the
>>> idea of the initramfs (well, one of them at least), was to initialize it
>>> early
>>> enough for this kind of things.
>>
>> this isn't my understanding.
>
> Indeed, in the initrd era, the ramdisk was initialized too late for this kind
> of stuff, but it was one of the features of the initramfs ramdisks to
> initialize it earlier, which made firmware loading possible.
>
>> my understanding is that the kernel fully initializes all built-in drivers,
>> then loads userspace and starts running it.
>
> Well, there is userspace and userspace.
>
>> that userspace can be on a device that it knows how to read, or it can be
>> userspace on initramfs so that you can load additional modules to give you
>> access to the hardware that you want to run on.
>
> Yep, but initramfs is initialized ways earlier than normal userspace.
>
>> however this is not soon enough to supply the firmware for devices like
>> this.
>
> Are you sure of this ? This is somewhat contrary to what i have heard, and it
> sure would make sense to be able to access the initramfs ramdisk much earlier.

I could easily be wrong about this. can someone who really knows weigh in on 
this?

>>> If on the other side, it is more important to not have an initramfs
>>> (because
>>> of security issues, or bootloader constraints or what not), then sure,
>>> there
>>> is not much choice than putting the firmware in the driver or in the kernel
>>> directly.
>>>
>>> But again, the initramfs is just a memory space available at the end of the
>>> kernel, and there is no hardware-related constraint to access it which are
>>> in
>>> any way different from having the firmware linked in together with the
>>> kernel,
>>> so it is only a matter of organisation of code, as well as taking a
>>> decision
>>> on the above, and to act accordyingly.
>>
>> if the firmware needed for any drivers compiled in was appended to the
>> kernel the same way that initramfs is, without requireing the other things
>> needed to make initrmfs useable I think that would be reasonable (bundling
>> them togeather as opposed to embedding the firmware in the kernel). it may
>> even be possible to have the firmware as files in a initramfs that contains
>> nothing else, and the kernel knows how to read the data directly (without
>> the hotplug firmware request userspace stuff)
>
> Indeed, and it seems to me that exactly this kind of use was indeed considered
> when the initramfs infrastructure was designed. Not sure about the latest bit
> concerning hotplug though.

this gets back to the question of how early this early userspace is

David Lang

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Jim Crilly]

On 08/30/06 02:11:51PM -0700, David Lang wrote:
> >Yep, but initramfs is initialized ways earlier than normal userspace.
> >
> >>however this is not soon enough to supply the firmware for devices like
> >>this.
> >
> >Are you sure of this ? This is somewhat contrary to what i have heard, and 
> >it
> >sure would make sense to be able to access the initramfs ramdisk much 
> >earlier.
> 
> I could easily be wrong about this. can someone who really knows weigh in 
> on this?
> 

>From looking at my current boot logs it appears that initramfs is setup right
after the CPUs are brought up, so it should be available before any drivers
are initialized and they should be able to get to their firmware in the
initramfs as long as it's in the right path in the image.  I don't have any
drivers that require external firmware to test that theory out with though.

[4294668.249000] checking TSC synchronization across 2 CPUs: passed.
[4294668.250000] Brought up 2 CPUs
[4294668.277000] migration_cost=1000
[4294668.277000] checking if image is initramfs... it is
[4294668.452000] Freeing initrd memory: 1358k freed
[4294668.452000] NET: Registered protocol family 16
[4294668.453000] ACPI: bus type pci registered
[4294668.453000] PCI: PCI BIOS revision 2.10 entry at 0xfd7e0, last bus=1

Jim.

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Olaf Hering]

On Wed, Aug 30, David Lang wrote:

> >initramfs is always in use.
> 
> not on my machines.

klibc can be build like:

        cd linux-2.6.*
        make headers_install INSTALL_HDR_PATH=/dev/shm/$$
        cd ..
        wget http://www.kernel.org/pub/linux/libs/klibc/Testing/klibc-1.4.29.tar.bz2
        tar xfj klibc-*.tar.bz2
        cd klibc-*
        ln -sfvbn /dev/shm/$$ linux
        make

Every other libc will do it as well, adjust the filelist as needed.
Try this as CONFIG_INITRAMFS_SOURCE= :

# A simple initramfs
dir /dev 0755 0 0
nod /dev/console 0600 0 0 c 5 1
nod /dev/kmsg 0600 0 0 c 1 11
dir /sbin 0755 0 0
dir /lib 0755 0 0
dir /lib/firmware 0755 0 0
file /lib/firmware/name.ext /tmp/foofirmware.bin 0755 0 0
file /sbin/hotplug /home/olaf/kernel/hotplug.sh 0755 0 0
file /sbin/cat /home/olaf/kernel/klibc-1.4.29/usr/utils/static/cat 0755 0 0
file /sbin/mount /home/olaf/kernel/klibc-1.4.29/usr/utils/static/mount 0755 0 0
file /sbin/mkdir /home/olaf/kernel/klibc-1.4.29/usr/utils/static/mkdir 0755 0 0
file /sbin/mknod /home/olaf/kernel/klibc-1.4.29/usr/utils/static/mknod 0755 0 0
file /sbin/minips /home/olaf/kernel/klibc-1.4.29/usr/utils/static/minips 0755 0 0
file /sbin/umount /home/olaf/kernel/klibc-1.4.29/usr/utils/static/umount 0755 0 0
file /sbin/uname /home/olaf/kernel/klibc-1.4.29/usr/utils/static/uname 0755 0 0
file /sh /home/olaf/kernel/klibc-1.4.29/usr/dash/sh 0755 0 0
#optional:
file /init /home/olaf/kernel/init.sh 0755 0 0

Try this as hotplug.sh:

#!/sh
if test "$SEQNUM" -lt 42 ; then
echo "$1 SEQNUM=$SEQNUM ACTION=$ACTION DEVPATH=$DEVPATH" > /dev/kmsg
fi
if test "$SEQNUM" = 1 ; then
        mkdir /sys
        mkdir /proc
        mount -t sysfs sysfs /sys
        mount -t proc proc /proc
        set > /1.env
        echo "$*" > /1.arg
        cat /proc/cpuinfo > /cpuinfo
        cat /sys/power/state > /state
        umount /proc
        umount /sys
fi


Try this as init.sh:
#!/sh
/sh


I assume you can adjust hotplug.sh for your ipw needs.

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Michael Buesch]

On Tuesday 29 August 2006 04:15, Oleg Verych wrote:
> James Bottomley wrote:
> > On Tue, 2006-08-29 at 02:35 +0200, Oleg Verych wrote:
> > 
> >>request_firmware() is dead also.
> >>YMMV, but three years, and there are still big chunks of binary in kernel.
> >>And please don't add new useless info _in_ it.
> > 
> > 
> > I er don't think so.
> > 
> Hell, what can be as easy as this:
> ,-
> |modprobe $drv
> |(dd </lib/firmware/$drv.bin>/dev/blobs && echo OK) || echo Error
> `-
> where /dev/blobs is similar to /dev/port or even /dev/null char device.
> if synchronization is needed, add `echo $drv >/dev/blobs`, remove modprobe,

Please tell me how this is going to work, if we don't
know which firmware (version) is needed before be actually
initialize the device?

-- 
Greetings Michael.

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Marcel Holtmann]

Hi James,

> From: Jon Masters <jcm@redhat.com>
> 
> Right now, various kernel modules are being migrated over to use
> request_firmware in order to pull in binary firmware blobs from userland
> when the module is loaded. This makes sense.
> 
> However, there is right now little mechanism in place to automatically
> determine which binary firmware blobs must be included with a kernel in
> order to satisfy the prerequisites of these drivers. This affects
> vendors, but also regular users to a certain extent too.
> 
> The attached patch introduces MODULE_FIRMWARE as a mechanism for
> advertising that a particular firmware file is to be loaded - it will
> then show up via modinfo and could be used e.g. when packaging a kernel.

this patch was debated, but we never came to a final conclusion. I am
all for it. It is simple and can improve the current situation.

> Signed-off-by: Jon Masters <jcm@redhat.com>

Signed-off-by: Marcel Holtmann <marcel@holtmann.org>

Regards

Marcel

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Greg KH]

On Tue, Aug 29, 2006 at 03:13:34PM +0200, Marcel Holtmann wrote:
> Hi James,
> 
> > From: Jon Masters <jcm@redhat.com>
> > 
> > Right now, various kernel modules are being migrated over to use
> > request_firmware in order to pull in binary firmware blobs from userland
> > when the module is loaded. This makes sense.
> > 
> > However, there is right now little mechanism in place to automatically
> > determine which binary firmware blobs must be included with a kernel in
> > order to satisfy the prerequisites of these drivers. This affects
> > vendors, but also regular users to a certain extent too.
> > 
> > The attached patch introduces MODULE_FIRMWARE as a mechanism for
> > advertising that a particular firmware file is to be loaded - it will
> > then show up via modinfo and could be used e.g. when packaging a kernel.
> 
> this patch was debated, but we never came to a final conclusion. I am
> all for it. It is simple and can improve the current situation.
> 
> > Signed-off-by: Jon Masters <jcm@redhat.com>
> 
> Signed-off-by: Marcel Holtmann <marcel@holtmann.org>

Since your tree will depend on this change, you can add it there and
add:

Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>

from me to the patch.

thanks,

greg k-h

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Marcel Holtmann]

Hi Greg,

> > > Right now, various kernel modules are being migrated over to use
> > > request_firmware in order to pull in binary firmware blobs from userland
> > > when the module is loaded. This makes sense.
> > > 
> > > However, there is right now little mechanism in place to automatically
> > > determine which binary firmware blobs must be included with a kernel in
> > > order to satisfy the prerequisites of these drivers. This affects
> > > vendors, but also regular users to a certain extent too.
> > > 
> > > The attached patch introduces MODULE_FIRMWARE as a mechanism for
> > > advertising that a particular firmware file is to be loaded - it will
> > > then show up via modinfo and could be used e.g. when packaging a kernel.
> > 
> > this patch was debated, but we never came to a final conclusion. I am
> > all for it. It is simple and can improve the current situation.
> > 
> > > Signed-off-by: Jon Masters <jcm@redhat.com>
> > 
> > Signed-off-by: Marcel Holtmann <marcel@holtmann.org>
> 
> Since your tree will depend on this change, you can add it there and
> add:
> 
> Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>
> 
> from me to the patch.

do we consider this 2.6.18 or 2.6.19 material? For the Bluetooth
subsystem I can easily write a patch that makes use of it.

Regards

Marcel

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Bodo Eggert]

Michael Buesch <mb@bu3sch.de> wrote:

> The ipw needs the firmware on insmod time (in contrast to bcm43xx
> for example, which needs it on ifconfig up time).
> So ipw needs to call request_firmware at insmod time. In case of
> built-in, that is when the initcall happens. No userland is available
> and request_firmware can not call the userspace helpers to upload
> the firmware to sysfs.
> Well, not really easy to find a sane solution for this. :)

Can you trigger loading the firmware from userspace?
echo 1 > /sys/module/iw2200/parameters/load_firmware_now
-- 
Ich danke GMX dafür, die Verwendung meiner Adressen mittels per SPF
verbreiteten Lügen zu sabotieren.

http://david.woodhou.se/why-not-spf.html

[PATCH] MODULE_FIRMWARE for binary firmware(s)

[Jon Masters]

From: Jon Masters <jcm@redhat.com>

Right now, various kernel modules are being migrated over to use
request_firmware in order to pull in binary firmware blobs from userland
when the module is loaded. This makes sense.

However, there is right now little mechanism in place to automatically
determine which binary firmware blobs must be included with a kernel in
order to satisfy the prerequisites of these drivers. This affects
vendors, but also regular users to a certain extent too.

The attached patch introduces MODULE_FIRMWARE as a mechanism for
advertising that a particular firmware file is to be loaded - it will
then show up via modinfo and could be used e.g. when packaging a kernel.

Signed-off-by: Jon Masters <jcm@redhat.com>

diff -urN linux-2.6.16.2_orig/include/linux/module.h linux-2.6.16.2_dev/include/linux/module.h
--- linux-2.6.16.2_orig/include/linux/module.h  2006-04-07 17:56:47.000000000 +0100
+++ linux-2.6.16.2_dev/include/linux/module.h   2006-04-12 13:51:56.000000000 +0100
@@ -155,6 +155,8 @@
 */
 #define MODULE_VERSION(_version) MODULE_INFO(version, _version)

+#define MODULE_FIRMWARE(_firmware) MODULE_INFO(firmware, _firmware)
+
 /* Given an address, look for it in the exception tables */
 const struct exception_table_entry *search_exception_tables(unsigned long add);

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[David Lang]

Jon,
   would it be possible to have something less then an initrd that would 
allow the firmware blob to be packaged with the kernel? Your approach is 
just fine if the things that will need firmware are compiled as modules 
(the firmware can be placed on the same filesystem image as the modules 
that need it) but some people prefer to build monolithic kernel images, 
and in those cases there is no way for the kernel to get the firmware as 
there is no filesystem available yet.

I know that Rob Landley is doing some tricks with firmware linux to append 
an initramfs to the kernel image (http://www.landley.net/code/firmware/). 
There is already support in the kconfig for specifying the initramfs 
source, why culdn't we have the build process fetch any firmware needed by 
the non-modular portions and store them at the end of the kernel image so 
that they can be found during boot.

if nothing else make a skeleton initramfs that contains the modules and 
just enough other logic to continue the boot process as if the initramfs 
wasn't involved.

right now I have a laptop that's not working on wireless due to this exact 
problem (ipw2200 driver, and I haven't taken the time to setup modules for 
it yet)

David Lang


On Wed, 19 Apr 2006, Jon Masters wrote:

> Date: Wed, 19 Apr 2006 00:41:56 +0100
> From: Jon Masters <jonathan@jonmasters.org>
> To: akpm@osdl.org, linux-kernel@vger.kernel.org
> Subject: [PATCH] MODULE_FIRMWARE for binary firmware(s)
> 
> From: Jon Masters <jcm@redhat.com>
>
> Right now, various kernel modules are being migrated over to use
> request_firmware in order to pull in binary firmware blobs from userland
> when the module is loaded. This makes sense.
>
> However, there is right now little mechanism in place to automatically
> determine which binary firmware blobs must be included with a kernel in
> order to satisfy the prerequisites of these drivers. This affects
> vendors, but also regular users to a certain extent too.
>
> The attached patch introduces MODULE_FIRMWARE as a mechanism for
> advertising that a particular firmware file is to be loaded - it will
> then show up via modinfo and could be used e.g. when packaging a kernel.
>
> Signed-off-by: Jon Masters <jcm@redhat.com>
>
> diff -urN linux-2.6.16.2_orig/include/linux/module.h linux-2.6.16.2_dev/include/linux/module.h
> --- linux-2.6.16.2_orig/include/linux/module.h  2006-04-07 17:56:47.000000000 +0100
> +++ linux-2.6.16.2_dev/include/linux/module.h   2006-04-12 13:51:56.000000000 +0100
> @@ -155,6 +155,8 @@
> */
> #define MODULE_VERSION(_version) MODULE_INFO(version, _version)
>
> +#define MODULE_FIRMWARE(_firmware) MODULE_INFO(firmware, _firmware)
> +
> /* Given an address, look for it in the exception tables */
> const struct exception_table_entry *search_exception_tables(unsigned long add);
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

-- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
  -- C.A.R. Hoare

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Jon Masters]

On 4/19/06, David Lang <dlang@digitalinsight.com> wrote:

>    would it be possible to have something less then an initrd that would
> allow the firmware blob to be packaged with the kernel?

With some modifications perhaps. I don't know if I see the value tbh :-)

> Your approach is just fine if the things that will need firmware are
> compiled as modules

Hmmm. Yeah. I'm not sure what the general feeling is on this - I'm
tempted to say that we expect modules to be used and that if they're
not then the vendor/user has to do the hoop jumping for themselves.
This code won't stop you from making a monolithic kernel and
satisfying any module requirements for yourself :-)

Jon.

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[David Lang]

On Wed, 19 Apr 2006, Jon Masters wrote:

>> Your approach is just fine if the things that will need firmware are
>> compiled as modules
>
> Hmmm. Yeah. I'm not sure what the general feeling is on this - I'm
> tempted to say that we expect modules to be used and that if they're
> not then the vendor/user has to do the hoop jumping for themselves.
> This code won't stop you from making a monolithic kernel and
> satisfying any module requirements for yourself :-)

Two things with this.

1. there is no way to satisfy the firmware requirements currently

2. I thought I heard Linus state recently that makeing something only work 
as a module was unacceptable, officially stateing that modules are 
required and monolithic kernels aren't allowed anymore doesn't sound 
reasonable.

David Lang

-- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
  -- C.A.R. Hoare

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Jon Masters]

On 4/19/06, David Lang <dlang@digitalinsight.com> wrote:

> 2. I thought I heard Linus state recently that makeing something only work
> as a module was unacceptable, officially stateing that modules are
> required and monolithic kernels aren't allowed anymore doesn't sound
> reasonable.

It works fine with or without modules - it's just that if you don't
have the module then you have to supply the firmware for yourself.
It's just like any of the other MODULE_ tags.

Jon.

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Duncan Sands]

Hi Jon,

> However, there is right now little mechanism in place to automatically
> determine which binary firmware blobs must be included with a kernel in
> order to satisfy the prerequisites of these drivers. This affects
> vendors, but also regular users to a certain extent too.
> 
> The attached patch introduces MODULE_FIRMWARE as a mechanism for
> advertising that a particular firmware file is to be loaded - it will
> then show up via modinfo and could be used e.g. when packaging a kernel.

I haven't really understood what problem this solves.  Is this just a
standardised form of documentation, or are you imagining that an automatic
tool will use this to auto include a minimal set of firmware files in an
initrd?  I'm thinking of something like this: (1) redhat (or whoever) ships
firmware files for every driver under the sun in /lib/firmware; (2) redhat
wants to allow users to have a customized initrd with only essential drivers;
(3) the tool goes through the list of essential drivers, looks up the firmware
string via MODULE_FIRMWARE, finds the file in /lib/firmware, and includes it
in the initrd.

All the best,

Duncan.

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Jon Masters]

On 4/19/06, Duncan Sands <duncan.sands@math.u-psud.fr> wrote:

> Hi Jon,

Hi Duncan.

> > However, there is right now little mechanism in place to automatically
> > determine which binary firmware blobs must be included with a kernel in
> > order to satisfy the prerequisites of these drivers. This affects
> > vendors, but also regular users to a certain extent too.
> >
> > The attached patch introduces MODULE_FIRMWARE as a mechanism for
> > advertising that a particular firmware file is to be loaded - it will
> > then show up via modinfo and could be used e.g. when packaging a kernel.

> I haven't really understood what problem this solves.  Is this just a
> standardised form of documentation, or are you imagining that an automatic
> tool will use this to auto include a minimal set of firmware files in an
> initrd?

I'm imagining that the resultant modinfo output can be used by a tool
for anyone to package up the correct firmware to go with a given
driver. Right now, there's no way to do that - i.e. we've gone
backwards from a standpoint of coupling a kernel with firmware. I
completely understand why firmware doesn't really belong in the
kernel, so let's add this :-)

> I'm thinking of something like this: (1) redhat (or whoever) ships
> firmware files for every driver under the sun in /lib/firmware; (2) redhat
> wants to allow users to have a customized initrd with only essential drivers;
> (3) the tool goes through the list of essential drivers, looks up the firmware
> string via MODULE_FIRMWARE, finds the file in /lib/firmware, and includes it
> in the initrd.

That kind of thing. It's not just Red Hat who benefit - anyone who
wants to package up a kernel and do something with it will want to
know about firmware they might need. Including everything in
/lib/firmware "just in case" is as ugly as having userspace tools with
duplicated logic that need to understand about the internals of a
driver module.

Jon.

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Duncan Sands]

> > I haven't really understood what problem this solves.  Is this just a
> > standardised form of documentation, or are you imagining that an automatic
> > tool will use this to auto include a minimal set of firmware files in an
> > initrd?
> 
> I'm imagining that the resultant modinfo output can be used by a tool
> for anyone to package up the correct firmware to go with a given
> driver.

If a tool is to do the packaging, then this means that the firmware must
already be present on the machine, for example in /lib/firmware.  Logically
speaking, that means the role of any tool is to select a subset of the files
in /lib/firmware, for example a minimal set for inclusion in an initrd.  After
all, it can't add new files that don't exist on the machine, since where would
it get them from?  (I suppose it could download them from "firmware central").
Is there any use for such a tool, besides creating small initrds?  How big an
issue is that?  Is there really any reason not to simply throw everything in
/lib/firmware into any initrd that is created?  But perhaps you are thinking of
the suppliers of a distribution who have a gazillion firmware files that they've
collected over the years, some of which are too old and some of which are too
recent for the drivers that will be shipped; and the tool is to select the subset
which is relevant for the shipped driver versions?  i.e. the firmware selection
process is not done on each end user's machine, but by the people preparing the
distribution.

> Right now, there's no way to do that - i.e. we've gone 
> backwards from a standpoint of coupling a kernel with firmware. I
> completely understand why firmware doesn't really belong in the
> kernel, so let's add this :-)

I guess a big difference between the speedtouch and the kinds of drivers
you seem to be thinking of, is that in your case there is a fairly tight
coupling between firmware and driver versions: a given driver version will
only work with a certain version of the firmware and vice-versa.  In the case
of the speedtouch, we have no control over (and not much knowledge about)
which firmware gets given to people along with their modems, so there is really
no coupling at all between firmware versions and driver versions.

> > I'm thinking of something like this: (1) redhat (or whoever) ships
> > firmware files for every driver under the sun in /lib/firmware; (2) redhat
> > wants to allow users to have a customized initrd with only essential drivers;
> > (3) the tool goes through the list of essential drivers, looks up the firmware
> > string via MODULE_FIRMWARE, finds the file in /lib/firmware, and includes it
> > in the initrd.
> 
> That kind of thing. It's not just Red Hat who benefit - anyone who
> wants to package up a kernel and do something with it will want to
> know about firmware they might need.

For this they could just read the documentation.  They'll need to anyway
just to find out where they are supposed to get the firmware from.

> Including everything in 
> /lib/firmware "just in case" is as ugly as having userspace tools with
> duplicated logic that need to understand about the internals of a
> driver module.

Don't distributions need to ship vast quantities of firmware in /lib/firmware
anyway, I mean firmware for every driver in the kernel, since they don't know
what hardware the end user may have?  So I guess you are talking about individual
users who compile their own kernels here.


Ciao,

D.

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Jon Masters]

On 4/19/06, Duncan Sands <duncan.sands@math.u-psud.fr> wrote:

> > > I haven't really understood what problem this solves.  Is this just a
> > > standardised form of documentation, or are you imagining that an automatic
> > > tool will use this to auto include a minimal set of firmware files in an
> > > initrd?
> >
> > I'm imagining that the resultant modinfo output can be used by a tool
> > for anyone to package up the correct firmware to go with a given
> > driver.

> If a tool is to do the packaging, then this means that the firmware must
> already be present on the machine, for example in /lib/firmware.

Yes. Although, potentially more clever things could happen in
userspace in the future.

> that means the role of any tool is to select a subset of the files
> in /lib/firmware, for example a minimal set for inclusion in an initrd.

For example.

> Is there really any reason not to simply throw everything in
> /lib/firmware into any initrd that is created?

You /could/ just build every driver into the initrd too, and every
firmware, and... but it might be nice if it was possible to know what
should be where. Right now, we've lost that because decoupling
firmware from the kernel means that tools outside of the kernel can't
tell what firmware files should be around.

> > Right now, there's no way to do that - i.e. we've gone
> > backwards from a standpoint of coupling a kernel with firmware. I
> > completely understand why firmware doesn't really belong in the
> > kernel, so let's add this :-)
>
> I guess a big difference between the speedtouch and the kinds of drivers
> you seem to be thinking of, is that in your case there is a fairly tight
> coupling between firmware and driver versions: a given driver version will
> only work with a certain version of the firmware and vice-versa.  In the case
> of the speedtouch, we have no control over (and not much knowledge about)
> which firmware gets given to people along with their modems, so there is really
> no coupling at all between firmware versions and driver versions.

I've also repeatedly said that I don't think this really helps with
things like speedtouch since you can't distribute that firmware
anyway. This patch does help people who play nicely with the kernel
who are migrating away from shoving blobs into the kernel (quite
rightly) and who supply redistributable firmware. One example might be
the QLogic driver in my RFC.

> > That kind of thing. It's not just Red Hat who benefit - anyone who
> > wants to package up a kernel and do something with it will want to
> > know about firmware they might need.
>
> For this they could just read the documentation.  They'll need to anyway
> just to find out where they are supposed to get the firmware from.

You're assuming firmware is always on some random vendor website under
a questionable license and can't be redistributed. That wasn't my
first consideration - this doesn't really help much there, except you
get to know what you don't have :-)

Jon.

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Jon Masters]

On 4/19/06, Jon Masters <jonathan@jonmasters.org> wrote:

> The attached patch introduces MODULE_FIRMWARE as a mechanism for
> advertising that a particular firmware file is to be loaded - it will
> then show up via modinfo and could be used e.g. when packaging a kernel.

I (and several others who met at OLS) would really like for this to
get upstream. Can you sling that one line patch (will forward again)
into -mm please?

Jon.