[PATCH] firmware: Allow release-specific firmware dir

[PATCH] firmware: Allow release-specific firmware dir

[Jeff Mahoney]

 Every kernel package trying to provide files under /lib/firmware runs
 into problems really quickly with multiple kernels installed, especially
 with respect to distribution package tracking of the files.

 This patch moves them to /lib/firmware/$KERNELRELEASE. udev v127's
 firmware.sh looks there first before falling back to /lib/firmware.

Signed-off-by: Jeff Mahoney <jeffm@suse.com>
---
 Makefile |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- a/Makefile
+++ b/Makefile
@@ -1025,7 +1025,7 @@ depend dep:
 
 # ---------------------------------------------------------------------------
 # Firmware install
-INSTALL_FW_PATH=$(INSTALL_MOD_PATH)/lib/firmware
+INSTALL_FW_PATH=$(INSTALL_MOD_PATH)/lib/firmware/$(KERNELRELEASE)
 export INSTALL_FW_PATH
 
 PHONY += firmware_install

-- 
Jeff Mahoney
SUSE Labs

Re: [PATCH] firmware: Allow release-specific firmware dir

[Andrew Morton]

On Tue, 09 Sep 2008 10:15:35 -0400
Jeff Mahoney <jeffm@suse.com> wrote:

>  Every kernel package trying to provide files under /lib/firmware runs
>  into problems really quickly with multiple kernels installed, especially
>  with respect to distribution package tracking of the files.

For how long has this problem existed?

>  This patch moves them to /lib/firmware/$KERNELRELEASE. udev v127's
>  firmware.sh looks there first before falling back to /lib/firmware.
> 
> Signed-off-by: Jeff Mahoney <jeffm@suse.com>
> ---
>  Makefile |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> --- a/Makefile
> +++ b/Makefile
> @@ -1025,7 +1025,7 @@ depend dep:
>  
>  # ---------------------------------------------------------------------------
>  # Firmware install
> -INSTALL_FW_PATH=$(INSTALL_MOD_PATH)/lib/firmware
> +INSTALL_FW_PATH=$(INSTALL_MOD_PATH)/lib/firmware/$(KERNELRELEASE)
>  export INSTALL_FW_PATH
>  
>  PHONY += firmware_install
> 

Re: [PATCH] firmware: Allow release-specific firmware dir

[David Woodhouse]

On Tue, 2008-09-09 at 10:15 -0400, Jeff Mahoney wrote:
>  This patch moves them to /lib/firmware/$KERNELRELEASE. udev v127's
>  firmware.sh looks there first before falling back to /lib/firmware.

NAK.

This isn't compatible with the version of udev that people actually have
installed, today. And it's also the _wrong_ thing to do.

Firmware really _isn't_ version-specific. If the firmware changes in
some incompatible way, then the filename given to request_firmware() has
to change too. Think of it like an soname on a library.

Besides, incompatible changes are fairly uncommon; firmware changes
rarely change the host<->device ABI in a way which breaks older drivers.
And that's _especially_ true for the firmwares which were present in the
kernel tree, which is all that you're moving around with this patch.

You shouldn't need more than one firmware package installed at a time --
if you are trying to do so, then that's a bug in your packaging.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

Re: [PATCH] firmware: Allow release-specific firmware dir

[David Miller]

From: David Woodhouse <dwmw2@infradead.org>
Date: Wed, 10 Sep 2008 16:01:59 -0700

> On Tue, 2008-09-09 at 10:15 -0400, Jeff Mahoney wrote:
> >  This patch moves them to /lib/firmware/$KERNELRELEASE. udev v127's
> >  firmware.sh looks there first before falling back to /lib/firmware.
> 
> NAK.
> 
> This isn't compatible with the version of udev that people actually have
> installed, today. And it's also the _wrong_ thing to do.
> 
> Firmware really _isn't_ version-specific.

Tell that to every Debian and Debian derived system on the planet.

To my knowledge, it is only fedora and possibly one or two other dists
that put the firmware files in a unary /lib/firmware location, rather
than a versioned /lib/firmware/$KERNELRELASE one.

Re: [PATCH] firmware: Allow release-specific firmware dir

[David Woodhouse]

On Wed, 2008-09-10 at 16:05 -0700, David Miller wrote:
> 
> Tell that to every Debian and Debian derived system on the planet.
> 
> To my knowledge, it is only fedora and possibly one or two other dists
> that put the firmware files in a unary /lib/firmware location, rather
> than a versioned /lib/firmware/$KERNELRELASE one.

So every time they upgrade from one kernel to the next, do they
automatically run b43-fwcutter again to make a new copy of the firmware,
for the new kernel?

Or if they use QLogic SCSI cards, do they download a new (but identical)
version of the firmware each time they upgrade their kernel?

It sounds like a daft arrangement to me -- although of course it's
possible for the packager to do whatever they like, by overriding
$(INSTALL_FW_PATH) in their package build.

It's definitely not something we should be doing upstream though.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

Re: [PATCH] firmware: Allow release-specific firmware dir

[Andrew Morton]

On Wed, 10 Sep 2008 16:15:11 -0700
David Woodhouse <dwmw2@infradead.org> wrote:

> On Wed, 2008-09-10 at 16:05 -0700, David Miller wrote:
> > 
> > Tell that to every Debian and Debian derived system on the planet.
> > 
> > To my knowledge, it is only fedora and possibly one or two other dists
> > that put the firmware files in a unary /lib/firmware location, rather
> > than a versioned /lib/firmware/$KERNELRELASE one.
> 
> So every time they upgrade from one kernel to the next, do they
> automatically run b43-fwcutter again to make a new copy of the firmware,
> for the new kernel?
> 
> Or if they use QLogic SCSI cards, do they download a new (but identical)
> version of the firmware each time they upgrade their kernel?
> 
> It sounds like a daft arrangement to me -- although of course it's
> possible for the packager to do whatever they like, by overriding
> $(INSTALL_FW_PATH) in their package build.
> 
> It's definitely not something we should be doing upstream though.
> 

It all seems a bit academic.  Breaking pre-v127 udev is a showstopper.

A suitable approach might be to copy the files to both /lib/firmware
and to /lib/firmware/$KERNELRELEASE, and to stop copying the files to
/lib/firmware in, oh, 2018 or so..

Re: [PATCH] firmware: Allow release-specific firmware dir

[David Miller]

From: David Woodhouse <dwmw2@infradead.org>
Date: Wed, 10 Sep 2008 16:15:11 -0700

> It's definitely not something we should be doing upstream though.

So you think it's ok that every Debian user has to learn this
magic incantation just to use current kernels?

I don't think it's nice to break things like this on people,
especially such a large group.  Getting this stuff to work is hard
enough, and we're just putting yet another barrier into the situation
and that can only mean less testers and contributors.

I do know several people who aren't testing and contributing because
the whole firmware shakeup is so bolixed and they really are exasperated
after spending hours trying to get it to work.

This costs people's time, which is valuable.

I hate to say it, but this kind of attitude seems to be pervasive
every time some negative ramification of the firmware changes shows
up.

Re: [PATCH] firmware: Allow release-specific firmware dir

[David Woodhouse]

On Wed, 2008-09-10 at 16:24 -0700, David Miller wrote:
> From: David Woodhouse <dwmw2@infradead.org>
> Date: Wed, 10 Sep 2008 16:15:11 -0700
> 
> > It's definitely not something we should be doing upstream though.
> 
> So you think it's ok that every Debian user has to learn this
> magic incantation just to use current kernels?

They don't. Debian uses firmware from /lib/firmware if it doesn't exist
in /lib/firmware/$(version) -- so they can install to the normal
location quite happily, and it should work just fine.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

Re: [PATCH] firmware: Allow release-specific firmware dir

[David Miller]

From: David Woodhouse <dwmw2@infradead.org>
Date: Wed, 10 Sep 2008 16:36:56 -0700

> On Wed, 2008-09-10 at 16:24 -0700, David Miller wrote:
> > From: David Woodhouse <dwmw2@infradead.org>
> > Date: Wed, 10 Sep 2008 16:15:11 -0700
> > 
> > > It's definitely not something we should be doing upstream though.
> > 
> > So you think it's ok that every Debian user has to learn this
> > magic incantation just to use current kernels?
> 
> They don't. Debian uses firmware from /lib/firmware if it doesn't exist
> in /lib/firmware/$(version) -- so they can install to the normal
> location quite happily, and it should work just fine.

That's not what I've heard from people more clueful from me.

For example, Herbert Xu (CC:'d) told me that because of this directory
usage, his wireless card couldn't find it's firmware with current
kernels.

Re: [PATCH] firmware: Allow release-specific firmware dir

[Herbert Xu]

On Wed, Sep 10, 2008 at 04:42:20PM -0700, David Miller wrote:
>
> For example, Herbert Xu (CC:'d) told me that because of this directory
> usage, his wireless card couldn't find it's firmware with current
> kernels.

Actually my complaint is still theoretical.

It's about firmware that's closely tied to the driver and where a
driver may not operate at all with a firmware of the wrong version.
There are wireless drivers which have version dependencies, e.g.,
newer ipw2200 drivers often require newer firmware which may not
be backwards compatible.

So I don't want to be in a situation where I install a new kernel,
find that its wireless driver doesn't function correctly for
whatever reason (which has happened to me previously at an airport),
and then not be able to boot back into my old kernel to fix it
because the firmware has now been overwritten.

I appreciate the thought on saving disk space.  However, I think
the safest default is to install firmware (at least those from the
kernel) into a version-specific directory.  Alternatively we should
ensure that all firmware have version strings embedded in their
names (e.g., ipw2200 versions their firmware so they would be OK),
and that they're kept up-to-date whenever the firmware changes.

Thanks,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH] firmware: Allow release-specific firmware dir

[David Woodhouse]

On Thu, 2008-09-11 at 10:23 +1000, Herbert Xu wrote:
> So I don't want to be in a situation where I install a new kernel,
> find that its wireless driver doesn't function correctly for
> whatever reason (which has happened to me previously at an airport),
> and then not be able to boot back into my old kernel to fix it
> because the firmware has now been overwritten.

Absolutely -- nobody should ever find themselves in that situation.
If a firmware changes in an incompatible way, its filename needs to
change too. The old one should still be available.

> I appreciate the thought on saving disk space.  However, I think
> the safest default is to install firmware (at least those from the
> kernel) into a version-specific directory. 

Even safer would be to have content-addressed firmware. Instead of
requesting it by filename, you request it by its md5sum. That way,
you're _guaranteed_ to have precisely what you expected.

But I think that's a bad idea too. As a general rule firmware isn't, and
shouldn't be, tied intimately to one particular version of the kernel.
It isn't even tied intimately to one particular version of the _driver_.

> Alternatively we should ensure that all firmware have version strings
> embedded in their names (e.g., ipw2200 versions their firmware so they
> would be OK), and that they're kept up-to-date whenever the firmware
> changes.

That's what we are doing already, as you say -- and what we should
always be doing.

And remember, ipw2200 isn't a particularly good example, because it's a
relatively recent driver and thus has always had its firmware shipped
_separately_ from the kernel.

All we're talking about here is the behaviour for the handful of older
drivers which I've recently dragged into this century by converting them
to use request_firmware(). Those drivers, by virtue of their age, are
mostly quite unlikely to receive _any_ kind of firmware update --
especially an update which changes their ABI.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

Re: [PATCH] firmware: Allow release-specific firmware dir

[Theodore Tso]

On Wed, Sep 10, 2008 at 04:24:04PM -0700, Andrew Morton wrote:
> 
> A suitable approach might be to copy the files to both /lib/firmware
> and to /lib/firmware/$KERNELRELEASE, and to stop copying the files to
> /lib/firmware in, oh, 2018 or so..

Or make it be a KCONFIG build option, so that people who are building
kernels using the distribution-specific kernel packaging tool can pass
in whatever is right for their distribution....

							- Ted

Re: [PATCH] firmware: Allow release-specific firmware dir

[David Newall]

David Miller wrote:
> From: David Woodhouse <dwmw2@infradead.org>
>   
>> Firmware really _isn't_ version-specific.
>>     
>
> Tell that to every Debian and Debian derived system on the planet.


And a damned nuisance it is, too.  Firmware *really* isn't kernel
version-specific, and I'd like Debian to "get" that.

Re: [PATCH] firmware: Allow release-specific firmware dir

[Faidon Liambotis]

David Miller wrote:
>> Firmware really _isn't_ version-specific.
> 
> Tell that to every Debian and Debian derived system on the planet.
> 
> To my knowledge, it is only fedora and possibly one or two other dists
> that put the firmware files in a unary /lib/firmware location, rather
> than a versioned /lib/firmware/$KERNELRELASE one.
[not speaking on behalf of the project, the kernel team or the
respective maintainers]

Apparently and afaik you are misinformed:

$ dpkg -L firmware-bnx2 (or [1])
...
/lib/firmware/bnx2-06-4.0.5.fw
/lib/firmware/bnx2-09-4.0.5.fw
...

$ dpkg -L firmware-iwlwifi (or [2])
...
/lib/firmware/iwlwifi-3945-1.ucode
/lib/firmware/iwlwifi-4965-1.ucode
/lib/firmware/iwlwifi-4965-2.ucode
...

$ dpkg -L firmware-qlogic (or [3])
...
/lib/firmware/ql2100_fw.bin
/lib/firmware/ql2200_fw.bin
/lib/firmware/ql2300_fw.bin
/lib/firmware/ql2322_fw.bin
/lib/firmware/ql2400_fw.bin
/lib/firmware/ql2500_fw.bin
...

etc. for the other firmware packages[4].

Regards,
Faidon

1: http://packages.debian.org/sid/all/firmware-bnx2/filelist
2: http://packages.debian.org/sid/all/firmware-iwlwifi/filelist
3: http://packages.debian.org/sid/all/firmware-qlogic/filelist
4: http://packages.debian.org/firmware

Re: [PATCH] firmware: Allow release-specific firmware dir

[Jeff Mahoney]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Faidon Liambotis wrote:
> David Miller wrote:
>>> Firmware really _isn't_ version-specific.
>> Tell that to every Debian and Debian derived system on the planet.
>>
>> To my knowledge, it is only fedora and possibly one or two other dists
>> that put the firmware files in a unary /lib/firmware location, rather
>> than a versioned /lib/firmware/$KERNELRELASE one.
> [not speaking on behalf of the project, the kernel team or the
> respective maintainers]
> 
> Apparently and afaik you are misinformed:

Well that's the thing. These are specific firmware packages that exist
outside of those created by the kernel. My original patch wasn't
proposing using /lib/firmware/$KERNELRELEASE for *all* firmware blobs,
just those that are generated and installed by the kernel build process.

The problem occurs when you have multiple versions of the kernel
installed and they all want to put firmware blobs into /lib/firmware.
The firmware blobs created by the kernel aren't versioned liked those in
the examples you've provided, so you end up with a bunch of blobs with
potentially different contents, all with the same names.

I'm not suggesting killing /lib/firmware-located blobs at all. That's
where you look once you discover that /lib/firmware/$KERNELRELEASE
doesn't have the blob you're looking for. That's where firmware packages
like the examples below would still place their files. I see no reason
to change that. If the blobs are versioned so that a particular name
always contains the same contents, that's fine too. What we can't have
is files in the same location with different contents.

In the end, though, Andrew's right. We can't go breaking udev prior to
127 with this.

- -Jeff

> $ dpkg -L firmware-bnx2 (or [1])
> ...
> /lib/firmware/bnx2-06-4.0.5.fw
> /lib/firmware/bnx2-09-4.0.5.fw
> ...
> 
> $ dpkg -L firmware-iwlwifi (or [2])
> ...
> /lib/firmware/iwlwifi-3945-1.ucode
> /lib/firmware/iwlwifi-4965-1.ucode
> /lib/firmware/iwlwifi-4965-2.ucode
> ...
> 
> $ dpkg -L firmware-qlogic (or [3])
> ...
> /lib/firmware/ql2100_fw.bin
> /lib/firmware/ql2200_fw.bin
> /lib/firmware/ql2300_fw.bin
> /lib/firmware/ql2322_fw.bin
> /lib/firmware/ql2400_fw.bin
> /lib/firmware/ql2500_fw.bin
> ...
> 
> etc. for the other firmware packages[4].
> 
> Regards,
> Faidon
> 
> 1: http://packages.debian.org/sid/all/firmware-bnx2/filelist
> 2: http://packages.debian.org/sid/all/firmware-iwlwifi/filelist
> 3: http://packages.debian.org/sid/all/firmware-qlogic/filelist
> 4: http://packages.debian.org/firmware


- --
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iEYEARECAAYFAkjIxYAACgkQLPWxlyuTD7LuSgCgmAOte+y8dU1TWODKPNwRnrCK
KO8An1VnOh+KvsfHrgtXZ1H/xSdCYuVL
=3/a7
-----END PGP SIGNATURE-----

Re: [PATCH] firmware: Allow release-specific firmware dir

[Marcel Holtmann]

Hi Dave,

> > It's definitely not something we should be doing upstream though.
> 
> So you think it's ok that every Debian user has to learn this
> magic incantation just to use current kernels?
> 
> I don't think it's nice to break things like this on people,
> especially such a large group.  Getting this stuff to work is hard
> enough, and we're just putting yet another barrier into the situation
> and that can only mean less testers and contributors.
> 
> I do know several people who aren't testing and contributing because
> the whole firmware shakeup is so bolixed and they really are exasperated
> after spending hours trying to get it to work.

it is that the Debian maintainer screwed this up. Upstream never
promoted kernel versioned directories for the firmware. I had a long
discussion with Dave about it at OLS and using the kernel version is
just plain wrong. The driver maintainers should version the firmware if
they break it in an API incompatible way.

To shed some light into the problem with Debian/Ubuntu. They install the
firmware in /lib/firmware/`uname -r`/ and everytime they bump their ABI
number, they have to install all the firmware again. This is just
braindead. Their udev script actually checks /lib/firmware first and
then the kernel versioned directory. So that is just fine.

Problem comes when installing let say 2.6.27 since then the firmware
will be looked up in /lib/firmware/ and /lib/firmware/2.6.27/ and
actually it will not be found. Since it is in /lib/firmware/2.6.26-xx/
or something similar.

So having the kernel install everything in /lib/firmware works just fine
with every distro. However looking for firmware that is not shipped with
the kernel, we have a problem since Debian/Ubuntu just not installs it
in the right directory. And there is nothing the kernel can do about it
since it will not touch firmware it doesn't ship. The distro has to fix
their firmware or the users have to place a copy in /lib/firmware/ where
it actually should have been in the first place.

Regards

Marcel

Re: [PATCH] firmware: Allow release-specific firmware dir

[Jeff Mahoney]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Marcel Holtmann wrote:
> Hi Dave,
> 
>>> It's definitely not something we should be doing upstream though.
>> So you think it's ok that every Debian user has to learn this
>> magic incantation just to use current kernels?
>>
>> I don't think it's nice to break things like this on people,
>> especially such a large group.  Getting this stuff to work is hard
>> enough, and we're just putting yet another barrier into the situation
>> and that can only mean less testers and contributors.
>>
>> I do know several people who aren't testing and contributing because
>> the whole firmware shakeup is so bolixed and they really are exasperated
>> after spending hours trying to get it to work.
> 
> it is that the Debian maintainer screwed this up. Upstream never
> promoted kernel versioned directories for the firmware. I had a long
> discussion with Dave about it at OLS and using the kernel version is
> just plain wrong. The driver maintainers should version the firmware if
> they break it in an API incompatible way.

They "should," but is that happening now? Out of all the firmware blobs
installed with 2.6.27-rcX, only the edgeport drier actually versions
them - and they're versioned because the driver requires different
versions for different hardware.

> To shed some light into the problem with Debian/Ubuntu. They install the
> firmware in /lib/firmware/`uname -r`/ and everytime they bump their ABI
> number, they have to install all the firmware again. This is just
> braindead. Their udev script actually checks /lib/firmware first and
> then the kernel versioned directory. So that is just fine.

Well, what we're doing is including the firmware blobs inside the kernel
packages. So, yeah, we install all the firmware blobs again, but they're
installed with the modules that we're installing all over again anyway.
All the firmware blobs currently included in 2.6.27-rcs total about
540k, so the wasted disk space is largely negligible when compared to
the size of installing an additional kernel and module set.

We check /lib/firmware/$(uname -r) /lib/firmware
/usr/local/lib/firmware, in that order.

> Problem comes when installing let say 2.6.27 since then the firmware
> will be looked up in /lib/firmware/ and /lib/firmware/2.6.27/ and
> actually it will not be found. Since it is in /lib/firmware/2.6.26-xx/
> or something similar.

In what case? After you install a new kernel and then reboot? You have
the same problem with the modules that need the firmware blobs.

> So having the kernel install everything in /lib/firmware works just fine
> with every distro. However looking for firmware that is not shipped with
> the kernel, we have a problem since Debian/Ubuntu just not installs it
> in the right directory. And there is nothing the kernel can do about it
> since it will not touch firmware it doesn't ship. The distro has to fix
> their firmware or the users have to place a copy in /lib/firmware/ where
> it actually should have been in the first place.

It works just fine until the user wants to install an additional, newer,
kernel and the blob has changed but the filename hasn't.

- -Jeff


- --
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iEYEARECAAYFAkjI0z8ACgkQLPWxlyuTD7LPkQCeOGxTOsUivMmjabTrkHd7ssJ3
OBIAnidEMgduuQ4WCXN8JSEhJ4Gij7Iv
=3Lng
-----END PGP SIGNATURE-----

Re: [PATCH] firmware: Allow release-specific firmware dir

[Faidon Liambotis]

Marcel Holtmann wrote:
> it is that the Debian maintainer screwed this up. Upstream never
> promoted kernel versioned directories for the firmware. I had a long
> discussion with Dave about it at OLS and using the kernel version is
> just plain wrong. The driver maintainers should version the firmware if
> they break it in an API incompatible way.
> 
> To shed some light into the problem with Debian/Ubuntu. They install the
> firmware in /lib/firmware/`uname -r`/ and everytime they bump their ABI
> number, they have to install all the firmware again. This is just
> braindead. Their udev script actually checks /lib/firmware first and
> then the kernel versioned directory. So that is just fine.
I could not find any packages in Debian that contain a
/lib/firmware/$(KERNELRELEASE) directory.

Perhaps this is just an Ubuntu issue?

Regards,
Faidon

Re: [PATCH] firmware: Allow release-specific firmware dir

[Frans Pop]

[-- Attachment #1: Type: text/plain, Size: 8248 bytes --]

David Woodhouse wrote:
> On Wed, 2008-09-10 at 16:05 -0700, David Miller wrote:
>> Tell that to every Debian and Debian derived system on the planet.
>> 
>> To my knowledge, it is only fedora and possibly one or two other dists
>> that put the firmware files in a unary /lib/firmware location, rather
>> than a versioned /lib/firmware/$KERNELRELASE one.

I have to correct David M. here: *official* Debian packages _do_ have 
firmware in /lib/firmware and not in versioned subdirs.

The problem for Debian users is NOT official Debian kernel packages.
The problem is for people, like me, building their own kernels directly 
from upstream source *in the form of Debian packages*. More about this 
later.

It is also about the firmware change breaking *existing* tools used to 
this and *that* has always been the main argument from people like David 
M., Jeff Garzik and me. More about this in my second mail.

> It sounds like a daft arrangement to me -- although of course it's
> possible for the packager to do whatever they like, by overriding
> $(INSTALL_FW_PATH) in their package build.
> 
> It's definitely not something we should be doing upstream though.

David, you clearly fail to see the problem and it seems to me that you 
don't know the first thing about package management.

First let me get something off my chest, and sorry for shouting:
                 !!! THIS IS NOT ABOUT DEBIAN !!!

This has never been about Debian. It is about package management and 
backwards compatibility with existing tools be it package creation, 
package management or udev.

Let me start with a statement about package management.
    Software packages should not overwrite files installed by other
    software packages. Any package management system that allows a
    package to do that without complaining loudly is fundamentally
    broken.

This is a hard requirement in Debian. It is also the main reason why there 
are systems running Debian that have been installed 10 years ago, have 
been upgraded with each new Debian release (i.e. have never been 
reinstalled from scratch) and are still "clean" (without any file or 
library conflicts). In most cases Debian even allows you to _downgrade_ 
packages without any problem.

This is the "raison the existance" for distributions: to ensure software 
from different sources can be co-installed without conflicts. It also 
creates some limitations how packages are to be put together.

Problem description
===================
Premise: different kernel versions should be co-installable.

Consequence of this that different kernel versions will be different 
packages, not versions of the same package.

Example (simplified)
--------------------
Here is how *existing* packaging tools will create Debian kernel packages 
from upstream source for a few different versions.

linux-image-2.6.25_1_i386.deb contains:
/boot/vmlinuz-2.6.25-1
/lib/modules/2.6.25-1/...

linux-image-2.6.26_1_i386.deb contains:
/boot/vmlinuz-2.6.26-1
/lib/modules/2.6.26-1/...

linux-image-2.6.27-rc1_1_i386.deb contains:
/boot/vmlinuz-2.6.27-rc1
/lib/modules/2.6.27-rc1/...
/lib/firmware/firmware.dat

linux-image-2.6.27-rc1_2_i386.deb contains:
/boot/vmlinuz-2.6.27-rc1
/lib/modules/2.6.27-rc1/...
/lib/firmware/firmware.dat

linux-image-2.6.27-rc2_1_i386.deb contains:
/boot/vmlinuz-2.6.27-rc2
/lib/modules/2.6.27-rc2/...
/lib/firmware/firmware.dat

The package name consists of 4 parts; taking the first as example:
- linux-image-2.6.25 : package name, includes kernel version
- 1	: package version
- i386	: architecture
- deb	: extention

It is clear to see that 2.6.25_1, 2.6.26_1 and even 2.6.27-rc1_1 can be 
co-installed without problem: no file conflicts.
It is even possible to upgrade from 2.6.27-rc1_1 to 2.6.27-rc1_2 without 
problem: as the package *name* is the same the package manager will allow 
the firmware.dat file to be overwritten by the new version.

However, the package manager will complain loudly when you try to install 
2.6.27-rc2_1. Attached is a real life example from my own system when I 
try to install rc6 alongside rc5. The most relevant bits are:
<snip>
# dpkg -i linux-2.6.27-rc6_8.g98df367_amd64.deb
Unpacking linux-2.6.27-rc6 dpkg:
 error processing ../linux-2.6.27-rc6_8.g98df367_amd64.deb (--install):
 trying to overwrite `/lib/firmware/kaweth/new_code.bin',
 which is also in package linux-2.6.27-rc5
dpkg-deb: subprocess paste killed by signal (Broken pipe)
Errors were encountered while processing:
 linux-2.6.27-rc6_8.g98df367_amd64.deb
</snip>

The attachment also shows how I've been working around this:
<snip>
# dpkg -i --force-overwrite linux-2.6.27-rc6_8.g98df367_amd64.deb
Unpacking linux-2.6.27-rc6 dpkg - warning, overriding problem 
because --force enabled:
 trying to overwrite `/lib/firmware/kaweth/new_code.bin',
 which is also in package linux-2.6.27-rc5
</snip>
The '--force-overwrite' option disables the "duplicate files" check.

IMPORTANT: the packages in this case were *not* built using some vague 
Debian tool; they were built using the *in tree* option to build a Debian 
package (see scripts/package/builddeb):
$ fakeroot make -j4 deb-pkg

So the current kernel source builds fundamentally broken Debian packages!

Possible solutions
==================
There are two possible solutions to this.

1) Move the firmware files to a versioned directory (avoid the conflict)
This would give:

linux-image-2.6.27-rc1_1_i386.deb contains:
/boot/vmlinuz-2.6.27-rc1
/lib/modules/2.6.27-rc1/...
/lib/firmware/2.6.27-rc1/firmware.dat

linux-image-2.6.27-rc2_1_i386.deb contains:
/boot/vmlinuz-2.6.27-rc2
/lib/modules/2.6.27-rc2/...
/lib/firmware/2.6.27-rc2/firmware.dat

2) Move the firmware files into a separate package
This would give:

linux-image-2.6.27-rc1_1_i386.deb contains:
/boot/vmlinuz-2.6.27-rc1
/lib/modules/2.6.27-rc1/...

linux-firmware_2.6.27-rc1.1_i386.deb contains:
/lib/firmware/firmware.dat

linux-image-2.6.27-rc2_1_i386.deb contains:
/boot/vmlinuz-2.6.27-rc2
/lib/modules/2.6.27-rc2/...

linux-firmware_2.6.27-rc2.1_i386.deb contains:
/lib/firmware/firmware.dat

Note that the kernel version is now part of the package _version_ instead 
of the package _name_. The linux-firmware package can now be upgraded 
from version 2.6.27-rc1.1 to 2.6.27-rc2.1.

Analysis of the solutions
=========================
Oops, seems like we have a problem here.

As you rightly say, option 1) is not acceptable as it is not supported by 
e.g. older udev versions.
And option 2) is not acceptable because *existing* tools to build custom 
(!) kernels just do not split out firmware into a separate package like 
that. Why? Because until 2.6.27 they just did not need to!

And note that option 2) is almost certainly what will be used for 
*official* Debian packages. As others have already shown that method is 
*already* used by Debian for various firmware files, both from out of 
tree sources and split-off from in-tree source.

So the problem is basically that this issue cannot be resolved: existing 
tools will always break one way or the other. The *only* solution that 
allows users to build and run custom kernels using existing tools is to 
provide a compatibility option to build firmware into the kernel AND into 
modules, exactly as has been argued in the previous discussion.

Question to Jeff Garzik
-----------------------
Hi Jeff. You said you were planning on implementing an option to build 
firmware into modules. Care to give an update on that?

FAQ
===
Q: So why use Debian packages at all for custom kernels?
A: Essentially because it helps ensure my system remains clean while
   building, testing and especially removing multiple kernel versions.
   It also simplifies things like creating the initrd and updating the
   bootloader menu and having "upstream" kernels co-installed with
   official Debian kernels.

Q: So why did you not submit a patch to "fix" 'make deb-pkg'?
A: I had a look at implementing solution 1) for deb-pkg at some point,
   but the make complexity defeated my skills. I could not get the
   firmware location to change. Implementing solution 2) is even more
   complex.
   I would be happy to work with others to implement this though.

Cheers,
FJP

P.S. I am not a member of the Debian kernel team.


[-- Attachment #2: firmware.madness --]
[-- Type: text/plain, Size: 2624 bytes --]

$ sudo dpkg -i ../linux-2.6.27-rc6_8.g98df367_amd64.deb
(Reading database ... 164011 files and directories currently installed.)
Unpacking linux-2.6.27-rc6 (from .../linux-2.6.27-rc6_8.g98df367_amd64.deb) ...
dpkg: error processing ../linux-2.6.27-rc6_8.g98df367_amd64.deb (--install):
 trying to overwrite `/lib/firmware/kaweth/new_code.bin', which is also in package linux-2.6.27-rc5
dpkg-deb: subprocess paste killed by signal (Broken pipe)
arguments: 2.6.27-rc6 abort-install
Searching for GRUB installation directory ... found: /boot/grub
Searching for default file ... found: /boot/grub/default
Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
Searching for splash image ... none found, skipping ...
Found kernel: /vmlinuz-2.6.27-rc5
Found kernel: /vmlinuz-2.6.27-rc4
Found kernel: /vmlinuz-2.6.26.3
Found kernel: /vmlinuz-2.6.26-1-amd64
Found kernel: /vmlinuz-2.6.25-2-amd64
Found kernel: /vmlinuz-2.6.18-4-amd64
Updating /boot/grub/menu.lst ... done

Errors were encountered while processing:
 ../linux-2.6.27-rc6_8.g98df367_amd64.deb

$ sudo dpkg -i --force-overwrite ../linux-2.6.27-rc6_8.g98df367_amd64.deb
(Reading database ... 164011 files and directories currently installed.)
Unpacking linux-2.6.27-rc6 (from .../linux-2.6.27-rc6_8.g98df367_amd64.deb) ...
dpkg - warning, overriding problem because --force enabled:
 trying to overwrite `/lib/firmware/kaweth/new_code.bin', which is also in package linux-2.6.27-rc5
dpkg - warning, overriding problem because --force enabled:
 trying to overwrite `/lib/firmware/kaweth/trigger_code.bin', which is also in package linux-2.6.27-rc5
dpkg - warning, overriding problem because --force enabled:
 trying to overwrite `/lib/firmware/kaweth/new_code_fix.bin', which is also in package linux-2.6.27-rc5
dpkg - warning, overriding problem because --force enabled:
 trying to overwrite `/lib/firmware/kaweth/trigger_code_fix.bin', which is also in package linux-2.6.27-rc5
Setting up linux-2.6.27-rc6 (8.g98df367) ...
arguments: 2.6.27-rc6 configure
update-initramfs: Generating /boot/initrd.img-2.6.27-rc6
Searching for GRUB installation directory ... found: /boot/grub
Searching for default file ... found: /boot/grub/default
Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
Searching for splash image ... none found, skipping ...
Found kernel: /vmlinuz-2.6.27-rc6
Found kernel: /vmlinuz-2.6.27-rc5
Found kernel: /vmlinuz-2.6.27-rc4
Found kernel: /vmlinuz-2.6.26.3
Found kernel: /vmlinuz-2.6.26-1-amd64
Found kernel: /vmlinuz-2.6.25-2-amd64
Found kernel: /vmlinuz-2.6.18-4-amd64
Updating /boot/grub/menu.lst ... done

Re: [PATCH] firmware: Allow release-specific firmware dir

[Frans Pop]

Marcel Holtmann wrote:
> it is that the Debian maintainer screwed this up.

WRONG! NOT TRUE! FUD!
The Debian maintainer has done nothing of the kind.

> To shed some light into the problem with Debian/Ubuntu. They install the
> firmware in /lib/firmware/`uname -r`/

Debian does no such thing! It may be that Ubuntu does this, but that does 
NOT mean that Debian does too.

Feel free to examine any of the packages here and check where they install 
their firmware files:
http://packages.debian.org/search?suite=sid&searchon=names&keywords=firmware

Examples:
http://packages.debian.org/sid/all/firmware-qlogic/filelist
http://packages.debian.org/sid/all/firmware-bnx2/filelist

> Problem comes when installing let say 2.6.27 since then the firmware
> will be looked up in /lib/firmware/ and /lib/firmware/2.6.27/ and
> actually it will not be found. Since it is in /lib/firmware/2.6.26-xx/
> or something similar.
> 
> So having the kernel install everything in /lib/firmware works just fine
> with every distro. However looking for firmware that is not shipped with
> the kernel, we have a problem since Debian/Ubuntu just not installs it
> in the right directory. And there is nothing the kernel can do about it
> since it will not touch firmware it doesn't ship. The distro has to fix
> their firmware or the users have to place a copy in /lib/firmware/ where
> it actually should have been in the first place.

This is a ridiculously simplified view of the issue. Why can't you at 
least acknowledge that people were faced with a real problem _without_ 
proper guidance, support or transition path from their upstream source 
and made their best effort to solve that issue.

Note that this thread was started by someone from Suse, not Debian or 
Ubuntu.

Cheers,
FJP

Re: [PATCH] firmware: Allow release-specific firmware dir

[Frans Pop]

On Thursday 11 September 2008, Frans Pop wrote:
> Possible solutions
> ==================
> There are two possible solutions to this.
>
> 1) Move the firmware files to a versioned directory
[...]
> 2) Move the firmware files into a separate package

As mentioned in my previous mail Debian currently uses 2) for its kernel 
packages up to and including 2.6.26. I expect the Debian kernel team to 
also use 2) for 2.6.27, but that's in the future. Debian is currently 
concentrating on 2.6.26 as that's the kernel targeted for the soon-to-be- 
released Lenny release.

>From what I understand from this list (I'm a Debian person and don't 
really follow Ubuntu), Ubuntu seems to have chosen 1).

There are pros and cons to both these solutions and given the different 
philosophies of Debian and Ubuntu those do help explain the different 
choices.

Solution 1:
a) keeps kernel, modules and firmware firmly together
b) ensures there will never be mismatches, even if a firmware file "ABI"
   changes without being properly versioned
c) makes system installation simple: firmware is always just "there"

Solution 2:
a) naturally allows firmware to be split between "free" and "non-free"
b) places a burden on the end-user to install/upgrade the firmware he
   needs separately from the kernel
c) makes system installation more complex as the right additional firmware
   package(s) need to be retrieved and installed
d) will break installed systems on upgrades (unless special measures are
   taken): users may need to install additional firmware packages on a
   kernel upgrade to keep a driver working

Ubuntu has a "things should just work" philosophy. Choosing 1) keeps 
things simple from a user PoV and is a logical choice from that 
perspective.

Debian has strict licencing rules and has never shirked from placing some 
responsibility in the hands of users. Especially 2a) makes 2) the more 
natural choice for Debian.


Note that with 2) you cannot just create one big "linux-firmware" package. 
You will need to package all firmware separately which means that users 
will need to somehow select the specific firmware packages they need for 
their hardware.
The reason for this is the possibility that a driver and its firmware may 
be dropped from the upstream kernel source.

Example
-------
linux-image-2.6.27_1_i386.deb contains:
/boot/vmlinuz-2.6.27
/lib/modules/2.6.27/kernel/net/a.ko
/lib/modules/2.6.27/kernel/net/b.ko

linux-firmware_2.6.27.1_i386.deb contains:
/lib/firmware/a.dat
/lib/firmware/b.dat

For 2.6.28 driver "a" is dropped, which would result in:
linux-image-2.6.28_1_i386.deb contains:
/boot/vmlinuz-2.6.28
/lib/modules/2.6.28/kernel/net/b.ko

linux-firmware_2.6.28.1_i386.deb contains:
/lib/firmware/b.dat

The user installs linux-image-2.6.28 alongside linux-image-2.6.27 and 
upgrades linux-firmware from version 2.6.27.1 to 2.6.28.1.
Oops! The upgrade of the linux-firmware package removed b.dat, which 
effectively means the still installed 2.6.27 kernel is now broken.

To avoid this you need separate packages which can be installed/removed 
independently:
a-firmware_<a-firmware-version>_i386.deb contains:
/lib/firmware/a.dat

b-firmware_<b-firmware-version>_i386.deb contains:
/lib/firmware/b.dat

Re: [PATCH] firmware: Allow release-specific firmware dir

[Thierry Vignaud]

David Miller <davem@davemloft.net> writes:

> > This isn't compatible with the version of udev that people actually have
> > installed, today. And it's also the _wrong_ thing to do.
> > 
> > Firmware really _isn't_ version-specific.
> 
> Tell that to every Debian and Debian derived system on the planet.
> 
> To my knowledge, it is only fedora and possibly one or two other dists
> that put the firmware files in a unary /lib/firmware location, rather
> than a versioned /lib/firmware/$KERNELRELASE one.

We (Mandriva Linux) do too.
Why other distro cannot package firmware separatly for all their kernel
flavors?

Firmware are not version-specific.

Re: [PATCH] firmware: Allow release-specific firmware dir

[Greg KH]

On Wed, Sep 10, 2008 at 03:35:07PM -0700, Andrew Morton wrote:
> On Tue, 09 Sep 2008 10:15:35 -0400
> Jeff Mahoney <jeffm@suse.com> wrote:
> 
> >  Every kernel package trying to provide files under /lib/firmware runs
> >  into problems really quickly with multiple kernels installed, especially
> >  with respect to distribution package tracking of the files.
> 
> For how long has this problem existed?

Since the firmware target got added to the kernel, so it's a 2.6.27
thing, no need for -stable here, older kernels do not have this problem.

thanks,

greg k-h

Re: [PATCH] firmware: Allow release-specific firmware dir

[Theodore Tso]

On Thu, Sep 11, 2008 at 03:15:12AM -0400, Jeff Mahoney wrote:
> 
> In the end, though, Andrew's right. We can't go breaking udev prior to
> 127 with this.

But it's OK to break various distribution's kernel packaging tools?

    	       	     	     		    	   	     - Ted

Re: [PATCH] firmware: Allow release-specific firmware dir

[Greg KH]

On Thu, Sep 11, 2008 at 01:29:46PM +0200, Thierry Vignaud wrote:
> David Miller <davem@davemloft.net> writes:
> 
> > > This isn't compatible with the version of udev that people actually have
> > > installed, today. And it's also the _wrong_ thing to do.
> > > 
> > > Firmware really _isn't_ version-specific.
> > 
> > Tell that to every Debian and Debian derived system on the planet.
> > 
> > To my knowledge, it is only fedora and possibly one or two other dists
> > that put the firmware files in a unary /lib/firmware location, rather
> > than a versioned /lib/firmware/$KERNELRELASE one.
> 
> We (Mandriva Linux) do too.
> Why other distro cannot package firmware separatly for all their kernel
> flavors?
> 
> Firmware are not version-specific.

Yes it is, when it is bundled with the kernel source tree itself.

Why do people not realize this?

This is a real problem for distros, and for anyone who wants to have
multiple kernel versions on a single machine.  Jeff's patch fixes this.

thanks,

greg k-h

Re: [PATCH] firmware: Allow release-specific firmware dir

[Jeff Mahoney]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Theodore Tso wrote:
> On Thu, Sep 11, 2008 at 03:15:12AM -0400, Jeff Mahoney wrote:
>> In the end, though, Andrew's right. We can't go breaking udev prior to
>> 127 with this.
> 
> But it's OK to break various distribution's kernel packaging tools?

No. I still think the right answer is to keep them in a versioned
directory and I'm keeping this patch applied to the SUSE kernel trees.
Udev in the next release understands where to look. It's just clear that
there's some disagreement if that's the right answer for everyone. My
opinion is that if the firmware blobs are built with and shipped with
the kernel, then they're already associated with a particular kernel
version as a dependency the same way we'd treat module dependencies.

It's silly to willfully complicate the situation beyond just including
the firmware blobs with the kernel. Even though it sounds a lot simpler
to just say, "keep a separate firmware package," the reality is that it
creates more work for everyone, developers, users, and packagers alike.
I recently updated the kernel on my notebook to a 2.6.27-rc, only to
find out that the firmware blob that I already had installed was out of
date and the driver was useless without it. This is an example of an
externally-maintained firmware blob, but creating a separate package out
of firmware blobs essentially makes all of them externally maintained
and forces in-kernel developers to jump through the same hoops that
third-party driver maintainers must.

Packagers will need to keep track of every firmware version associated
with every kernel, since users may want to install multiple kernel
versions. This is the entire point of the thread, and it assumes that
the user installing the kernel is even using a packaged kernel. If
they've just built their own and have done a make modules_install ; make
install, then they're out of luck.

Driver maintainers need to worry about remembering to up the version
every time the firmware blob changes. If the blobs are always kept with
the associated kernel version, then it's fire and forget, the same way
internal API changes are.

Putting them in their own directory just makes it obvious and easy for
everyone. The issue is that firmware.sh isn't looking there right now.
Perhaps making optional symlinks is the answer.

- -Jeff

- --
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iEYEARECAAYFAkjJLP4ACgkQLPWxlyuTD7I5aQCbBRYn5wv3GbLw1cxUsDr/PxuC
ml0An2WjJwkYEz4hQadUi0PosZe6n6ng
=rY53
-----END PGP SIGNATURE-----

Re: [PATCH] firmware: Allow release-specific firmware dir

[David Woodhouse]

On Thu, 2008-09-11 at 11:52 +0200, Frans Pop wrote:
> Solution 2:
(separate firmware package)

This is what we're doing in Fedora, and what I think makes most sense.

At the moment, Fedora's package is still taken from the kernel source --
but fairly shortly, we'll move to building it from the linux-firmware
repository on kernel.org. That repository contains everything that's in
the kernel tree, and more -- now that we have it completely separate,
more companies are willing to allow us to include their firmware. We've
already rounded up some which were previously separate, and we'll
shortly be adding more firmware files which weren't previously available
to Linux users at all, except by copying it from the Windows installer.


> a) naturally allows firmware to be split between "free" and "non-free"

Indeed. So it allows the 'free software or nothing' nutters to have
their own replacement firmware package with none of the nasty stuff they
don't like.

> b) places a burden on the end-user to install/upgrade the firmware he
>    needs separately from the kernel

Not really. The new kernel package can just require the firmware
package. There's not really much of a burden on the end-user, is there?

> c) makes system installation more complex as the right additional firmware
>    package(s) need to be retrieved and installed

It's a simple dependency, handled just like dependencies normally are --
it's not exactly a giant leap in complexity.

> d) will break installed systems on upgrades (unless special measures are
>    taken): users may need to install additional firmware packages on a
>    kernel upgrade to keep a driver working

I have yet to see reports of it breaking Fedora. You upgrade the kernel,
the firmware package is automatically installed to satisfy the
dependency. And it just works.

Really, people seem to be imagining problems where they don't exist. I
don't want to just dismiss real problems -- but I don't _see_ real
problems either. Just people like Dave screaming "you broke it for real
users, like Herbert"... and Herbert saying "er, no you didn't really."
So I have to take it with a pinch of salt.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

Re: [PATCH] firmware: Allow release-specific firmware dir

[Frans Pop]

On Thursday 11 September 2008, David Woodhouse wrote:
> > b) places a burden on the end-user to install/upgrade the firmware he
> >    needs separately from the kernel
>
> Not really. The new kernel package can just require the firmware
> package. There's not really much of a burden on the end-user, is there?

Did you read the second part of the mail explaining why having a 
single "firmware" package is not a good idea?
Also, if some firmware is assigned to non-free in Debian the kernel 
package CANNOT depend on it.

> > c) makes system installation more complex as the right additional
> > firmware package(s) need to be retrieved and installed
>
> It's a simple dependency, handled just like dependencies normally are
> -- it's not exactly a giant leap in complexity.

Nope, see above.

> > d) will break installed systems on upgrades (unless special measures
> > are taken): users may need to install additional firmware packages on
> > a kernel upgrade to keep a driver working
>
> I have yet to see reports of it breaking Fedora. You upgrade the
> kernel, the firmware package is automatically installed to satisfy the
> dependency. And it just works.

Nope, see above.

> Really, people seem to be imagining problems where they don't exist.

You have completely ignored my point in my other mail about 'make deb-pkg' 
being demonstrably broken as it results in packages that are guaranteed 
to produce file conflicts.
You have completely ignored my point about other existing packaging 
utilities being similarly broken.

Let me repeat again:
            THIS IS NOT ABOUT THE OFFICIAL DEBIAN PACKAGES

The Debian kernel team _will_ find a way to make this work properly. As 
far as separate firmware packages and upgrading is an issue the Debian 
project will deal with it, maybe using a userland "firmware-installer" 
tool or even if it is just by documenting the need to install firmware 
packages in the Release Notes.

This is about users, LIKE ME, being unable *NOW* to build correct custom 
kernel packages from upstream source *using existing tools*.
I am confronted with the broken packages as a result of your changes 
*every time I build/install a kernel* currently.

As you can see in other mail I'm currently working around it by using an 
extra option when installing the packages, but it's an option I should 
not have to use! And I'll probably change to just not building any 
drivers that need firmware at all as luckily none of my hardware needs 
in-tree firmware.

Sorry for all the emphasis, but it looks like you need it.
Your comments about how everything works for official Fedora package are 
completely irrelevant. Please try to look a bit further than your own 
narrow use-case.

I'm seriously disappointed with your reply.

Cheers,
FJP

Re: [PATCH] firmware: Allow release-specific firmware dir

[David Woodhouse]

On Thu, 2008-09-11 at 17:24 +0200, Frans Pop wrote:
> Did you read the second part of the mail explaining why having a 
> single "firmware" package is not a good idea?

Yes, but it was nonsense, so I ignored it. "The reason for this is the
possibility that a driver and its firmware may be dropped from the
upstream kernel source." That doesn't happen very often, and you
_certainly_ don't need to immediately drop the firmware if it does.

It's another reason why it's better to be shipping from the external
linux-firmware repository at
git://git.kernel.org/pub/scm/linux/kernel/git/dwmw2/linux-firmware.git
-- because if a driver _does_ go away, that repository would continue to
carry the firmware files. But since that's such an infrequent
occurrence, there's no massive rush. Using the firmware in the kernel
tree is fine for now.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

Re: [PATCH] firmware: Allow release-specific firmware dir

[Frans Pop]

On Thursday 11 September 2008, David Woodhouse wrote:
> On Thu, 2008-09-11 at 17:24 +0200, Frans Pop wrote:
> > Did you read the second part of the mail explaining why having a
> > single "firmware" package is not a good idea?
>
> Yes, but it was nonsense, so I ignored it. "The reason for this is the
> possibility that a driver and its firmware may be dropped from the
> upstream kernel source." That doesn't happen very often, and you
> _certainly_ don't need to immediately drop the firmware if it does.

Fine. Again you are concentrating on the bright and shiny future and 
magically wishing into existence tools to support the new situation.

Just keep on ignoring current issues. As I've said, I can work around 
them. However, I seriously expect others will have stopped their kernel 
testing and I also expect "real users" to run into problems building 
custom kernels.

I think I've made my point and will (try to) keep out of the discussion. 
It's way too much like discussing religion anyway: people who "believe" 
are rarely able to listen properly.

Re: [PATCH] firmware: Allow release-specific firmware dir

[Linus Torvalds]

On Thu, 11 Sep 2008, Frans Pop wrote:
> 
> Just keep on ignoring current issues.

Aren't _you_ the one who are ignoring current issues?

The fact is, _currently_ everybody looks in /lib/firmware. The fact is, 
most people don't want millions of copies of the firmware (one copy for 
each kernel they compile - think about those of us who compile kernels 
every day).

I actually like a release-specific firmware directory, but I think it's 
the *vendor* kernel that should do it, not the normal kernel. The _vendor_ 
should put its firmware files in some vendor-specific place, and then add 
that to the end of the firmware loading path. 

Notice? The vendor can do that. The generic kernel can NOT - since the 
generic kernel cannot impact the firmware loading paths of all the various 
distros. 

So all of the noise in this whole discussion has been totally inane and 
idiotic. 

		Linus

Re: [PATCH] firmware: Allow release-specific firmware dir

[David Woodhouse]

On Thu, 2008-09-11 at 17:49 +0200, Frans Pop wrote:
> On Thursday 11 September 2008, David Woodhouse wrote:
> > On Thu, 2008-09-11 at 17:24 +0200, Frans Pop wrote:
> > > Did you read the second part of the mail explaining why having a
> > > single "firmware" package is not a good idea?
> >
> > Yes, but it was nonsense, so I ignored it. "The reason for this is the
> > possibility that a driver and its firmware may be dropped from the
> > upstream kernel source." That doesn't happen very often, and you
> > _certainly_ don't need to immediately drop the firmware if it does.
> 
> Fine. Again you are concentrating on the bright and shiny future and 
> magically wishing into existence tools to support the new situation.

No, the 'new' situation is just like the situation we've had for years
for most recent drivers anyway.

You've pulled this magic requirement that we split up firmware into
separate packages out of a hat, and it's nonsense. Please go away and
stop spreading FUD. If there _are_ real issues to deal with, I'd like to
deal with them. I believe I already have. But we'd be better off without
the pointless noise that some people are creating.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

Re: [PATCH] firmware: Allow release-specific firmware dir

[Marcel Holtmann]

Hi Jeff,

> >>> It's definitely not something we should be doing upstream though.
> >> So you think it's ok that every Debian user has to learn this
> >> magic incantation just to use current kernels?
> >>
> >> I don't think it's nice to break things like this on people,
> >> especially such a large group.  Getting this stuff to work is hard
> >> enough, and we're just putting yet another barrier into the situation
> >> and that can only mean less testers and contributors.
> >>
> >> I do know several people who aren't testing and contributing because
> >> the whole firmware shakeup is so bolixed and they really are exasperated
> >> after spending hours trying to get it to work.
> > 
> > it is that the Debian maintainer screwed this up. Upstream never
> > promoted kernel versioned directories for the firmware. I had a long
> > discussion with Dave about it at OLS and using the kernel version is
> > just plain wrong. The driver maintainers should version the firmware if
> > they break it in an API incompatible way.
> 
> They "should," but is that happening now? Out of all the firmware blobs
> installed with 2.6.27-rcX, only the edgeport drier actually versions
> them - and they're versioned because the driver requires different
> versions for different hardware.

let me repeat this, if a driver depends on newer firmware version, then
it should make sure that the firmware itself is versioned. Everything
else will break eventually.

The Intel wireless drivers started to version their firmware a long time
ago. That is the way to go. How will a user tell different firmwares
apart otherwise.

> > To shed some light into the problem with Debian/Ubuntu. They install the
> > firmware in /lib/firmware/`uname -r`/ and everytime they bump their ABI
> > number, they have to install all the firmware again. This is just
> > braindead. Their udev script actually checks /lib/firmware first and
> > then the kernel versioned directory. So that is just fine.
> 
> Well, what we're doing is including the firmware blobs inside the kernel
> packages. So, yeah, we install all the firmware blobs again, but they're
> installed with the modules that we're installing all over again anyway.
> All the firmware blobs currently included in 2.6.27-rcs total about
> 540k, so the wasted disk space is largely negligible when compared to
> the size of installing an additional kernel and module set.
> 
> We check /lib/firmware/$(uname -r) /lib/firmware
> /usr/local/lib/firmware, in that order.

Checking /lib/firmware/$(uname -r) is not the issue. The issue is that
when installing a self-compiled kernel, the firmware is not present
in /lib/firmware.

> > Problem comes when installing let say 2.6.27 since then the firmware
> > will be looked up in /lib/firmware/ and /lib/firmware/2.6.27/ and
> > actually it will not be found. Since it is in /lib/firmware/2.6.26-xx/
> > or something similar.
> 
> In what case? After you install a new kernel and then reboot? You have
> the same problem with the modules that need the firmware blobs.

If the firmware is not in /lib/firmware (especially external firmware)
it will not be found and the module will be useless if it depends on
that firmware. And the biggest part of the firmware is not distributed
with the kernel because of GPL restrictions. So whatever we decide for
the kernel right now helps nothing if the user needs external firmware
for their devices.

Just take the Intel wireless drivers as an example. If I boot my own
kernel, I have to make sure I either copied or linked them from the
distro directory to /lib/firmware or downloaded them. Whatever the
kernel firmware prefix is makes no difference here.

> > So having the kernel install everything in /lib/firmware works just fine
> > with every distro. However looking for firmware that is not shipped with
> > the kernel, we have a problem since Debian/Ubuntu just not installs it
> > in the right directory. And there is nothing the kernel can do about it
> > since it will not touch firmware it doesn't ship. The distro has to fix
> > their firmware or the users have to place a copy in /lib/firmware/ where
> > it actually should have been in the first place.
> 
> It works just fine until the user wants to install an additional, newer,
> kernel and the blob has changed but the filename hasn't.

That is the whole point and Dave and I discussed this deeply. If the
blob changes in an incompatible way or if a driver needs a newer
firmware, the firmware filename should change.

Regards

Marcel

Re: [PATCH] firmware: Allow release-specific firmware dir

[Marcel Holtmann]

Hi Faidon,

> > it is that the Debian maintainer screwed this up. Upstream never
> > promoted kernel versioned directories for the firmware. I had a long
> > discussion with Dave about it at OLS and using the kernel version is
> > just plain wrong. The driver maintainers should version the firmware if
> > they break it in an API incompatible way.
> > 
> > To shed some light into the problem with Debian/Ubuntu. They install the
> > firmware in /lib/firmware/`uname -r`/ and everytime they bump their ABI
> > number, they have to install all the firmware again. This is just
> > braindead. Their udev script actually checks /lib/firmware first and
> > then the kernel versioned directory. So that is just fine.
> I could not find any packages in Debian that contain a
> /lib/firmware/$(KERNELRELEASE) directory.
> 
> Perhaps this is just an Ubuntu issue?

I looked through my Debian boxes and you are right. Sorry for this. I
was under the assumption Ubuntu took the Debian package, but it seems
that they even modified that one. So this issue is than distros derived
from Ubuntu and _not_ Debian (unless udev is unmodified).

Regards

Marcel

Re: [PATCH] firmware: Allow release-specific firmware dir

[Marcel Holtmann]

Hi Frans,

> > it is that the Debian maintainer screwed this up.
> 
> WRONG! NOT TRUE! FUD!
> The Debian maintainer has done nothing of the kind.
> 
> > To shed some light into the problem with Debian/Ubuntu. They install the
> > firmware in /lib/firmware/`uname -r`/
> 
> Debian does no such thing! It may be that Ubuntu does this, but that does 
> NOT mean that Debian does too.

I apologize for this. You are absolute right. Debian does the correct
thing and only Ubuntu made this a problem.

> > Problem comes when installing let say 2.6.27 since then the firmware
> > will be looked up in /lib/firmware/ and /lib/firmware/2.6.27/ and
> > actually it will not be found. Since it is in /lib/firmware/2.6.26-xx/
> > or something similar.
> > 
> > So having the kernel install everything in /lib/firmware works just fine
> > with every distro. However looking for firmware that is not shipped with
> > the kernel, we have a problem since Debian/Ubuntu just not installs it
> > in the right directory. And there is nothing the kernel can do about it
> > since it will not touch firmware it doesn't ship. The distro has to fix
> > their firmware or the users have to place a copy in /lib/firmware/ where
> > it actually should have been in the first place.
> 
> This is a ridiculously simplified view of the issue. Why can't you at 
> least acknowledge that people were faced with a real problem _without_ 
> proper guidance, support or transition path from their upstream source 
> and made their best effort to solve that issue.

That is just plain naive on the package maintainer side. If the package
maintainer has to solve problems, then contacting upstream should be the
first thing to do. I might be wrong here, but I don't recall any
requests like this on LKML or linux-hotplug. They just did it their way
without contacting either kernel guys or udev people.

Regards

Marcel

Re: [PATCH] firmware: Allow release-specific firmware dir

[Marcel Holtmann]

Hi Faidon,

> >> Firmware really _isn't_ version-specific.
> > 
> > Tell that to every Debian and Debian derived system on the planet.
> > 
> > To my knowledge, it is only fedora and possibly one or two other dists
> > that put the firmware files in a unary /lib/firmware location, rather
> > than a versioned /lib/firmware/$KERNELRELASE one.
> [not speaking on behalf of the project, the kernel team or the
> respective maintainers]
> 
> Apparently and afaik you are misinformed:
> 
> $ dpkg -L firmware-bnx2 (or [1])
> ...
> /lib/firmware/bnx2-06-4.0.5.fw
> /lib/firmware/bnx2-09-4.0.5.fw
> ...
> 
> $ dpkg -L firmware-iwlwifi (or [2])
> ...
> /lib/firmware/iwlwifi-3945-1.ucode
> /lib/firmware/iwlwifi-4965-1.ucode
> /lib/firmware/iwlwifi-4965-2.ucode
> ...

that is different. It is the version of the firmware. It doesn't depend
on the version of the Linux kernel. Yes, there is a tiny dependency on
the kernel, but that comes from the driver requirements and we would
have MODULE_FIRMWARE to clearly indicate which drivers needs which
firmware.

Tying it to the kernel version is wrong. It depends on the version of
the driver.

Regards

Marcel

Re: [PATCH] firmware: Allow release-specific firmware dir

[Marcel Holtmann]

Hi Jeff,

> >> In the end, though, Andrew's right. We can't go breaking udev prior to
> >> 127 with this.
> > 
> > But it's OK to break various distribution's kernel packaging tools?
> 
> No. I still think the right answer is to keep them in a versioned
> directory and I'm keeping this patch applied to the SUSE kernel trees.
> Udev in the next release understands where to look. It's just clear that
> there's some disagreement if that's the right answer for everyone. My
> opinion is that if the firmware blobs are built with and shipped with
> the kernel, then they're already associated with a particular kernel
> version as a dependency the same way we'd treat module dependencies.

no they don't depend on a kernel version. The firmware depends on a
specific driver version (and hardware in some cases). The kernel version
is totally irrelevant in this case.

> It's silly to willfully complicate the situation beyond just including
> the firmware blobs with the kernel. Even though it sounds a lot simpler
> to just say, "keep a separate firmware package," the reality is that it
> creates more work for everyone, developers, users, and packagers alike.
> I recently updated the kernel on my notebook to a 2.6.27-rc, only to
> find out that the firmware blob that I already had installed was out of
> date and the driver was useless without it. This is an example of an
> externally-maintained firmware blob, but creating a separate package out
> of firmware blobs essentially makes all of them externally maintained
> and forces in-kernel developers to jump through the same hoops that
> third-party driver maintainers must.
> 
> Packagers will need to keep track of every firmware version associated
> with every kernel, since users may want to install multiple kernel
> versions. This is the entire point of the thread, and it assumes that
> the user installing the kernel is even using a packaged kernel. If
> they've just built their own and have done a make modules_install ; make
> install, then they're out of luck.

We do have MODULE_FIRMWARE for that.

> Putting them in their own directory just makes it obvious and easy for
> everyone. The issue is that firmware.sh isn't looking there right now.
> Perhaps making optional symlinks is the answer.

Making symlinks is just a workaround. It will not solve anything in the
long term.

Getting all the firmware out of the kernel and getting it into a
separate package is the solution. And to avoid any breakage or problems
the firmware files itself have to be versioned. Firmware like every
other software has its version numbers already. It is like any other API
and once you break it in an incompatible why, you should increase the
major version number. The kernel version has nothing to do with it.

Regards

Marcel

Re: [PATCH] firmware: Allow release-specific firmware dir

[Frans Pop]

On Thursday 11 September 2008, Linus Torvalds wrote:
> On Thu, 11 Sep 2008, Frans Pop wrote:
> > Just keep on ignoring current issues.
>
> Aren't _you_ the one who are ignoring current issues?

I don't think so.
_The_ current issue for me is that I can no longer build Debian kernel 
packages using 'make deb-pkg', which is part of _your_ tree, and install 
the resulting package *NOW*.

> The fact is, _currently_ everybody looks in /lib/firmware. The fact is,
> most people don't want millions of copies of the firmware (one copy for
> each kernel they compile - think about those of us who compile kernels
> every day).

Right, I totally agree. I did about 40 kernel builds the past two days.
(Yes, I know, there are people doing way more builds than that.)
I regularly have 5 kernels installed simultaneously, and sometimes more if 
I'm actively trying to trace an issue.

> I actually like a release-specific firmware directory, but I think it's
> the *vendor* kernel that should do it, not the normal kernel. The
> _vendor_ should put its firmware files in some vendor-specific place,
> and then add that to the end of the firmware loading path.

Hey, I'm NOT propagating versioned dirs in this thread! I've only 
mentioned them as one of the two options and explained why I think it's a 
valid choice in some scenario's.
I've also explained that I don't think Debian will go that way, but that 
is irrelevant. I basically don't care what Debian does for its official 
packages.

> So all of the noise in this whole discussion has been totally inane and
> idiotic.

Please explain what is idiotic about mentioning that 'make deb-pkg' 
produces broken packages solely because of the separate firmware and that 
that is *not* something that is easily solvable because of the way 
package management works?
Please explain why it is idiotic to say that having a compatibility option 
that would allow me to build firmware _inside_ modules, just like we 
always used to, would solve that problem?

That has nothing to do with vendors or distros. It has to do with me (and 
others like me, as a kernel tester, having to live with broken tools NOW.

I like 'make dep-pkg' because it very quickly produces a solid, *single* 
Debian package I can install and use for kernel testing. I'd like to keep 
using that as I have been, but the separation of firmware essentially 
kills my current workflow.

Re: [PATCH] firmware: Allow release-specific firmware dir

[Marcel Holtmann]

Hi Greg,

> > > > This isn't compatible with the version of udev that people actually have
> > > > installed, today. And it's also the _wrong_ thing to do.
> > > > 
> > > > Firmware really _isn't_ version-specific.
> > > 
> > > Tell that to every Debian and Debian derived system on the planet.
> > > 
> > > To my knowledge, it is only fedora and possibly one or two other dists
> > > that put the firmware files in a unary /lib/firmware location, rather
> > > than a versioned /lib/firmware/$KERNELRELASE one.
> > 
> > We (Mandriva Linux) do too.
> > Why other distro cannot package firmware separatly for all their kernel
> > flavors?
> > 
> > Firmware are not version-specific.
> 
> Yes it is, when it is bundled with the kernel source tree itself.
> 
> Why do people not realize this?
> 
> This is a real problem for distros, and for anyone who wants to have
> multiple kernel versions on a single machine.  Jeff's patch fixes this.

once you move the firmware into its own package built from a separate
source tree (and not the kernel) this is no longer a problem.

Nobody ever made the claim that the firmware depends on the kernel
version. It might depend on a specific version of the firmware, but that
is different from the kernel version. If you break the firmware or if a
driver requires a specific firmware version it should indicate this.

And yes, the in-kernel firmware versions should follow here at some
point, since the external firmware versions had to do it already for
quite some time, because otherwise more people would have complained
loudly when installing their own kernels.

What is wrong with moving the firmware into its own package with its own
version number?

Regards

Marcel

Re: [PATCH] firmware: Allow release-specific firmware dir

[David Woodhouse]

On Thu, 2008-09-11 at 18:39 +0200, Marcel Holtmann wrote:
> What is wrong with moving the firmware into its own package with its
> own version number?

Nothing. You can do this today, building from the linux-firmware
repository. That repository is starting to contain extra firmware that
vendors are now willing to let us distribute -- where they _weren't_
willing to let us distribute it as part of the kernel.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

Re: [PATCH] firmware: Allow release-specific firmware dir

[Linus Torvalds]

On Thu, 11 Sep 2008, Frans Pop wrote:
> >
> > Aren't _you_ the one who are ignoring current issues?
> 
> I don't think so.
> _The_ current issue for me is that I can no longer build Debian kernel 
> packages using 'make deb-pkg', which is part of _your_ tree, and install 
> the resulting package *NOW*.

Well, complain to the right people then. The patch that started this all 
is wrong. The people you are complaining to are not the rigth people. We 
can't do anything about debian packaging.

		Linus

Re: [PATCH] firmware: Allow release-specific firmware dir

[Frans Pop]

On Thursday 11 September 2008, Linus Torvalds wrote:
> On Thu, 11 Sep 2008, Frans Pop wrote:
> > > Aren't _you_ the one who are ignoring current issues?
> >
> > I don't think so.
> > _The_ current issue for me is that I can no longer build Debian
> > kernel packages using 'make deb-pkg', which is part of _your_ tree,
> > and install the resulting package *NOW*.
>
> Well, complain to the right people then. The patch that started this
> all is wrong.

I totally agree with that.

What I would still like to see instead is the *option* to just include 
firmware inside modules, where it's always been.

Either that, or stop building firmware as part of the kernel build process 
altogether (which seems to be David's vision of the future).

> The people you are complaining to are not the rigth people. We can't do
> anything about debian packaging. 

This isn't about Debian packaging or Debian tools. I'm not using a Debian 
tool [1]! Why is that so hard to understand?

I'm using an option in *your* kernel tree that produces a broken Debian 
package. Please have a look in scripts/package/builddeb.
That offers the option to just do 'make deb-pkg' to build a kernel package 
in Debian package format.

The other part of my complaint is that this simple example proves that the 
firmware split *has* broken existing tools while you and others have been 
maintaining that everything would continue to work fine "because everyone 
has been using /lib/firmware for ages". The reason things have broken 
because we never used to generate files as a basically inseparable part 
of a kernel build before.


[1] I abandoned make-kpkg about a year ago because it was terminally 
insane and broken. AFAIK it still does not support the unified x86 dir...

Re: [PATCH] firmware: Allow release-specific firmware dir

[Greg KH]

On Thu, Sep 11, 2008 at 06:39:25PM +0200, Marcel Holtmann wrote:
> Hi Greg,
> 
> > > > > This isn't compatible with the version of udev that people actually have
> > > > > installed, today. And it's also the _wrong_ thing to do.
> > > > > 
> > > > > Firmware really _isn't_ version-specific.
> > > > 
> > > > Tell that to every Debian and Debian derived system on the planet.
> > > > 
> > > > To my knowledge, it is only fedora and possibly one or two other dists
> > > > that put the firmware files in a unary /lib/firmware location, rather
> > > > than a versioned /lib/firmware/$KERNELRELASE one.
> > > 
> > > We (Mandriva Linux) do too.
> > > Why other distro cannot package firmware separatly for all their kernel
> > > flavors?
> > > 
> > > Firmware are not version-specific.
> > 
> > Yes it is, when it is bundled with the kernel source tree itself.
> > 
> > Why do people not realize this?
> > 
> > This is a real problem for distros, and for anyone who wants to have
> > multiple kernel versions on a single machine.  Jeff's patch fixes this.
> 
> once you move the firmware into its own package built from a separate
> source tree (and not the kernel) this is no longer a problem.

Agreed, but that's not what is going on here at all.

This is the firmware that is in the kernel source tree, that has been
moved to use request_firmware, and was originally tied tightly to the
kernel drivers themselves.

Now I can't install multiple kernel versions of a rpm package on my
machine at once.  Or even a single rpm package and then my own build,
without stomping on files owned by a different package.

That's unacceptable and can not work and is why Jeff made this patch.

thanks,

greg k-h

Re: [PATCH] firmware: Allow release-specific firmware dir

[Greg KH]

On Thu, Sep 11, 2008 at 09:45:38AM -0700, David Woodhouse wrote:
> On Thu, 2008-09-11 at 18:39 +0200, Marcel Holtmann wrote:
> > What is wrong with moving the firmware into its own package with its
> > own version number?
> 
> Nothing. You can do this today, building from the linux-firmware
> repository. That repository is starting to contain extra firmware that
> vendors are now willing to let us distribute -- where they _weren't_
> willing to let us distribute it as part of the kernel.

That's great and wonderful and has absolutely nothing to do with the
real problem we have right now in 2.6.27-rc.

Jeff's patch is one proposed solution for the problem, what's the
objection to it?  Reliance on a udev change to catch this as well?  It
looks like SuSE and Ubuntu already handle this, and I'm sure Fedora can
as well, right?

If the udev script requirement is a problem, then how does anyone else
suggest to solve the problem that we currently have with the existing
code?

thanks,

greg k-h

Re: [PATCH] firmware: Allow release-specific firmware dir

[David Woodhouse]

On Thu, 2008-09-11 at 13:15 -0700, Greg KH wrote:
> This is the firmware that is in the kernel source tree, that has been
> moved to use request_firmware, and was originally tied tightly to the
> kernel drivers themselves.

Not really. Most of it hasn't changed for years; it isn't _really_ tied
that closely to the kernel. 

You can just ignore what the kernel ships with, and install the firmware
from the linux-firmware.git repository instead.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

Re: [PATCH] firmware: Allow release-specific firmware dir

[Greg KH]

On Thu, Sep 11, 2008 at 01:38:38PM -0700, David Woodhouse wrote:
> On Thu, 2008-09-11 at 13:15 -0700, Greg KH wrote:
> > This is the firmware that is in the kernel source tree, that has been
> > moved to use request_firmware, and was originally tied tightly to the
> > kernel drivers themselves.
> 
> Not really. Most of it hasn't changed for years; it isn't _really_ tied
> that closely to the kernel. 

Some are and some aren't (I have two in my -staging tree that are
changing as the driver changes, so it does happen.)

> You can just ignore what the kernel ships with, and install the firmware
> from the linux-firmware.git repository instead.

So you are now forcing distros to ship the linux-firmware.git repo
instead?  That's not nice and is a totally new dependancy for them to
handle.

What about the very basic fact that kernel versions will stomp on files
from other kernel versions if you install multiple kernels on the same
machine?  That's just bad and ripe for problems in any package
management system.

thanks,

greg k-h

Re: [PATCH] firmware: Allow release-specific firmware dir

[David Woodhouse]

On Thu, 2008-09-11 at 13:57 -0700, Greg KH wrote:
> On Thu, Sep 11, 2008 at 01:38:38PM -0700, David Woodhouse wrote:
> > On Thu, 2008-09-11 at 13:15 -0700, Greg KH wrote:
> > > This is the firmware that is in the kernel source tree, that has been
> > > moved to use request_firmware, and was originally tied tightly to the
> > > kernel drivers themselves.
> > 
> > Not really. Most of it hasn't changed for years; it isn't _really_ tied
> > that closely to the kernel. 
> 
> Some are and some aren't (I have two in my -staging tree that are
> changing as the driver changes, so it does happen.)

That's fine. Just make sure the filename changes when an incompatible
change is made to the firmware -- just like you handle sonames in
libraries. It's not hard, and you should _always_ have been doing it.

And it's a completely bogus example _anyway_, because you're not adding
this extra firmware to the kernel tree.

Remember, all recent drivers have been using request_firmware() for
years anyway, and even the older drivers with active and on-the-ball
maintainers have been switching to request_firmware().

All I've done _recently_ is a bit of a sweep on the stragglers. And
because of the amount of stupid whining, I made it possible to keep it
in the kernel tree rather than just evicting it, as we did in the past.

> > You can just ignore what the kernel ships with, and install the firmware
> > from the linux-firmware.git repository instead.
> 
> So you are now forcing distros to ship the linux-firmware.git repo
> instead?  That's not nice and is a totally new dependancy for them to
> handle.

Not at all; that's the ideal situation, but nobody's forced to do it
that way.

> What about the very basic fact that kernel versions will stomp on files
> from other kernel versions if you install multiple kernels on the same
> machine?  That's just bad and ripe for problems in any package
> management system.

Only if you do stupid things in your packaging. So don't do that.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

Re: [PATCH] firmware: Allow release-specific firmware dir

[Greg KH]

On Thu, Sep 11, 2008 at 02:15:25PM -0700, David Woodhouse wrote:
> On Thu, 2008-09-11 at 13:57 -0700, Greg KH wrote:
> > On Thu, Sep 11, 2008 at 01:38:38PM -0700, David Woodhouse wrote:
> > > On Thu, 2008-09-11 at 13:15 -0700, Greg KH wrote:
> > > > This is the firmware that is in the kernel source tree, that has been
> > > > moved to use request_firmware, and was originally tied tightly to the
> > > > kernel drivers themselves.
> > > 
> > > Not really. Most of it hasn't changed for years; it isn't _really_ tied
> > > that closely to the kernel. 
> > 
> > Some are and some aren't (I have two in my -staging tree that are
> > changing as the driver changes, so it does happen.)
> 
> That's fine. Just make sure the filename changes when an incompatible
> change is made to the firmware -- just like you handle sonames in
> libraries. It's not hard, and you should _always_ have been doing it.
> 
> And it's a completely bogus example _anyway_, because you're not adding
> this extra firmware to the kernel tree.

No, it shows that the firmware does change over time.

> All I've done _recently_ is a bit of a sweep on the stragglers. And
> because of the amount of stupid whining, I made it possible to keep it
> in the kernel tree rather than just evicting it, as we did in the past.

But it was that "sweep" that has caused problems by having the kernel
put these files into the lib/firmware directory, without any version
information unlike any other file that our kernel installs.

That's the only point here.

It seems that the point that it is a bad thing for multiple kernel
packages to be installing into the same location, no matter if the file
has the same content or not (date will change which will cause
problems) isn't coming across here.  Why is that?

Think of the very real example here:

	rpm -Uhv kernel-2.6.27.1.rpm
	rpm -Uhv kernel-2.6.27.2.rpm
	rpm -e kernel-2.6.28.1

Now what just happened to the firmware that the 2.6.27.2 kernel package
had installed into lib/firmware?  It would be gone.  That is if rpm even
_allowed_ the second kernel package to be installed due to the
conflicting files.

So distros _HAVE_ to install the kernel-based firmware into
lib/firmware/uname-r/ to get their packaging to work, and to prevent bad
things from happening if a user installs their own kernel by hand and
then removes the kernel package.

That is why this patch was submitted, it is necessary.  If every distro
has to carry it on their own, due to packaging requirements, don't you
think that it should be in the main kernel tree as well?

> > > You can just ignore what the kernel ships with, and install the firmware
> > > from the linux-firmware.git repository instead.
> > 
> > So you are now forcing distros to ship the linux-firmware.git repo
> > instead?  That's not nice and is a totally new dependancy for them to
> > handle.
> 
> Not at all; that's the ideal situation, but nobody's forced to do it
> that way.

It's either use that package or add a patch that upstream seems to be
objecting to.  That seems a bit "forced".

> > What about the very basic fact that kernel versions will stomp on files
> > from other kernel versions if you install multiple kernels on the same
> > machine?  That's just bad and ripe for problems in any package
> > management system.
> 
> Only if you do stupid things in your packaging. So don't do that.

How do you expect anyone to package this up so that no conflicts occur?

thanks,

greg k-h

Re: [PATCH] firmware: Allow release-specific firmware dir

[David Woodhouse]

On Thu, 2008-09-11 at 15:07 -0700, Greg KH wrote:
> > And it's a completely bogus example _anyway_, because you're not adding
> > this extra firmware to the kernel tree.
> 
> No, it shows that the firmware does change over time.

For _new_ drivers.

Current drivers.

The ones which would be using request_firmware() anyway.

What is true for those is irrelevant to the discussion at hand. The only
drivers with firmware in the firmware/ directory of the kernel source
tree are the _old_, _unloved_ drivers which don't get updated. If they
had active and competent maintainers, they'd have been switched to
request_firmware() long ago _anyway_.

DIFFERENT DRIVERS, Greg. With DIFFERENT CHARACTERISTICS.

> > All I've done _recently_ is a bit of a sweep on the stragglers. And
> > because of the amount of stupid whining, I made it possible to keep it
> > in the kernel tree rather than just evicting it, as we did in the past.
> 
> But it was that "sweep" that has caused problems by having the kernel
> put these files into the lib/firmware directory, without any version
> information unlike any other file that our kernel installs.
>
> That's the only point here.

Yes. Except that it's nonsense because it's not really a problem, unless
you _choose_ to be stupid in the way you package it.

> It seems that the point that it is a bad thing for multiple kernel
> packages to be installing into the same location, no matter if the file
> has the same content or not (date will change which will cause
> problems) isn't coming across here.  Why is that?

Of _course_ it's coming across. So DON'T DO THAT THEN!

Take a look at the Fedora packaging. See how it doesn't do that.

> Think of the very real example here:

 < snip contrived bogus example >

Yeah, that would be broken. Don't do it like that. Or if you _must_ do
it that stupid way, feel free to override INSTALL_FW_PATH. But don't
expect us to pat you on the back for it. It's the _wrong_ thing to do.

> > > > You can just ignore what the kernel ships with, and install the firmware
> > > > from the linux-firmware.git repository instead.
> > > 
> > > So you are now forcing distros to ship the linux-firmware.git repo
> > > instead?  That's not nice and is a totally new dependancy for them to
> > > handle.
> > 
> > Not at all; that's the ideal situation, but nobody's forced to do it
> > that way.
> 
> It's either use that package or add a patch that upstream seems to be
> objecting to.  That seems a bit "forced".
> 
> > > What about the very basic fact that kernel versions will stomp on files
> > > from other kernel versions if you install multiple kernels on the same
> > > machine?  That's just bad and ripe for problems in any package
> > > management system.
> > 
> > Only if you do stupid things in your packaging. So don't do that.
> 
> How do you expect anyone to package this up so that no conflicts occur?

Take a look at how Fedora does it. There is a 'kernel-firmware'
subpackage which is automatically generated with the kernel build, but
you don't have multiple such packages installed at the same time -- you
only have the _latest_ one.

This has a theoretical problem if drivers are removed from the tree, or
if firmware is removed from the tree (for example, because a newer
incompatible version supersedes it). That's fixed by packaging firmware
from the linux-firmware.git repository, in which the old firmware will
live on even after the current Linux kernel no longer has it. But since
it _is_ a purely theoretical problem, there's no particular rush for
that to happen. As I said, these ancient unloved drivers aren't getting
their firmware updated _anyway_.

Now, can we _please_ stop being bloody stupid? If there are _real_
problems, like the one I fixed with commit 1cede1af last week, then I'm
perfectly happy to deal with them. But stop making crap up.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

Re: [PATCH] firmware: Allow release-specific firmware dir

[Joseph Fannin]

On Thu, Sep 11, 2008 at 03:25:43PM -0700, David Woodhouse wrote:
> On Thu, 2008-09-11 at 15:07 -0700, Greg KH wrote:
<snip>

> Now, can we _please_ stop being bloody stupid? If there are _real_
> problems, like the one I fixed with commit 1cede1af last week, then I'm
> perfectly happy to deal with them. But stop making crap up.

I CAN'T INSTALL MORE THAN ONE KERNEL PACKAGE > v2.6.26.

IT'S NOT IMAGINARY.

IT USED TO WORK AND YOU BROKE IT.


--
Joseph Fannin
jfannin@gmail.com

Re: [PATCH] firmware: Allow release-specific firmware dir

[Gene Heskett]

On Friday 12 September 2008, Joseph Fannin wrote:
>On Thu, Sep 11, 2008 at 03:25:43PM -0700, David Woodhouse wrote:
>> On Thu, 2008-09-11 at 15:07 -0700, Greg KH wrote:
>
><snip>
>
>> Now, can we _please_ stop being bloody stupid? If there are _real_
>> problems, like the one I fixed with commit 1cede1af last week, then I'm
>> perfectly happy to deal with them. But stop making crap up.
>
>I CAN'T INSTALL MORE THAN ONE KERNEL PACKAGE > v2.6.26.
>
>IT'S NOT IMAGINARY.
>
>IT USED TO WORK AND YOU BROKE IT.
>
I've been building my own, and currently have about 10 kernels newer than 
2.6.26 sitting in my /boot partition, and they all work.  I'd suggest you 
investigate how you are maintaining your .config(uration)

>--
>Joseph Fannin
>jfannin@gmail.com
>
>--
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/



-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
You would if you could but you can't so you won't.

Re: [PATCH] firmware: Allow release-specific firmware dir

[David Woodhouse]

On Fri, 2008-09-12 at 04:39 -0400, Joseph Fannin wrote:
> On Thu, Sep 11, 2008 at 03:25:43PM -0700, David Woodhouse wrote:
> > On Thu, 2008-09-11 at 15:07 -0700, Greg KH wrote:
> <snip>
> 
> > Now, can we _please_ stop being bloody stupid? If there are _real_
> > problems, like the one I fixed with commit 1cede1af last week, then I'm
> > perfectly happy to deal with them. But stop making crap up.
> 
> I CAN'T INSTALL MORE THAN ONE KERNEL PACKAGE > v2.6.26.
> 
> IT'S NOT IMAGINARY.
> 
> IT USED TO WORK AND YOU BROKE IT.

Then your kernel packaging is broken. Fix it.

Or go an stand in the corner with the muppets who are whining that the
move from arch/i386 to arch/x86 also broke their kernel packaging....
and who refuse to fix that too.

Oh, wait...there weren't any of those; it wasn't even a _consideration_.

You're just being an idiot, because people like to whine about firmware.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

Re: [PATCH] firmware: Allow release-specific firmware dir

[Kai Henningsen]

On Fri, Sep 12, 2008 at 16:32, David Woodhouse <dwmw2@infradead.org> wrote:
> On Fri, 2008-09-12 at 04:39 -0400, Joseph Fannin wrote:
>> On Thu, Sep 11, 2008 at 03:25:43PM -0700, David Woodhouse wrote:
>> > On Thu, 2008-09-11 at 15:07 -0700, Greg KH wrote:
>> <snip>
>>
>> > Now, can we _please_ stop being bloody stupid? If there are _real_
>> > problems, like the one I fixed with commit 1cede1af last week, then I'm
>> > perfectly happy to deal with them. But stop making crap up.
>>
>> I CAN'T INSTALL MORE THAN ONE KERNEL PACKAGE > v2.6.26.
>>
>> IT'S NOT IMAGINARY.
>>
>> IT USED TO WORK AND YOU BROKE IT.
>
> Then your kernel packaging is broken. Fix it.

I might be confused here, but my impression is that he complains about
packaging that is part of the kernel sources. That really makes it a
stretch to call it "his" packaging.

Let's see ... yes, scripts/packaging stuff ... assuming I parsed
MAINTAINERS correctly, that means it belongs to

KERNEL BUILD (kbuild: Makefile, scripts/Makefile.*)
P:      Sam Ravnborg
M:      sam@ravnborg.org
T:      git kernel.org:/pub/scm/linux/kernel/git/sam/kbuild-next.git
T:      git kernel.org:/pub/scm/linux/kernel/git/sam/kbuild-fixes.git
L:      linux-kbuild@vger.kernel.org
S:      Maintained

No?

I've never used that stuff myself. I prefer working with the (Debian)
kernel-package (make-kpkg) tool, which is also used to create the
distribution packages I believe, and which when I last used it
certainly had an active maintainer.

Re: [PATCH] firmware: Allow release-specific firmware dir

[Andrew Morton]

On Thu, 24 Feb 2011 21:39:35 -0500
Jeff Mahoney <jeffm@suse.com> wrote:

>  Every kernel package trying to provide files under /lib/firmware runs
>  into problems really quickly with multiple kernels installed.
> 
>  This patch moves them to /lib/firmware/$KERNELRELEASE. udev v127's
>  firmware.sh looks there first before falling back to /lib/firmware.
> 
> ...
>
> --- a/Makefile
> +++ b/Makefile
> @@ -975,7 +975,7 @@ depend dep:
>  
>  # ---------------------------------------------------------------------------
>  # Firmware install
> -INSTALL_FW_PATH=$(INSTALL_MOD_PATH)/lib/firmware
> +INSTALL_FW_PATH=$(INSTALL_MOD_PATH)/lib/firmware/$(KERNELRELEASE)
>  export INSTALL_FW_PATH
>  
>  PHONY += firmware_install

What happens if the user is running older udev?

Re: [PATCH] firmware: Allow release-specific firmware dir

[David Woodhouse]

On Fri, 2011-02-25 at 10:03 -0500, Jeff Mahoney wrote:
> 
> Because every time you run 'make modules_install', it overwrites
> whatever is in /lib/firmware when there is a collision. When you're
> working with multiple kernel versions, you can end up with firmware
> blobs from older builds, and that may not be what you want. If the
> right answer is to always use kernel-firmware, then the firmwares
> should probably not be in the kernel tree at all.

The right answer is to always use kernel-firmware. The firmwares should
not be in the kernel tree at all.

Changing the default behaviour of 'make modules_install' so that it no
longer installs firmware would be the better option. As a stop-gap until
we finally do drop the legacy firmware images from the source tree.

-- 
dwmw2

Re: [PATCH] firmware: Allow release-specific firmware dir

[Jeff Mahoney]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 02/25/2011 12:01 AM, Michael Tokarev wrote:
> 25.02.2011 05:39, Jeff Mahoney wrote:
>>  Every kernel package trying to provide files under /lib/firmware runs
>>  into problems really quickly with multiple kernels installed.
> 
> Which problems?
> 
>>  This patch moves them to /lib/firmware/$KERNELRELEASE. udev v127's
>>  firmware.sh looks there first before falling back to /lib/firmware.
> 
> This was the way it worked before, -- all fw in kernel-specific dir.
> Nowadays it moved to a common place, /lib/firmware, since the blobs
> there are not kernel-specific but hardware-specific.  And now you
> propose to move it back.  Why?

Because every time you run 'make modules_install', it overwrites
whatever is in /lib/firmware when there is a collision. When you're
working with multiple kernel versions, you can end up with firmware
blobs from older builds, and that may not be what you want. If the right
answer is to always use kernel-firmware, then the firmwares should
probably not be in the kernel tree at all.

As I mentioned in the header, udev looks in /lib/firmware/$(uname -r)
already. Since Aug 2008, in fact.

- -Jeff

- -- 
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/

iEYEARECAAYFAk1nxKYACgkQLPWxlyuTD7KohACfRmDshnoXwVTpSs+xblUfQSKQ
svkAnikK8sk7S6Fd8NKT1Ui2ugaP7IzC
=odyW
-----END PGP SIGNATURE-----

Re: [PATCH] firmware: Allow release-specific firmware dir

[Michael Tokarev]

25.02.2011 05:39, Jeff Mahoney wrote:
>  Every kernel package trying to provide files under /lib/firmware runs
>  into problems really quickly with multiple kernels installed.

Which problems?

>  This patch moves them to /lib/firmware/$KERNELRELEASE. udev v127's
>  firmware.sh looks there first before falling back to /lib/firmware.

This was the way it worked before, -- all fw in kernel-specific dir.
Nowadays it moved to a common place, /lib/firmware, since the blobs
there are not kernel-specific but hardware-specific.  And now you
propose to move it back.  Why?

/mjt

[PATCH] firmware: Allow release-specific firmware dir

[Jeff Mahoney]

 Every kernel package trying to provide files under /lib/firmware runs
 into problems really quickly with multiple kernels installed.

 This patch moves them to /lib/firmware/$KERNELRELEASE. udev v127's
 firmware.sh looks there first before falling back to /lib/firmware.

Signed-off-by: Jeff Mahoney <jeffm@suse.com>
---
 Makefile |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- a/Makefile
+++ b/Makefile
@@ -975,7 +975,7 @@ depend dep:
 
 # ---------------------------------------------------------------------------
 # Firmware install
-INSTALL_FW_PATH=$(INSTALL_MOD_PATH)/lib/firmware
+INSTALL_FW_PATH=$(INSTALL_MOD_PATH)/lib/firmware/$(KERNELRELEASE)
 export INSTALL_FW_PATH
 
 PHONY += firmware_install
-- 
Jeff Mahoney
SUSE Labs