[GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

Linus, please pull from the for-2.6.27 branch of:
	git://git.infradead.org/users/dwmw2/firmware-2.6.git for-2.6.27

This does the following (not quite in this order):

 - Provides a method for firmware files to be built into the kernel, so
   that drivers can be updated to use request_firmware() without forcing
   people to use an initrd.

 - In doing that, mark fw->data as 'const', and fix a few drivers to cope.

 - Update a bunch of drivers to use request_firmware(), and move their
   existing firmware, previously hardcoded in all kinds of interesting
   ways, into the firmware/ directory of the kernel tree.

 - Add a FIRMWARE_IN_KERNEL config option which allows the firmware for
   the above (and below) drivers to continue to be built into the kernel.

 - Add a simple method for representing Intel HEX records in binary form,
   since a bunch of drivers need that kind of thing. And a tool to
   convert HEX files into that binary form, for the kernel to load.

 - Drop special cases from a few drivers which could either use
   request_firmware() or build their firmware in; they don't need to
   do that for themselves any more.

 - Implement 'make firmware_install' target, which installs all firmware
   shipped with the kernel.

 - Modify 'make modules_install' to install any firmware required by the
   modules it installs (where that firmware is shipped in the kernel tree).

There are more drivers to be converted to request_firmware() -- notably
drivers/net and drivers/scsi, but it's probably best to ensure that the
infrastructure is all in place and working correctly before we update
those. (Although since modern drivers have been using request_firmware()
for some time, I don't think there's likely to be a problem there; we'll
probably clean up the stragglers in time for 2.6.28.)

David Howells (5):
      Fix a const pointer usage warning in the Digigram VX soundcard driver
      Fix a const assignment in moxa_load_fw()
      Fix a const pointer error in the Conexant cx23418 MPEG encoder driver
      Fix a const pointer usage warning in the Digigram pcxhr soundcard driver
      Fix a const pointer usage warning in the Digigram miXart soundcard driver

Full shortlog and diffstat below, although the really interesting part
is probably between commits fd4f80de..59890f74 -- seen at 
http://git.infradead.org/users/dwmw2/firmware-2.6.git?a=shortlog;h=59890f74

David Woodhouse (52):
      libertas: treat firmware data as const
      bluetooth: treat firmware data as const
      cyclades: treat firmware data as const
      cx25840: treat firmware data as const
      myri10ge: treat firmware data as const
      vx222: treat firmware data as const
      riptide: treat firmware data as const
      pcxhr: treat firmware data as const
      vx: treat firmware data as const
      ueagle-atm: treat firmware data as const
      cxacru: treat firmware data as const
      aic94xx: treat firmware data as const
      zd1201: treat firmware data as const
      rt2x00: treat firmware data as const
      p54: treat firmware data as const
      atmel: treat firmware data as const
      irda-usb: treat firmware data as const
      cxgb3: treat firmware data as const
      bt8xx: treat firmware data as const
      ttusb-dec: treat firmware data as const
      dvb frontends: treat firmware data as const
      cxusb: treat firmware data as const
      gp8psk: treat firmware data as const
      tuners: treat firmware data as const
      maestro3: treat firmware data as const
      ymfpci: treat firmware data as const
      firmware: make fw->data const
      firmware: allow firmware files to be built into kernel image
      firmware: Add CONFIG_EXTRA_FIRMWARE option
      firmware: Add CONFIG_FIRMWARE_IN_KERNEL option.
      firmware: Add firmware installation to modules_install, add firmware_install
      ihex.h: binary representation of ihex records
      ihex: request_ihex_firmware() function to load and validate firmware
      ihex: add ihex2fw tool for converting HEX files into firmware images
      ihex: Add support for long records to ihex2fw.c
      firmware: convert korg1212 driver to use firmware loader exclusively
      firmware: convert maestro3 driver to use firmware loader exclusively
      firmware: convert ymfpci driver to use firmware loader exclusively
      smctr: use request_firmware()
      kaweth: use request_firmware()
      ttusb-budget: use request_firmware()
      keyspan: use request_firmware()
      keyspan_pda: use request_firmware()
      emi26: use request_firmware()
      emi62: use request_firmware()
      ti_usb_3410_5052: use request_firmware()
      whiteheat: use request_firmware()
      firmware: convert Ambassador ATM driver to request_firmware()
      ip2: use request_firmware()
      cpia2: use request_firmware()
      dabusb: use request_firmware()
      Merge git://git.kernel.org/.../torvalds/linux-2.6.git

Greg Kroah-Hartman (3):
      dell_rbu: firmware data is const
      x86 microcode: firmware data is const
      isight: treat firmware data as const

Jaswinder Singh (5):
      vicam: use request_firmware()
      edgeport: use request_firmware()
      edgeport-ti: use request_firmware()
      dsp56k: use request_firmware
      firmware: convert sb16_csp driver to use firmware loader exclusively

 Makefile                                           |   15 +-
 arch/x86/kernel/microcode.c                        |    4 +-
 drivers/atm/Makefile                               |    6 +-
 drivers/atm/ambassador.c                           |  140 +-
 drivers/atm/ambassador.h                           |   11 -
 drivers/atm/atmsar11.data                          | 2063 -----
 drivers/atm/atmsar11.regions                       |    6 -
 drivers/atm/atmsar11.start                         |    4 -
 drivers/base/Kconfig                               |   64 +
 drivers/base/firmware_class.c                      |   35 +-
 drivers/bluetooth/bfusb.c                          |    3 +-
 drivers/bluetooth/bt3c_cs.c                        |    3 +-
 drivers/char/cyclades.c                            |   10 +-
 drivers/char/dsp56k.c                              |   84 +-
 drivers/char/ip2/fip_firm.h                        | 2149 -----
 drivers/char/ip2/ip2base.c                         |    5 +-
 drivers/char/ip2/ip2main.c                         |   47 +-
 drivers/char/moxa.c                                |    4 +-
 drivers/firmware/dell_rbu.c                        |    2 +-
 drivers/media/common/tuners/tuner-xc2028.c         |    2 +-
 drivers/media/common/tuners/xc5000.c               |    2 +-
 drivers/media/dvb/dvb-usb/cxusb.c                  |   21 +-
 drivers/media/dvb/dvb-usb/gp8psk.c                 |    3 +-
 drivers/media/dvb/frontends/bcm3510.c              |    5 +-
 drivers/media/dvb/frontends/nxt200x.c              |    3 +-
 drivers/media/dvb/frontends/or51211.c              |    4 +-
 drivers/media/dvb/frontends/sp8870.c               |    2 +-
 drivers/media/dvb/frontends/sp887x.c               |    2 +-
 drivers/media/dvb/frontends/tda10048.c             |    2 +-
 drivers/media/dvb/frontends/tda1004x.c             |    2 +-
 drivers/media/dvb/ttusb-budget/dvb-ttusb-budget.c  |   16 +-
 .../media/dvb/ttusb-budget/dvb-ttusb-dspbootcode.h | 1644 ----
 drivers/media/dvb/ttusb-dec/ttusb_dec.c            |    2 +-
 drivers/media/video/bt8xx/bttv-cards.c             |    3 +-
 drivers/media/video/cpia2/cpia2_core.c             |   46 +-
 drivers/media/video/cpia2/cpia2patch.h             |  233 -
 drivers/media/video/cx18/cx18-av-firmware.c        |    2 +-
 drivers/media/video/cx25840/cx25840-firmware.c     |   27 +-
 drivers/media/video/dabfirmware.h                  | 1415 ----
 drivers/media/video/dabusb.c                       |   44 +-
 drivers/media/video/usbvideo/vicam.c               |  317 +-
 drivers/net/cxgb3/common.h                         |    5 +-
 drivers/net/cxgb3/t3_hw.c                          |    7 +-
 drivers/net/irda/irda-usb.c                        |   12 +-
 drivers/net/myri10ge/myri10ge.c                    |   11 +-
 drivers/net/tokenring/smctr.c                      |   56 +-
 drivers/net/tokenring/smctr.h                      |    2 -
 drivers/net/tokenring/smctr_firmware.h             |  978 ---
 drivers/net/usb/kaweth.c                           |   43 +-
 drivers/net/usb/kawethfw.h                         |  557 --
 drivers/net/wireless/atmel.c                       |    6 +-
 drivers/net/wireless/libertas/if_cs.c              |    2 +-
 drivers/net/wireless/libertas/if_sdio.c            |    4 +-
 drivers/net/wireless/libertas/if_usb.c             |    4 +-
 drivers/net/wireless/p54/p54usb.c                  |    5 +-
 drivers/net/wireless/rt2x00/rt2x00.h               |    4 +-
 drivers/net/wireless/rt2x00/rt2x00pci.h            |    2 +-
 drivers/net/wireless/rt2x00/rt61pci.c              |    4 +-
 drivers/net/wireless/rt2x00/rt73usb.c              |    6 +-
 drivers/net/wireless/zd1201.c                      |    2 +-
 drivers/scsi/aic94xx/aic94xx_sds.c                 |   12 +-
 drivers/scsi/aic94xx/aic94xx_sds.h                 |    4 +-
 drivers/scsi/aic94xx/aic94xx_seq.c                 |    7 +-
 drivers/usb/atm/cxacru.c                           |    2 +-
 drivers/usb/atm/ueagle-atm.c                       |   20 +-
 drivers/usb/misc/emi26.c                           |   96 +-
 drivers/usb/misc/emi26_fw.h                        | 5779 -------------
 drivers/usb/misc/emi62.c                           |  130 +-
 drivers/usb/misc/emi62_fw_m.h                      | 8853 --------------------
 drivers/usb/misc/emi62_fw_s.h                      | 8837 -------------------
 drivers/usb/misc/isight_firmware.c                 |    2 +-
 drivers/usb/serial/Kconfig                         |   24 +-
 drivers/usb/serial/io_edgeport.c                   |  183 +-
 drivers/usb/serial/io_fw_boot.h                    |  556 --
 drivers/usb/serial/io_fw_boot2.h                   |  546 --
 drivers/usb/serial/io_fw_down.h                    | 1229 ---
 drivers/usb/serial/io_fw_down2.h                   | 1133 ---
 drivers/usb/serial/io_fw_down3.h                   |  847 --
 drivers/usb/serial/io_ti.c                         |   73 +-
 drivers/usb/serial/keyspan.c                       |   78 +-
 drivers/usb/serial/keyspan.h                       |   84 -
 drivers/usb/serial/keyspan_mpr_fw.h                |  286 -
 drivers/usb/serial/keyspan_pda.S                   | 1124 ---
 drivers/usb/serial/keyspan_pda.c                   |   51 +-
 drivers/usb/serial/keyspan_pda_fw.h                |   99 -
 drivers/usb/serial/keyspan_usa18x_fw.h             |  447 -
 drivers/usb/serial/keyspan_usa19_fw.h              |  285 -
 drivers/usb/serial/keyspan_usa19qi_fw.h            |  284 -
 drivers/usb/serial/keyspan_usa19qw_fw.h            |  448 -
 drivers/usb/serial/keyspan_usa19w_fw.h             |  446 -
 drivers/usb/serial/keyspan_usa28_fw.h              |  466 -
 drivers/usb/serial/keyspan_usa28x_fw.h             |  447 -
 drivers/usb/serial/keyspan_usa28xa_fw.h            |  449 -
 drivers/usb/serial/keyspan_usa28xb_fw.h            |  448 -
 drivers/usb/serial/keyspan_usa49w_fw.h             |  464 -
 drivers/usb/serial/keyspan_usa49wlc_fw.h           |  476 --
 drivers/usb/serial/ti_fw_3410.h                    |  885 --
 drivers/usb/serial/ti_fw_5052.h                    |  885 --
 drivers/usb/serial/ti_usb_3410_5052.c              |   40 +-
 drivers/usb/serial/whiteheat.c                     |   77 +-
 drivers/usb/serial/whiteheat_fw.h                  | 1669 ----
 drivers/usb/serial/xircom_pgs.S                    | 1192 ---
 drivers/usb/serial/xircom_pgs_fw.h                 |  103 -
 firmware/Makefile                                  |  177 +
 firmware/WHENCE                                    |  341 +
 firmware/atmsar11.HEX                              |  204 +
 firmware/cpia2/stv0672_vp4.bin.ihex                |   73 +
 firmware/dabusb/bitstream.bin.ihex                 |  761 ++
 firmware/dabusb/firmware.HEX                       |  649 ++
 firmware/dsp56k/bootstrap.asm                      |   98 +
 firmware/dsp56k/bootstrap.bin.ihex                 |   26 +
 firmware/edgeport/boot.H16                         |   29 +
 firmware/edgeport/boot2.H16                        |   28 +
 firmware/edgeport/down.H16                         |   29 +
 firmware/edgeport/down2.H16                        |   29 +
 firmware/edgeport/down3.bin.ihex                   |  815 ++
 firmware/emi26/bitstream.HEX                       | 4391 ++++++++++
 firmware/emi26/firmware.HEX                        | 1261 +++
 firmware/emi26/loader.HEX                          |  116 +
 firmware/emi62/bitstream.HEX                       | 4372 ++++++++++
 firmware/emi62/loader.HEX                          |  107 +
 firmware/emi62/midi.HEX                            | 1266 +++
 firmware/emi62/spdif.HEX                           | 1257 +++
 firmware/ess/maestro3_assp_kernel.fw.ihex          |  120 +
 firmware/ess/maestro3_assp_minisrc.fw.ihex         |   51 +
 firmware/ihex2fw.c                                 |  268 +
 firmware/intelliport2.bin.ihex                     | 2147 +++++
 firmware/kaweth/new_code.bin.ihex                  |  206 +
 firmware/kaweth/new_code_fix.bin.ihex              |   40 +
 firmware/kaweth/trigger_code.bin.ihex              |   13 +
 firmware/kaweth/trigger_code_fix.bin.ihex          |    3 +
 firmware/keyspan/mpr.HEX                           |  104 +
 firmware/keyspan/usa18x.HEX                        |  141 +
 firmware/keyspan/usa19.HEX                         |  101 +
 firmware/keyspan/usa19qi.HEX                       |  101 +
 firmware/keyspan/usa19qw.HEX                       |  142 +
 firmware/keyspan/usa19w.HEX                        |  141 +
 firmware/keyspan/usa28.HEX                         |  148 +
 firmware/keyspan/usa28x.HEX                        |  141 +
 firmware/keyspan/usa28xa.HEX                       |  141 +
 firmware/keyspan/usa28xb.HEX                       |  142 +
 firmware/keyspan/usa49w.HEX                        |  145 +
 firmware/keyspan/usa49wlc.HEX                      |  153 +
 firmware/keyspan_pda/keyspan_pda.HEX               |   83 +
 firmware/keyspan_pda/keyspan_pda.S                 | 1124 +++
 firmware/keyspan_pda/xircom_pgs.HEX                |   87 +
 firmware/keyspan_pda/xircom_pgs.S                  | 1192 +++
 firmware/korg/k1212.dsp.ihex                       |  987 +++
 firmware/sb16/alaw_main.csp.ihex                   |   87 +
 firmware/sb16/ima_adpcm_capture.csp.ihex           |  121 +
 firmware/sb16/ima_adpcm_init.csp.ihex              |   70 +
 firmware/sb16/ima_adpcm_playback.csp.ihex          |  122 +
 firmware/sb16/mulaw_main.csp.ihex                  |   84 +
 firmware/ti_3410.fw.ihex                           |  862 ++
 firmware/ti_5052.fw.ihex                           |  862 ++
 firmware/tr_smctr.bin.ihex                         |  477 ++
 firmware/ttusb-budget/dspbootcode.bin.ihex         |  820 ++
 firmware/vicam/firmware.H16                        |    7 +
 firmware/whiteheat.HEX                             | 1097 +++
 firmware/whiteheat_loader.HEX                      |  314 +
 firmware/whiteheat_loader_debug.HEX                |  403 +
 firmware/yamaha/ds1_ctrl.fw.ihex                   |  769 ++
 firmware/yamaha/ds1_dsp.fw.ihex                    |    9 +
 firmware/yamaha/ds1e_ctrl.fw.ihex                  |  769 ++
 include/asm-generic/vmlinux.lds.h                  |    7 +
 include/linux/firmware.h                           |   23 +-
 include/linux/ihex.h                               |   74 +
 scripts/Makefile.fwinst                            |   45 +
 sound/drivers/vx/vx_core.c                         |    4 +-
 sound/isa/Kconfig                                  |    9 -
 sound/isa/sb/sb16_csp.c                            |   22 -
 sound/isa/sb/sb16_csp_codecs.h                     |  949 ---
 sound/pci/Kconfig                                  |   30 -
 sound/pci/korg1212/korg1212-firmware.h             |  987 ---
 sound/pci/korg1212/korg1212.c                      |   18 -
 sound/pci/maestro3.c                               |  186 +-
 sound/pci/mixart/mixart_hwdep.c                    |    2 +-
 sound/pci/pcxhr/pcxhr_core.c                       |    4 +-
 sound/pci/pcxhr/pcxhr_hwdep.c                      |    2 +-
 sound/pci/riptide/riptide.c                        |   10 +-
 sound/pci/vx222/vx222_ops.c                        |    2 +-
 sound/pci/ymfpci/ymfpci_image.h                    | 1565 ----
 sound/pci/ymfpci/ymfpci_main.c                     |   74 +-
 183 files changed, 31965 insertions(+), 52599 deletions(-)


-- 
dwmw2 @linux.intel.com

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Andrew Morton]

On Mon, 14 Jul 2008 16:23:26 -0700
David Woodhouse <dwmw2@infradead.org> wrote:

> Linus, please pull from the for-2.6.27 branch of:
> 	git://git.infradead.org/users/dwmw2/firmware-2.6.git for-2.6.27

The firmware flamewars seem to have subsided lately.  Is everyone happy
with this now?

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Mon, 2008-07-14 at 16:41 -0700, Andrew Morton wrote:
> On Mon, 14 Jul 2008 16:23:26 -0700
> David Woodhouse <dwmw2@infradead.org> wrote:
> 
> > Linus, please pull from the for-2.6.27 branch of:
> > 	git://git.infradead.org/users/dwmw2/firmware-2.6.git for-2.6.27
> 
> The firmware flamewars seem to have subsided lately.  Is everyone happy
> with this now?

I've deliberately left drivers/net out of this pull request... all
_other_ maintainers seemed quite happy to see their drivers updated to
use request_firmware().

Let's see how it pans out. I don't expect any problems -- after all,
modern drivers have been using request_firmware() for a _long_ time now,
and the distributions cope quite happily with it already. We're ready to
ship the firmware in a separate package in Fedora as soon as Linus pulls
from my tree. Once the dust has settled and it's all working fine, we
can look at converting drivers/net and drivers/scsi too.

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Yinghai Lu]

On Mon, Jul 14, 2008 at 4:47 PM, David Woodhouse <dwmw2@infradead.org> wrote:
> On Mon, 2008-07-14 at 16:41 -0700, Andrew Morton wrote:
>> On Mon, 14 Jul 2008 16:23:26 -0700
>> David Woodhouse <dwmw2@infradead.org> wrote:
>>
>> > Linus, please pull from the for-2.6.27 branch of:
>> >     git://git.infradead.org/users/dwmw2/firmware-2.6.git for-2.6.27
>>
>> The firmware flamewars seem to have subsided lately.  Is everyone happy
>> with this now?
>
> I've deliberately left drivers/net out of this pull request... all
> _other_ maintainers seemed quite happy to see their drivers updated to
> use request_firmware().
>
> Let's see how it pans out. I don't expect any problems -- after all,
> modern drivers have been using request_firmware() for a _long_ time now,
> and the distributions cope quite happily with it already. We're ready to
> ship the firmware in a separate package in Fedora as soon as Linus pulls
> from my tree. Once the dust has settled and it's all working fine, we
> can look at converting drivers/net and drivers/scsi too.

so after this patchset, if i put qllogic fw in firmware/, that fw will
work or not...

YH

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Arjan van de Ven]

On Mon, 14 Jul 2008 16:41:19 -0700
Andrew Morton <akpm@linux-foundation.org> wrote:

> On Mon, 14 Jul 2008 16:23:26 -0700
> David Woodhouse <dwmw2@infradead.org> wrote:
> 
> > Linus, please pull from the for-2.6.27 branch of:
> > 	git://git.infradead.org/users/dwmw2/firmware-2.6.git
> > for-2.6.27
> 
> The firmware flamewars seem to have subsided lately.  Is everyone
> happy with this now?
> 

this seems to have left the contentious pieces out....


-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[david]

On Mon, 14 Jul 2008, Arjan van de Ven wrote:

> On Mon, 14 Jul 2008 16:41:19 -0700
> Andrew Morton <akpm@linux-foundation.org> wrote:
>
>> On Mon, 14 Jul 2008 16:23:26 -0700
>> David Woodhouse <dwmw2@infradead.org> wrote:
>>
>>> Linus, please pull from the for-2.6.27 branch of:
>>> 	git://git.infradead.org/users/dwmw2/firmware-2.6.git
>>> for-2.6.27
>>
>> The firmware flamewars seem to have subsided lately.  Is everyone
>> happy with this now?
>>
>
> this seems to have left the contentious pieces out....

almost, the one thing that this could have done would be to offer an 
option to bundle the firmware with the module. currently it offers the 
option to load the firmware at the same time as the module, not _quite_ 
the same thing.

there was also the issue that was raised about how to handle firmware 
during suspend/resume. I don't remember seeing a happy solution to that 
one.

David Lang

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Mon, 2008-07-14 at 16:53 -0700, Yinghai Lu wrote:
> so after this patchset, if i put qllogic fw in firmware/, that fw will
> work or not...

The qlogic driver already uses request_firmware(), like most modern
drivers do. If you want to include that firmware in your kernel image
(assuming it's in /lib/firmware already), do something like this:

	CONFIG_EXTRA_FIRMWARE="ql2500_fw.bin"
	CONFIG_EXTRA_FIRMWARE_DIR=/lib/firmware

Then it'll find that firmware file and pull it in to your kernel, and
you can set CONFIG_SCSI_QLA_FC=y and expect it to work. Select the right
firmware for your device, of course.

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Yinghai Lu]

On Mon, Jul 14, 2008 at 5:23 PM, David Woodhouse <dwmw2@infradead.org> wrote:
> On Mon, 2008-07-14 at 16:53 -0700, Yinghai Lu wrote:
>> so after this patchset, if i put qllogic fw in firmware/, that fw will
>> work or not...
>
> The qlogic driver already uses request_firmware(), like most modern
> drivers do. If you want to include that firmware in your kernel image
> (assuming it's in /lib/firmware already), do something like this:
>
>        CONFIG_EXTRA_FIRMWARE="ql2500_fw.bin"
>        CONFIG_EXTRA_FIRMWARE_DIR=/lib/firmware
>
> Then it'll find that firmware file and pull it in to your kernel, and
> you can set CONFIG_SCSI_QLA_FC=y and expect it to work. Select the right
> firmware for your device, of course.
>

CONFIG_EXTRA_FIRMWARE="ql2100_fw.bin  ql2300_fw.bin  ql2400_fw.bin
ql2200_fw.bin  ql2322_fw.bin  ql2500_fw.bin"
CONFIG_EXTRA_FIRMWARE_DIR="firmware"


make[1]: *** No rule to make target
`include/config/builtin/firmware/dir.h', needed by
`firmware/ql2100_fw.bin.gen.S'.  Stop.
make: *** [firmware] Error 2

YH

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Mon, 14 Jul 2008, david@lang.hm wrote:
> 
> there was also the issue that was raised about how to handle firmware during
> suspend/resume. I don't remember seeing a happy solution to that one.

I don't understand why this even keeps coming up.

The only reliable way to handle it is to load the firmware before the 
suspend happens - preferably by simply never unloading it in the first 
place.

Anything else is fraught with problems.

Stuff that expects to eject itself anyway(*) over a suspend event can 
obviously decide to throw it away and require a reload at resume time (by 
virtue of simply re-discovering the whole device!).

Everybody else should just keep the firmware in memory in the driver.

Does it waste some ram? Sure. Tough. 

			Linus

(*) eg some random USB device that really _really_ doesn't care and isn't 
expected to be involved in any filesystem access - say a webcam.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Mon, 2008-07-14 at 17:06 -0700, david@lang.hm wrote:
> On Mon, 14 Jul 2008, Arjan van de Ven wrote:
> 
> > On Mon, 14 Jul 2008 16:41:19 -0700
> > Andrew Morton <akpm@linux-foundation.org> wrote:
> >
> >> On Mon, 14 Jul 2008 16:23:26 -0700
> >> David Woodhouse <dwmw2@infradead.org> wrote:
> >>
> >>> Linus, please pull from the for-2.6.27 branch of:
> >>> 	git://git.infradead.org/users/dwmw2/firmware-2.6.git
> >>> for-2.6.27
> >>
> >> The firmware flamewars seem to have subsided lately.  Is everyone
> >> happy with this now?
> >>
> >
> > this seems to have left the contentious pieces out....
> 
> almost, the one thing that this could have done would be to offer an 
> option to bundle the firmware with the module. currently it offers the 
> option to load the firmware at the same time as the module, not _quite_ 
> the same thing.

I see no real point in that. If you have userspace to load modules, then
you have userspace to load firmware.

We made 'make modules_install' install the firmware in /lib/firmware for
you at the same time as it installs the modules, to avoid problems if
people forgot to run 'make firmware_install'. That really ought to be
enough.

I know Jeff complained that it wasn't, and insisted that he _needed_ to
be able to scp a single .ko file around which contained both, and the
world was broken if he could not -- but I disagree with him.

But since I wanted this tree to be uncontentious and obviously the
correct thing to do, I've dropped the drivers/net changes for now
anyway.

It's odd that this request has suddenly come out of the blue when we've
been using request_firmware() from modules for years already.

> there was also the issue that was raised about how to handle firmware 
> during suspend/resume. I don't remember seeing a happy solution to that 
> one.

My b43 seems to work fine on suspend/resume, as do most other modern
drivers which use request_firmware(). There is certainly no fundamental,
generic problem with suspend/resume vs. the firmware loader.

If you see a problem with a specific driver which we've converted, do
please point it out.

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Mon, 14 Jul 2008, David Woodhouse wrote:
> 
> I see no real point in that. If you have userspace to load modules, then
> you have userspace to load firmware.

David, the case is different: the module is loaded _across_ the 
suspend/resume, but the device needs to have its firmware reloaded at 
resume time because it lost power.

And since the device may be resumed before the root filesystem disk 
device, it needs to do so _without_ calling request_firmware() at resume 
time.

The above is _normal_.

And the fact is, the only way to handle it sanely is to do the 
request_firmware() before suspending.

I _really_ don't understand why this keeps coming up all the time. It's a 
no-brainer, with a single possible solution, yet people continue to whine 
about it.

			Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Mon, 2008-07-14 at 17:40 -0700, Linus Torvalds wrote:
> 
> On Mon, 14 Jul 2008, David Woodhouse wrote:
> > 
> > I see no real point in that. If you have userspace to load modules, then
> > you have userspace to load firmware.
> 
> David, the case is different: the module is loaded _across_ the 
> suspend/resume, but the device needs to have its firmware reloaded at 
> resume time because it lost power.

Either I put my reply in response to the wrong quotation, or you thought
I did.

I intended the above to be a response to the "please put firmware _into_
the .ko file" request. Not to the suspend/resume thing.

Yes, I agree that suspend/resume is a non-issue.

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Mon, 2008-07-14 at 17:31 -0700, Yinghai Lu wrote:
> make[1]: *** No rule to make target
> `include/config/builtin/firmware/dir.h', needed by
> `firmware/ql2100_fw.bin.gen.S'.  Stop.
> make: *** [firmware] Error 2

Oh, pants. Sorry, I renamed the CONFIG_BUILTIN_FIRMWARE option to
CONFIG_EXTRA_FIRMWARE some time back, but all my own test builds had the
include/config/builtin/firmware/dir.h file still lying around so I
didn't notice that I'd forgotten to change that manual dependency. And
evidently nobody using linux-next actually tried to include arbitrary
_other_ firmware into their tree. This patch should fix it...

(Why _do_ we have stray files left in include/config/ when config
options go away, btw? Shouldn't they get deleted?)

----
Subject: firmware: Correct dependency on CONFIG_EXTRA_FIRMWARE_DIR

When CONFIG_EXTRA_FIRMWARE_DIR gets changed, the filename in the .S file
(which uses .incbin to include the binary) needs to change. When we
renamed the BUILTIN_FIRMWARE_DIR option to EXTRA_FIRMWARE_DIR, we forgot
to update the manual dependency in firmware/Makefile, so it was
depending on a non-existent file in include/config/

Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>

diff --git a/firmware/Makefile b/firmware/Makefile
index 5a1e61c..ec4f0cc 100644
--- a/firmware/Makefile
+++ b/firmware/Makefile
@@ -146,7 +146,7 @@ $(patsubst %,$(obj)/%.gen.S, $(fw-shipped-y)): %: $(wordsize_deps) \
 		| $(objtree)/$$(dir %)
 	$(call cmd,fwbin,$(patsubst %.gen.S,%,$@))
 $(patsubst %,$(obj)/%.gen.S, $(fw-external-y)): %: $(wordsize_deps) \
-		include/config/builtin/firmware/dir.h | $(objtree)/$$(dir %)
+		include/config/extra/firmware/dir.h | $(objtree)/$$(dir %)
 	$(call cmd,fwbin,$(fwabs)/$(patsubst $(obj)/%.gen.S,%,$@))
 
 # The .o files depend on the binaries directly; the .S files don't.


-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Yinghai Lu]

On Mon, Jul 14, 2008 at 5:23 PM, David Woodhouse <dwmw2@infradead.org> wrote:
> On Mon, 2008-07-14 at 16:53 -0700, Yinghai Lu wrote:
>> so after this patchset, if i put qllogic fw in firmware/, that fw will
>> work or not...
>
> The qlogic driver already uses request_firmware(), like most modern
> drivers do. If you want to include that firmware in your kernel image
> (assuming it's in /lib/firmware already), do something like this:
>
>        CONFIG_EXTRA_FIRMWARE="ql2500_fw.bin"
>        CONFIG_EXTRA_FIRMWARE_DIR=/lib/firmware
>
> Then it'll find that firmware file and pull it in to your kernel, and
> you can set CONFIG_SCSI_QLA_FC=y and expect it to work. Select the right
> firmware for your device, of course.

  MK_FW   firmware/tigon/tg3.bin.gen.S
make[1]: *** No rule to make target `firmware/tigon/tg3.bin', needed
by `firmware/tigon/tg3.bin.gen.o'.  Stop.
make: *** [firmware] Error 2

YH

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[david]

On Mon, 14 Jul 2008, Linus Torvalds wrote:

> On Mon, 14 Jul 2008, david@lang.hm wrote:
>>
>> there was also the issue that was raised about how to handle firmware during
>> suspend/resume. I don't remember seeing a happy solution to that one.
>
> I don't understand why this even keeps coming up.
>
> The only reliable way to handle it is to load the firmware before the
> suspend happens - preferably by simply never unloading it in the first
> place.
>
> Anything else is fraught with problems.
>
> Stuff that expects to eject itself anyway(*) over a suspend event can
> obviously decide to throw it away and require a reload at resume time (by
> virtue of simply re-discovering the whole device!).
>
> Everybody else should just keep the firmware in memory in the driver.
>
> Does it waste some ram? Sure. Tough.

I agree with this, but the proponents of the seperate firmware are listing 
the fact that the firmware doesn't tie up ram as one of the big reasons 
for making the change.

David Lang

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Mon, 2008-07-14 at 17:50 -0700, Yinghai Lu wrote:
>   MK_FW   firmware/tigon/tg3.bin.gen.S
> make[1]: *** No rule to make target `firmware/tigon/tg3.bin', needed
> by `firmware/tigon/tg3.bin.gen.o'.  Stop.
> make: *** [firmware] Error 2

I didn't include the tg3 patch in what I just asked Linus to pull. Are
you working from linux-next?

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Yinghai Lu]

On Mon, Jul 14, 2008 at 5:52 PM, David Woodhouse <dwmw2@infradead.org> wrote:
> On Mon, 2008-07-14 at 17:50 -0700, Yinghai Lu wrote:
>>   MK_FW   firmware/tigon/tg3.bin.gen.S
>> make[1]: *** No rule to make target `firmware/tigon/tg3.bin', needed
>> by `firmware/tigon/tg3.bin.gen.o'.  Stop.
>> make: *** [firmware] Error 2
>
> I didn't include the tg3 patch in what I just asked Linus to pull. Are
> you working from linux-next?

linus tree.

YH

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Mon, 14 Jul 2008, david@lang.hm wrote:
> > 
> > Does it waste some ram? Sure. Tough.
> 
> I agree with this, but the proponents of the seperate firmware are listing the
> fact that the firmware doesn't tie up ram as one of the big reasons for making
> the change.

That's a totally bogus argument.

The fact is, if you build it into your module, you'll waste at _least_ as 
much ram as if you just load it once at module load time.

So there is no actual valid reason to object to "request_firmware()".

I don't know why people get confused about this. I suspect that people 
kind of expect that since they need to reload the firmware when resuming 
the device, they should also do the "request_firmware()" at resume time.

Maybe it's worth explicitly documenting that request_firmware()/release() 
should be done as a module init/exit (or a device detect-eject) time 
option. Quite frankly, I think the current firmware docs are actually 
actively misleading, because they link the request_firmware() with the 
copying to device: quoting from Documentation/firmware_class/README:

	 High level behavior (driver code):
	 ==================================
	
	         if(request_firmware(&fw_entry, $FIRMWARE, device) == 0)
	                copy_fw_to_device(fw_entry->data, fw_entry->size);
	         release(fw_entry);

and that is a fundamentally broken world-view.

The logic _should_ be that the firmware is requested at module init or 
device discovery, and the release is done at module exit or device eject. 

The "request_firmware()" should absolutely *not* be mentally tied to 
"copy_fw_to_device" at all. They are very distinct issues, and in fact 
must be totally separate for any driver that supports hotplug.

		Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Yinghai Lu]

On Mon, Jul 14, 2008 at 5:50 PM, David Woodhouse <dwmw2@infradead.org> wrote:
> On Mon, 2008-07-14 at 17:31 -0700, Yinghai Lu wrote:
>> make[1]: *** No rule to make target
>> `include/config/builtin/firmware/dir.h', needed by
>> `firmware/ql2100_fw.bin.gen.S'.  Stop.
>> make: *** [firmware] Error 2
>
> Oh, pants. Sorry, I renamed the CONFIG_BUILTIN_FIRMWARE option to
> CONFIG_EXTRA_FIRMWARE some time back, but all my own test builds had the
> include/config/builtin/firmware/dir.h file still lying around so I
> didn't notice that I'd forgotten to change that manual dependency. And
> evidently nobody using linux-next actually tried to include arbitrary
> _other_ firmware into their tree. This patch should fix it...
>
> (Why _do_ we have stray files left in include/config/ when config
> options go away, btw? Shouldn't they get deleted?)
>
> ----
> Subject: firmware: Correct dependency on CONFIG_EXTRA_FIRMWARE_DIR
>
> When CONFIG_EXTRA_FIRMWARE_DIR gets changed, the filename in the .S file
> (which uses .incbin to include the binary) needs to change. When we
> renamed the BUILTIN_FIRMWARE_DIR option to EXTRA_FIRMWARE_DIR, we forgot
> to update the manual dependency in firmware/Makefile, so it was
> depending on a non-existent file in include/config/
>
> Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>
>
> diff --git a/firmware/Makefile b/firmware/Makefile
> index 5a1e61c..ec4f0cc 100644
> --- a/firmware/Makefile
> +++ b/firmware/Makefile
> @@ -146,7 +146,7 @@ $(patsubst %,$(obj)/%.gen.S, $(fw-shipped-y)): %: $(wordsize_deps) \
>                | $(objtree)/$$(dir %)
>        $(call cmd,fwbin,$(patsubst %.gen.S,%,$@))
>  $(patsubst %,$(obj)/%.gen.S, $(fw-external-y)): %: $(wordsize_deps) \
> -               include/config/builtin/firmware/dir.h | $(objtree)/$$(dir %)
> +               include/config/extra/firmware/dir.h | $(objtree)/$$(dir %)
>        $(call cmd,fwbin,$(fwabs)/$(patsubst $(obj)/%.gen.S,%,$@))
>
>  # The .o files depend on the binaries directly; the .S files don't.
>
>
> --
> dwmw2
>

thanks. it works

  MK_FW   firmware/ql2100_fw.bin.gen.S
  MK_FW   firmware/ql2300_fw.bin.gen.S
  MK_FW   firmware/ql2400_fw.bin.gen.S
  MK_FW   firmware/ql2200_fw.bin.gen.S
  MK_FW   firmware/ql2322_fw.bin.gen.S
  MK_FW   firmware/ql2500_fw.bin.gen.S
  AS      firmware/ql2100_fw.bin.gen.o
  AS      firmware/ql2300_fw.bin.gen.o
  AS      firmware/ql2400_fw.bin.gen.o
  AS      firmware/ql2200_fw.bin.gen.o
  AS      firmware/ql2322_fw.bin.gen.o
  AS      firmware/ql2500_fw.bin.gen.o
  LD      firmware/built-in.o


calling  qla2x00_module_init+0x0/0x131
QLogic Fibre Channel HBA Driver: 8.02.01-k4
ACPI: PCI Interrupt 0000:0c:02.0[A] -> GSI 57 (level, low) -> IRQ 57
qla2xxx 0000:0c:02.0: Found an ISP2312, irq 57, iobase 0xffffc2002d00c000
qla2xxx 0000:0c:02.0: Configuring PCI space...
qla2xxx 0000:0c:02.0: Configure NVRAM parameters...
qla2xxx 0000:0c:02.0: Verifying loaded RISC code...
firmware: using built-in firmware ql2300_fw.bin
qla2xxx 0000:0c:02.0: Allocated (412 KB) for firmware dump...

YH

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Mon, 2008-07-14 at 17:50 -0700, Yinghai Lu wrote:
>   MK_FW   firmware/tigon/tg3.bin.gen.S
> make[1]: *** No rule to make target `firmware/tigon/tg3.bin', needed
> by `firmware/tigon/tg3.bin.gen.o'.  Stop.
> make: *** [firmware] Error 2

I apologise; a line intended for the tg3 conversion accidentally slipped
into firmware/Makefile along with the dabusb patch, when I cherry-picked
from the old tree to omit the drivers/net patches. Again, it didn't show
up in my own testing because I already _had_ the file in question.

(Sorry, Linus. This and the previous fix are now in the for-2.6.27
branch of the same repo as before:
	git://git.infradead.org/users/dwmw2/firmware-2.6.git for-2.6.27
)

----
Subject: Fix accidental reference to tg3 firmware

We're not updating the tg3 driver to use request_firmware() yet, but a
reference to its firmware accidentally slipped in as part of commit
c4667746 ("dabusb: use request_firmware()"). Remove it again.

Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>

diff --git a/firmware/Makefile b/firmware/Makefile
index 10028ac..8dbdd3f 100644
--- a/firmware/Makefile
+++ b/firmware/Makefile
@@ -34,8 +34,6 @@ fw-shipped-$(CONFIG_SND_SB16_CSP) += sb16/mulaw_main.csp sb16/alaw_main.csp \
 				     sb16/ima_adpcm_capture.csp
 fw-shipped-$(CONFIG_SND_YMFPCI) += yamaha/ds1_ctrl.fw yamaha/ds1_dsp.fw \
 				   yamaha/ds1e_ctrl.fw
-fw-shipped-$(CONFIG_TIGON3) += tigon/tg3.bin tigon/tg3_tso.bin \
-			       tigon/tg3_tso5.bin
 fw-shipped-$(CONFIG_USB_DABUSB) += dabusb/firmware.fw dabusb/bitstream.bin
 fw-shipped-$(CONFIG_USB_EMI26) += emi26/loader.fw emi26/firmware.fw \
 				  emi26/bitstream.fw


-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[david]

On Mon, 14 Jul 2008, Linus Torvalds wrote:

> On Mon, 14 Jul 2008, david@lang.hm wrote:
>>>
>>> Does it waste some ram? Sure. Tough.
>>
>> I agree with this, but the proponents of the seperate firmware are listing the
>> fact that the firmware doesn't tie up ram as one of the big reasons for making
>> the change.
>
> That's a totally bogus argument.
>
> The fact is, if you build it into your module, you'll waste at _least_ as
> much ram as if you just load it once at module load time.
>
> So there is no actual valid reason to object to "request_firmware()".

you misunderstood me. the people pushing request_firmware() are doing so 
on the basis that they won't have to use kernel ram to hold the firmware. 
the people pushing for having the option of building the firmware into the 
module are acknowleding that this may use a little more ram, but they see 
it as being more reliable.

> I don't know why people get confused about this. I suspect that people
> kind of expect that since they need to reload the firmware when resuming
> the device, they should also do the "request_firmware()" at resume time.

according to David W they would, becouse the driver would not keep the 
firmware in kernel memory after it's initialized, so if you need to 
reinitialize the hardware after a resume you would need to do 
request_firmware() at resume time (the option of doing request_firmware() 
just before going to sleep has been floated as an option to avoid this 
problem)

> Maybe it's worth explicitly documenting that request_firmware()/release()
> should be done as a module init/exit (or a device detect-eject) time
> option. Quite frankly, I think the current firmware docs are actually
> actively misleading, because they link the request_firmware() with the
> copying to device: quoting from Documentation/firmware_class/README:
>
> 	 High level behavior (driver code):
> 	 ==================================
>
> 	         if(request_firmware(&fw_entry, $FIRMWARE, device) == 0)
> 	                copy_fw_to_device(fw_entry->data, fw_entry->size);
> 	         release(fw_entry);
>
> and that is a fundamentally broken world-view.

but that is what David W has been telling everyone is one of the big 
benifits of using request_firmware(), that the memory gets released once 
the device is initialized. (as always, it's possible I have misunderstood 
the argument, but I can't see any other way that it would save memory)

> The logic _should_ be that the firmware is requested at module init or
> device discovery, and the release is done at module exit or device eject.

this would go a long way to satisfying the people who have been objecting 
to the request_firmware() push. It would solve the various 
reinitialization problems (at resume, and at other times for some 
drivers).

the only thing it doesn't satisfy is the people who don't want to be 
forced to have seperate firmware files out on the filesystem.

As I understand the arguments, the reason to not force the firmeware files 
to be seperate is that it adds complexity and introduces the possibility 
that the firmware may be incompatible with the driver, forcing the 
creation of some sort of userspace/filesystem versioning.

the reson to force the firmware files to be seperate is that it avoids 
using kernel memory to hold them and it allows for the firmware to be 
removed from the kernel tree entirely and shipped seperately (apparently 
Fedora is eager to do the latter for perceived licensing reasons)

> The "request_firmware()" should absolutely *not* be mentally tied to
> "copy_fw_to_device" at all. They are very distinct issues, and in fact
> must be totally separate for any driver that supports hotplug.

in the last flamewar it seemed pretty clear that the expectation was that 
a hotplug event would trigger a new request_firmware() call out to 
userspace.

David Lang

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Miller]

From: Andrew Morton <akpm@linux-foundation.org>
Date: Mon, 14 Jul 2008 16:41:19 -0700

> On Mon, 14 Jul 2008 16:23:26 -0700
> David Woodhouse <dwmw2@infradead.org> wrote:
> 
> > Linus, please pull from the for-2.6.27 branch of:
> > 	git://git.infradead.org/users/dwmw2/firmware-2.6.git for-2.6.27
> 
> The firmware flamewars seem to have subsided lately.  Is everyone happy
> with this now?

I'm not happy for network drivers, although I'm happy to see that
David respected out NACKs on tg3/bnx2/etc. and didn't include those
bits.

I still haven't seen an answer to the chip reset issues.  The argument
has been that this stuff is going to save kernel memory when the
driver isn't loading firmware, but that's not really possible.

When a lot of these network drivers reset, due to a transmit timeout
or other exception, we need to load the firmware again.

So this means, that if request_firmware() dies or fails for some
reason in these scenerios, we can't reset the network card.  Something
as simple as a memory or other allocation failure deep inside of
request_firmware() means we lose the networking interface.

To me this seems like a very non-resilient setup.  It makes sense
to just keep the firmware in-core so we can load it without having
to even think about possible failures.

And that to me basically makes this transformation pointless.  It's
in fact a regression.

Furthermore, the issue of suspend and resume was brought up.  What if
request_firmware() fails then?  And can request_firmware() even be
allowed in that context?  What if the disk on which the firmware
resides hasn't been resumed yet?

In response to the suspend/resume queries, a lot of special voodoo was
mentioned that perhaps would allow request_firmware() to get invoked
in the correct context and with the firmware partition's block device
resumed already.

But who the heck is going to write all of that code and fix up all
of these drivers so that they can resume reliably again?

And for what?

None of the people who wrote and maintain these effected networking
drivers want these changes installed.  And they are the ones who will
have to diagnose strange request_firmware() failures that lead to
non-functioning devices during resume or after a transmit timeout.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

David Woodhouse wrote:
> Linus, please pull from the for-2.6.27 branch of:
> 	git://git.infradead.org/users/dwmw2/firmware-2.6.git for-2.6.27
> 
> This does the following (not quite in this order):
> 
>  - Provides a method for firmware files to be built into the kernel, so
>    that drivers can be updated to use request_firmware() without forcing
>    people to use an initrd.
> 
>  - In doing that, mark fw->data as 'const', and fix a few drivers to cope.
> 
>  - Update a bunch of drivers to use request_firmware(), and move their
>    existing firmware, previously hardcoded in all kinds of interesting
>    ways, into the firmware/ directory of the kernel tree.
> 
>  - Add a FIRMWARE_IN_KERNEL config option which allows the firmware for
>    the above (and below) drivers to continue to be built into the kernel.
> 
>  - Add a simple method for representing Intel HEX records in binary form,
>    since a bunch of drivers need that kind of thing. And a tool to
>    convert HEX files into that binary form, for the kernel to load.
> 
>  - Drop special cases from a few drivers which could either use
>    request_firmware() or build their firmware in; they don't need to
>    do that for themselves any more.
> 
>  - Implement 'make firmware_install' target, which installs all firmware
>    shipped with the kernel.
> 
>  - Modify 'make modules_install' to install any firmware required by the
>    modules it installs (where that firmware is shipped in the kernel tree).
> 
> There are more drivers to be converted to request_firmware() -- notably
> drivers/net and drivers/scsi, but it's probably best to ensure that the
> infrastructure is all in place and working correctly before we update
> those. (Although since modern drivers have been using request_firmware()
> for some time, I don't think there's likely to be a problem there; we'll
> probably clean up the stragglers in time for 2.6.28.)
> 
> David Howells (5):
>       Fix a const pointer usage warning in the Digigram VX soundcard driver
>       Fix a const assignment in moxa_load_fw()
>       Fix a const pointer error in the Conexant cx23418 MPEG encoder driver
>       Fix a const pointer usage warning in the Digigram pcxhr soundcard driver
>       Fix a const pointer usage warning in the Digigram miXart soundcard driver
> 
> Full shortlog and diffstat below, although the really interesting part
> is probably between commits fd4f80de..59890f74 -- seen at 
> http://git.infradead.org/users/dwmw2/firmware-2.6.git?a=shortlog;h=59890f74


After the long threads, you would think you would have at least 
mentioned controversy attached to these changes?

Or perhaps mention that you are specifically excluding the ability to 
build firmware into modules -- the most reliable method available for 
those who use modules, MORE reliable than a system where firmware and 
kernel module are separated.

Or some of the objections raised, that were not met?

Or even _acknowledge_ that these changes have a high probability of 
creating a non-working driver, and therefore advising distros with Big 
Fat Warnings and Be Carefuls sprinkled liberally?

Or perhaps describe the outside-the-kernel migration path for distros?

Isn't it silly to create firmware/ top-level dir, when the firmware more 
naturally lives in the same dir as the driver to which it is intimately 
tied?  We'll just have to start creating firmware/net firmware/char 
firmware/media etc. after a while.

What about keeping binary objects as binary objects?  Surely git can 
handle binary file.  And you are already converting each firmware's data 
from format A to format B.  Might as well convert it to unpacked, 
un-ASCII'd format that will be loaded directly, as that can be easier in 
some cases.

I think it's just downright shady to push this crap without mentioning 
any of the swirl of issues surrounding this, that are quite relevant to 
its inclusion.

It should be obvious that building firmware into a kernel module (or the 
kernel itself, if you don't use modules) is _the_ most reliable method 
of ensuring your driver will work [if it requires firmware].

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Miller]

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Mon, 14 Jul 2008 17:33:58 -0700 (PDT)

> The only reliable way to handle it is to load the firmware before the 
> suspend happens - preferably by simply never unloading it in the first 
> place.

That's what we were told was the huge benefit of moving some
of these drivers over to this request_firmware() stuff, it would
save memory.

To me, especially for the networking drivers for which I am familiar,
this transformation seems completely pointless from a technical
perspective.  In fact it's a regression as-is.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Miller]

From: david@lang.hm
Date: Mon, 14 Jul 2008 17:51:37 -0700 (PDT)

> I agree with this, but the proponents of the seperate firmware are listing 
> the fact that the firmware doesn't tie up ram as one of the big reasons 
> for making the change.

Exactly.

Otherwise these firmware changes are utterly pointless.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Arjan van de Ven wrote:
> On Mon, 14 Jul 2008 16:41:19 -0700
> Andrew Morton <akpm@linux-foundation.org> wrote:
> 
>> On Mon, 14 Jul 2008 16:23:26 -0700
>> David Woodhouse <dwmw2@infradead.org> wrote:
>>
>>> Linus, please pull from the for-2.6.27 branch of:
>>> 	git://git.infradead.org/users/dwmw2/firmware-2.6.git
>>> for-2.6.27
>> The firmware flamewars seem to have subsided lately.  Is everyone
>> happy with this now?
>>
> 
> this seems to have left the contentious pieces out....

Correction:  It seems to have completely ignored contentious issues, 
which is not the same thing.

It -removes- the ability to build firmware into the kernel module -- a 
currently deployed, highly reliable configuration.

I'm not surprised that fact was snipped in David's submission -- 
wouldn't want to mention that you are unconditionally replacing a 
reliable method of firmware loading with a less reliable one.

Building the firmware into the kernel is far less useful than it 
appears:  often users lack the ability to maintain their own vmlinuz 
kernel image build, but do build updated drivers.  e.g. driver disks for 
Red Hat Enterprise Linux.

IMO the newly added /inability/ to build firmware into kernel modules is 
a clear regression.

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Mon, 14 Jul 2008, david@lang.hm wrote:

> On Mon, 14 Jul 2008, Linus Torvalds wrote:
> > 
> > That's a totally bogus argument.
> 
> you misunderstood me. the people pushing request_firmware() are doing so on
> the basis that they won't have to use kernel ram to hold the firmware. the
> people pushing for having the option of building the firmware into the module
> are acknowleding that this may use a little more ram, but they see it as being
> more reliable.

I'm just saying that it's a totally bogus argument to claim that it takes 
less memory - Either way.

As to reliability, I don't buy that, especially with a generic interface, 
and with a way to link the thing in-kernel anyway. Using common 
infrastructure is going to be more reliable.

> > I don't know why people get confused about this. I suspect that people
> > kind of expect that since they need to reload the firmware when resuming
> > the device, they should also do the "request_firmware()" at resume time.
> 
> according to David W they would, becouse the driver would not keep the
> firmware in kernel memory after it's initialized

And if so, David W is a total moran, and shouldn't have been doing this.

The fact is, there _are_ good arguments for request_firmware(), but they 
have nothing what-so-ever to do with memory use or anything like that.

The argument for request_firmware() is that it's a good _single_ interface 
to the whole firmware issue, allowing us to split up the driver from the 
firmware without every driver having to do some hack of its own.

Not memory use.

So next time you see somebody arguing about memory use (either way), just 
slap them, and tell them Linus told you to.

			Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

David Woodhouse wrote:
> On Mon, 2008-07-14 at 17:06 -0700, david@lang.hm wrote:
>> On Mon, 14 Jul 2008, Arjan van de Ven wrote:
>>
>>> On Mon, 14 Jul 2008 16:41:19 -0700
>>> Andrew Morton <akpm@linux-foundation.org> wrote:
>>>
>>>> On Mon, 14 Jul 2008 16:23:26 -0700
>>>> David Woodhouse <dwmw2@infradead.org> wrote:
>>>>
>>>>> Linus, please pull from the for-2.6.27 branch of:
>>>>> 	git://git.infradead.org/users/dwmw2/firmware-2.6.git
>>>>> for-2.6.27
>>>> The firmware flamewars seem to have subsided lately.  Is everyone
>>>> happy with this now?
>>>>
>>> this seems to have left the contentious pieces out....
>> almost, the one thing that this could have done would be to offer an 
>> option to bundle the firmware with the module. currently it offers the 
>> option to load the firmware at the same time as the module, not _quite_ 
>> the same thing.
> 
> I see no real point in that. If you have userspace to load modules, then
> you have userspace to load firmware.

You have been provided with several counter-examples here.

Driver disks, initrd, and other such constructs are not necessarily the 
land of happy, fat and healthy userspace that you believe it universally 
to be.  Image build scripts need to be aware that they need to copy 
firmware.  Some already know -- but many, particularly with networking 
-- were such simple one-driver affairs that nobody bothered to script in 
the extra smarts.


> We made 'make modules_install' install the firmware in /lib/firmware for
> you at the same time as it installs the modules, to avoid problems if
> people forgot to run 'make firmware_install'. That really ought to be
> enough.

And it was like pulling teeth, just get that change in, ensuring the 
normal build process will not suddenly produce non-working drivers... 
which it did for several kernel hackers.  As predicted.


> I know Jeff complained that it wasn't, and insisted that he _needed_ to
> be able to scp a single .ko file around which contained both, and the
> world was broken if he could not -- but I disagree with him.

It's simple logic:  existing systems DO copy around modules.  You seem 
to have forgotten an example not so easily derided:  driver disks and 
their build scripts, which are used all over the place.

Particularly in cases, like enterprise distros, where you are updating a 
driver but never touch the main kernel image.

That will continue to be true even when enterprise distros are based off 
of 2.6.27 or later.


> But since I wanted this tree to be uncontentious and obviously the
> correct thing to do, I've dropped the drivers/net changes for now
> anyway.
> 
> It's odd that this request has suddenly come out of the blue when we've
> been using request_firmware() from modules for years already.

Why is it out of the blue to worry about a working driver suddenly 
ceasing to work?

This has nothing to do with request_firmware() itself -- its about 
having the infrastructure in place so that users do not notice the switch.

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Mon, 14 Jul 2008, Jeff Garzik wrote:
> 
> IMO the newly added /inability/ to build firmware into kernel modules is a
> clear regression.

IMO you're being stupid.

How about explainign why it makes any difference what-so-ever?

If you can load the module, you can load the firmware. Claiming anything 
else is just _stupid_.

And if you cannot see the advantage of having _one_ interface to firmware 
loading, instead of each driver picking a random approach, I don't know 
what to say.

The fact that some people have been confused and apparently blathered 
about memory savings is totally immaterial. Ignore the stupid people.

			Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Miller]

From: Jeff Garzik <jeff@garzik.org>
Date: Mon, 14 Jul 2008 22:12:24 -0400

> David Woodhouse wrote:
> > But since I wanted this tree to be uncontentious and obviously the
> > correct thing to do, I've dropped the drivers/net changes for now
> > anyway.
> > 
> > It's odd that this request has suddenly come out of the blue when we've
> > been using request_firmware() from modules for years already.
> 
> Why is it out of the blue to worry about a working driver suddenly 
> ceasing to work?
> 
> This has nothing to do with request_firmware() itself -- its about 
> having the infrastructure in place so that users do not notice the switch.

And I want to reiterate my adversion to the folks who keep saying
that being opposed to request_firmware() is just like being opposed
to modules.

That's is very far from the truth.

When module support was added, guess what?  I could still build a
completely static kernel image like I always could.

And in fact, to this day, that's what I personally do because that's
how I like my kernels.

But this request_firmware() change does not allow one to get what he
could get before, which is a completely self-contained driver module
object file.

This is the difference between providing an option and making
something mandatory.  This firmware split up is now mandatory.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Miller]

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Mon, 14 Jul 2008 19:13:16 -0700 (PDT)

> On Mon, 14 Jul 2008, Jeff Garzik wrote:
> > 
> > IMO the newly added /inability/ to build firmware into kernel modules is a
> > clear regression.
> 
> IMO you're being stupid.
> 
> How about explainign why it makes any difference what-so-ever?

Just like Jeff, I think it does.

> If you can load the module, you can load the firmware. Claiming anything 
> else is just _stupid_.

It's about having one foo.ko file that is self contained.

We've always been able to do that.

Just like how when module support was added to the kernel, and we
could still build everything statically into one kernel image if we
wanted to, we likewise should still be able to get everything
necessary for a driver module into a single foo.ko file.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Linus Torvalds wrote:
> On Mon, 14 Jul 2008, Jeff Garzik wrote:
>> IMO the newly added /inability/ to build firmware into kernel modules is a
>> clear regression.
> 
> IMO you're being stupid.
> 
> How about explainign why it makes any difference what-so-ever?
> 
> If you can load the module, you can load the firmware. Claiming anything 
> else is just _stupid_.

False -- for every router build script, driver disk build script, etc. 
that has not yet been updated to copy both a module and the 
/lib/firmware that the kernel tree is newly spitting out.

And for every case, the breakage is completely silent, until one boots 
into a firmware-less image and loads drivers.


> And if you cannot see the advantage of having _one_ interface to firmware 
> loading, instead of each driver picking a random approach, I don't know 
> what to say.

It's a great approach!  As I've said over and over and over (and now 
over) again.

My complaints are about --not breaking stuff--, not request_firmware().

You have to look at the build process, second stage image builders, 
embedded system image creators, and other "hangers on" that do not look 
and behave like a mainstream Linux distro.

It is a mistake to assume that all systems are _already_ prepared to 
install and digest new /lib/firmware files that the kernel build now 
spits out.

And the consequences of that assumption is a non-working driver, often a 
non-booting system, not just a few quirks here and there.

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Rene Herman]

On 15-07-08 03:52, David Miller wrote:

> From: david@lang.hm Date: Mon, 14 Jul 2008 17:51:37 -0700 (PDT)
> 
>> I agree with this, but the proponents of the seperate firmware are
>> listing the fact that the firmware doesn't tie up ram as one of the
>> big reasons for making the change.
> 
> Exactly.
> 
> Otherwise these firmware changes are utterly pointless.

The point of them is legal. Which everyone knows so could perhaps that 
particular dance be done away with?

The current patches allow to

1) load the firmware from userspace
2) load the firmware from the kernel image (vmlinux)

They lack the ability to load the firmware from the module image.

 From a technical standpoint that seems an odd setup. As a driver author 
you'd either be satisfied with loading the firmware from userspace, or 
if for example you feel you're very much tied to your specific firmware, 
you want it to be as close to you as possible -- in the kernel image 
while you yourself are a module makes little sense. You couldn't for 
example just copy the single module around.

That seems a clear technical point to fix but after that, the point is 
legal. At least for existing drivers, this consists of distributions 
worrying about GPL consequences and wanting to seperate out the existing 
firmware blobs.

They do worry -- whatever those worries might be worth -- so there's the 
point. You may feel it not a good point but that's something other than 
no point existing.

I followed along on these flamewars and I'd say to reject these current 
patches on technical grounds until the ability to build the firmware 
into the module exists and merge it after -- even though people might 
feel there's little or no upside to them, I'd say there's little of no 
downside either once that ability is in.

Rene.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Mon, 2008-07-14 at 19:17 -0700, David Miller wrote:
> When module support was added, guess what?  I could still build a
> completely static kernel image like I always could.
> 
> And in fact, to this day, that's what I personally do because that's
> how I like my kernels.

Good. You can still do precisely that, and build the firmware into your
kernel. You can have exactly what you like. Hey, you can build even
_more_ firmware into your kernel now. You can have NFS-root on devices
you previously had to use an initrd for. hth.

> But this request_firmware() change does not allow one to get what he
> could get before, which is a completely self-contained driver module
> object file.
> 
> This is the difference between providing an option and making
> something mandatory.  This firmware split up is now mandatory.

In all the years we've been using request_firmware(), nobody ever asked
for a way to build the firmware _into_ the .ko file, until now. Why is
it suddenly so important for a small handful of older network drivers,
when nobody else has ever seen the need for it -- even in modern network
drivers?

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Miller]

From: Rene Herman <rene.herman@keyaccess.nl>
Date: Tue, 15 Jul 2008 04:23:14 +0200

> On 15-07-08 03:52, David Miller wrote:
> 
> > From: david@lang.hm Date: Mon, 14 Jul 2008 17:51:37 -0700 (PDT)
> > 
> >> I agree with this, but the proponents of the seperate firmware are
> >> listing the fact that the firmware doesn't tie up ram as one of the
> >> big reasons for making the change.
> > 
> > Exactly.
> > 
> > Otherwise these firmware changes are utterly pointless.
> 
> The point of them is legal.

Thanks for proving something I tried to establish for weeks
but which Alan Cox, David W., and others vehemently denied.

They states that it was being done on a technical basis rather
than being predominantly a legal one.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Rene Herman]

On 15-07-08 04:24, David Miller wrote:

> From: Rene Herman <rene.herman@keyaccess.nl>
> Date: Tue, 15 Jul 2008 04:23:14 +0200
> 
>> On 15-07-08 03:52, David Miller wrote:
>>
>>> From: david@lang.hm Date: Mon, 14 Jul 2008 17:51:37 -0700 (PDT)
>>>
>>>> I agree with this, but the proponents of the seperate firmware are
>>>> listing the fact that the firmware doesn't tie up ram as one of the
>>>> big reasons for making the change.
>>> Exactly.
>>>
>>> Otherwise these firmware changes are utterly pointless.
>> The point of them is legal.
> 
> Thanks for proving something I tried to establish for weeks
> but which Alan Cox, David W., and others vehemently denied.
> 
> They states that it was being done on a technical basis rather
> than being predominantly a legal one.

Yes, they were obstinate or dishonest (I won't say "respectively"). As 
to "proving" though, I cannot prove anything, being a mere observer.

At this point I really believe this discussion should be about the other 
part of my reply -- the point mostly put forward by Jef Garzik about the 
firmware inside the module image. Without that ability, I don't believe 
these are good patches.

_With_ that ability, I myself do. Let's allow allow everyone their own 
level of fear, uncertainty and doubt.

Rene

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Mon, 14 Jul 2008, Jeff Garzik wrote:
> 
> My complaints are about --not breaking stuff--, not request_firmware().

Guys, request_firmware() has been around for about five years now.

If this discussion had happened in 2003, I'd agree.

As it is, I think it's time to just face it - many people do want to have 
a unified interface (and yes, I obviously count myself in that group), and 
you can't just continue do nothing and ignore that.

Your argument seems to be that you don't want to break anything, but 
you've also clearly not wanted to _fix_ anything for the last five years. 

At some point, somebody has to just do it. And five years is _way_ long 
enough.

			Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Mon, 14 Jul 2008, David Miller wrote:
> 
> They states that it was being done on a technical basis rather
> than being predominantly a legal one.

No.

Yes, the original reason for request_firmware() was obviously very much 
partly legal.

HOWEVER.

Once you have a model that is required (for whatever reasons) for some 
drivers, we're much better off using the _same_ model for all drivers, 
whether it is necessary for legal reasons for those other drivers.

Put this way: if you do a distro, you _need_ to support firmware loading 
anyway. And once you do that, it's just annoying how some drivers then do 
something odd and special for the same thing, for no real good reason.

It's like having each wireless driver do their own 802.11 stack. It works, 
and all those driver writers seem to be always convinced that it's the 
right thing to do. But as a network maintainer, do you like it?

Now, replace "802.11 stack" with "firmware loader", and then replace 
"network maintainer" with "kernel maintainer".

And maybe you can see why I like just making everybody do 
request_firmware().

Do you call that a legal reason? Or a technical one?

				Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Miller]

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Mon, 14 Jul 2008 19:33:25 -0700 (PDT)

> 
> 
> On Mon, 14 Jul 2008, Jeff Garzik wrote:
> > 
> > My complaints are about --not breaking stuff--, not request_firmware().
> 
> Guys, request_firmware() has been around for about five years now.

Jeff's not against request_firmware() in an of itself.

He's against the fact that it's not possible to build a
self-contained foo.ko module file any longer, and that
would definitely be possible with using request_firmware().

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Miller]

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Mon, 14 Jul 2008 19:39:03 -0700 (PDT)

> Put this way: if you do a distro, you _need_ to support firmware loading 
> anyway. And once you do that, it's just annoying how some drivers then do 
> something odd and special for the same thing, for no real good reason.

In what way is it annoying?

Most distribution people aren't even aware that drivers like tg3 and
bnx2 even have firmware.  In fact it's self contained and less for
them to worry about.

The correct firmware is right there in the driver, it's what the
driver maintainer tested as a unit, and everything.

It neither gets in the distro guys way nor causes any functional
problems.

It may cause potential legal issues, but that is another matter.

> It's like having each wireless driver do their own 802.11 stack. It works, 
> and all those driver writers seem to be always convinced that it's the 
> right thing to do. But as a network maintainer, do you like it?
> 
> Now, replace "802.11 stack" with "firmware loader", and then replace 
> "network maintainer" with "kernel maintainer".

Distro folks do ship external drivers that have their own 802.11
stack, so this is a very good example in fact :-)

I think the modular vs. static kernel image is a better analogy.
At least with that I have a real choice, with this stuff I suddently
won't.

Regardless of what you or I think about these changes, the fact is
that there were definitely some areas of contention and even some
level of misrepresentation of these changes.  And given that, some
of us feel that merging this in right now was on the premature side.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[david]

On Mon, 14 Jul 2008, Linus Torvalds wrote:

> On Mon, 14 Jul 2008, david@lang.hm wrote:
>
>> On Mon, 14 Jul 2008, Linus Torvalds wrote:
>>>
>>> That's a totally bogus argument.
>>
>> you misunderstood me. the people pushing request_firmware() are doing so on
>> the basis that they won't have to use kernel ram to hold the firmware. the
>> people pushing for having the option of building the firmware into the module
>> are acknowleding that this may use a little more ram, but they see it as being
>> more reliable.
>
> I'm just saying that it's a totally bogus argument to claim that it takes
> less memory - Either way.
>
> As to reliability, I don't buy that, especially with a generic interface,
> and with a way to link the thing in-kernel anyway. Using common
> infrastructure is going to be more reliable.
>
>>> I don't know why people get confused about this. I suspect that people
>>> kind of expect that since they need to reload the firmware when resuming
>>> the device, they should also do the "request_firmware()" at resume time.
>>
>> according to David W they would, becouse the driver would not keep the
>> firmware in kernel memory after it's initialized
>
> And if so, David W is a total moran, and shouldn't have been doing this.
>
> The fact is, there _are_ good arguments for request_firmware(), but they
> have nothing what-so-ever to do with memory use or anything like that.
>
> The argument for request_firmware() is that it's a good _single_ interface
> to the whole firmware issue, allowing us to split up the driver from the
> firmware without every driver having to do some hack of its own.

I don't think anyone objects to request_firmware() as a common interface. 
they object to forcing the firmware to be seperate.

in the monolithic kernel case the people pushing this bowed to preasure 
and improved request_firmware() so that the firmware could be embedded in 
the kernel image as opposed to the initial approach of requireing an 
initramfs.

Jeff and David M are arguing that it should be possible to also embed the 
firmware into the module .ko file. If that option was available I don't 
think there would be opposition to having the one interface. the lack of 
that option is causing them to object and point out cases where it's not 
as good as previously.

David Lang

> Not memory use.
>
> So next time you see somebody arguing about memory use (either way), just
> slap them, and tell them Linus told you to.
>
> 			Linus
>

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Mon, 2008-07-14 at 22:22 -0400, Jeff Garzik wrote:
> You have to look at the build process, second stage image builders, 
> embedded system image creators, and other "hangers on" that do not look 
> and behave like a mainstream Linux distro.
> 
> It is a mistake to assume that all systems are _already_ prepared to 
> install and digest new /lib/firmware files that the kernel build now 
> spits out.

I didn't just assume it. I checked the major distributions -- and they
_do_ check the MODULE_FIRMWARE() tags in the modules they include, and
include the corresponding firmware. The firmware loader has been around
for a _long_ time now, and they've had to support this for almost as
long.

Yes, there are occasional bugs -- and I'm sure there _are_ some minor
distributions or 'image creators' out there which still don't cope.

But that isn't a new problem -- and it's certainly not _caused_ by us
updating a small handful of the older drivers to match what modern
drivers do.

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Mon, 14 Jul 2008, David Miller wrote:
> 
> Jeff's not against request_firmware() in an of itself.
> 
> He's against the fact that it's not possible to build a
> self-contained foo.ko module file any longer, and that
> would definitely be possible with using request_firmware().

.. but is it really that big of a deal?

If it were about not being able to build a self-contained non-modular 
"vmlinux", I'd be upset too.

But once you can load a module, you can load the firmware. You just have 
to _remember_ to move it along with the module.

I mean really, you're the one who was complaining that there was no 
technical reason, but now you're apparently using the "alzheimer's 
defense". 

Besides, it's not even true that foo.ko modules are self-contained. We've 
for years and years (pretty much since day one) had nesting module models, 
where in order to load foo.ko you need to load baz.ko first. Yes, the 
firmware file is named differently, but it _is_ different.

			Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[david]

On Mon, 14 Jul 2008, David Woodhouse wrote:

> On Mon, 2008-07-14 at 19:17 -0700, David Miller wrote:
>> When module support was added, guess what?  I could still build a
>> completely static kernel image like I always could.
>>
>> And in fact, to this day, that's what I personally do because that's
>> how I like my kernels.
>
> Good. You can still do precisely that, and build the firmware into your
> kernel. You can have exactly what you like. Hey, you can build even
> _more_ firmware into your kernel now. You can have NFS-root on devices
> you previously had to use an initrd for. hth.

we recognise that a monolithic kernel can have the firmware embedded in it 
(and I thank you for doing so, this will help me with my Intel wireless 
cards). the objection is that you can't embed firmware into modules.

>> But this request_firmware() change does not allow one to get what he
>> could get before, which is a completely self-contained driver module
>> object file.
>>
>> This is the difference between providing an option and making
>> something mandatory.  This firmware split up is now mandatory.
>
> In all the years we've been using request_firmware(), nobody ever asked
> for a way to build the firmware _into_ the .ko file, until now. Why is
> it suddenly so important for a small handful of older network drivers,
> when nobody else has ever seen the need for it -- even in modern network
> drivers?

the reason the objections are showing up now is that the drivers that 
previously had the firmware embedded in them (so that nobody but the 
driver author needed to even know that there _was_ firmware involved) are 
now being converted to use request_firmware(). that change is breaking the 
invisability of the firmware that they have cultivated. if they had the 
option to keep things as they are (even while you provide the option for 
others to move the firmware out) they could not object like they are.

David Lang

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Mon, 2008-07-14 at 18:50 -0700, David Miller wrote:
> From: Linus Torvalds <torvalds@linux-foundation.org>
> Date: Mon, 14 Jul 2008 17:33:58 -0700 (PDT)
> 
> > The only reliable way to handle it is to load the firmware before the 
> > suspend happens - preferably by simply never unloading it in the first 
> > place.
> 
> That's what we were told was the huge benefit of moving some
> of these drivers over to this request_firmware() stuff, it would
> save memory.

There _are_ RAM benefits for a number of drivers, even when you load the
firmware just once at initialisation time and keep it resident all the
time the device is active (and no, I won't slap myself for saying it).

The tg3 driver is a prime example -- it contains three separate firmware
images for different revisions of the card, but you'll almost never
actually need all three. Mostly you'll only need _one_ of them.

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Linus Torvalds wrote:
> 
> On Mon, 14 Jul 2008, David Miller wrote:
>> Jeff's not against request_firmware() in an of itself.
>>
>> He's against the fact that it's not possible to build a
>> self-contained foo.ko module file any longer, and that
>> would definitely be possible with using request_firmware().
> 
> .. but is it really that big of a deal?
> 
> If it were about not being able to build a self-contained non-modular 
> "vmlinux", I'd be upset too.
> 
> But once you can load a module, you can load the firmware. You just have 
> to _remember_ to move it along with the module.

And failure to "remember" to update associated build image scripts 
silently and without complaint producing a non-working driver.  For 
drivers that, in many cases, have worked for 5 or 10 or more years 
without needing additional files.

So it's not about remembering.  You can't just wish new userland and 
build processes upon the universe.

Heck, you yourself have in the past supported building current kernels 
on older userlands -- this flies in the face of that.


> Besides, it's not even true that foo.ko modules are self-contained. We've 
> for years and years (pretty much since day one) had nesting module models, 
> where in order to load foo.ko you need to load baz.ko first. Yes, the 
> firmware file is named differently, but it _is_ different.

It's not only named differently, it is in a markedly different directory 
hierarchy that isn't automatically picked up alongside *.ko.

But fundamentally, why remove a capability that works, thereby depriving 
system builders of the ability to _choose_ when, if ever, to prefer 
/lib/firmware to built-in firmware?

Let's standardize on a request_firmware() sure, one that does not remove 
that choice.

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Rene Herman]

On 15-07-08 04:39, Linus Torvalds wrote:

> On Mon, 14 Jul 2008, David Miller wrote:
>> They states that it was being done on a technical basis rather
>> than being predominantly a legal one.
> 
> No.
> 
> Yes, the original reason for request_firmware() was obviously very much 
> partly legal.
> 
> HOWEVER.
> 
> Once you have a model that is required (for whatever reasons) for some 
> drivers, we're much better off using the _same_ model for all drivers, 
> whether it is necessary for legal reasons for those other drivers.
> 
> Put this way: if you do a distro, you _need_ to support firmware loading 
> anyway. And once you do that, it's just annoying how some drivers then do 
> something odd and special for the same thing, for no real good reason.

There's little alternative infrastructure in declaring a static array of 
unsigned char though.

Yes, it still makes sense to make everyone use request_firmware() if 
only because these other drivers now all of a sudden have a more 
accessible way of updating their firmwares, but David's objective is 
legal here.

Which is fine.

But the very clear _technical_ objection about this not providing for 
keeping modules and their firmware together for those that really do 
want it to be that way was not at all addressed. Note, this while still 
using request_firmware(), just at the option of the kernel builder with 
the firmware compiled into the module. Right now, this allows the 
firmware to be compiled into the vmlinux only which makes fairly little 
sense if the driver itself is a module.

Rene.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Linus Torvalds wrote:
> 
> On Mon, 14 Jul 2008, Jeff Garzik wrote:
>> My complaints are about --not breaking stuff--, not request_firmware().
> 
> Guys, request_firmware() has been around for about five years now.
> 
> If this discussion had happened in 2003, I'd agree.
> 
> As it is, I think it's time to just face it - many people do want to have 
> a unified interface (and yes, I obviously count myself in that group), and 
> you can't just continue do nothing and ignore that.
> 
> Your argument seems to be that you don't want to break anything, but 
> you've also clearly not wanted to _fix_ anything for the last five years. 
> 
> At some point, somebody has to just do it. And five years is _way_ long 
> enough.

In threads of years past, you've agreed with the plainly obvious -- 
built-in firmware is obviously more reliable, and is more likely to 
produce a working driver.

It doesn't break anything to keep this feature.

Keeping this feature doesn't prevent any other fixes.

Keeping this feature closes the window for potential regressions.

And keeping this feature means Linux users can make a _choice_, rather 
than forcing /lib/firmware on them.

Do you really think embedded system and micro distro authors are in 100% 
agreement that /lib/firmware is best for their situation?  Really?

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Mon, 14 Jul 2008, Jeff Garzik wrote:
> 
> Do you really think embedded system and micro distro authors are in 100%
> agreement that /lib/firmware is best for their situation?  Really?

Umm. Why are you bringing up these total red herrings?

Those embedded systems can just build everything into one image, and are 
much better off using no modules at all.

But how about instead of complaining, help add the infrastructure then, so 
that you can insert those firmware things into modules and still use just 
one single interface? 

Move _forward_, not backwards.

			Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Willy Tarreau]

Hi,

just noticed this thread which frightens me...

On Mon, Jul 14, 2008 at 07:23:55PM -0700, David Woodhouse wrote:
> In all the years we've been using request_firmware(), nobody ever asked
> for a way to build the firmware _into_ the .ko file, until now. Why is
> it suddenly so important for a small handful of older network drivers,
> when nobody else has ever seen the need for it -- even in modern network
> drivers?

Because most often, you need to bring the network driver from any location
to your machine by any method *before* you can transfer anything easily.
It's already painful to have to load modules from userland, please do not
complicate the installation process. I can assure you that it's painful
to boot a system and see that neither network nor USB work. When you
find a way to pass a network driver there through any method (included
writing it onto a CD), you're finally safe. If now you have to iterate
because once you modprobe, you notice that *some* firmware files are
missing, it'll get really really annoying.

You say that it is suddenly important, but I disagree. The only network
driver I have which needs a firmware is myri10ge. And guess what ? Getting
it to work on test platforms was a real hassle. Upon modprobe, the system
would wait 2 minutes (I never understood why we have to wait when a file
is missing, but that's another story of crappy design). From the first
second, I could think "oh sh*t, the firmware again...". The I had to get
the firmware files, hotplug script and also the script to configure /proc
and /sys entries just to be able to load this driver. Fortunately, I could
transfer these files via the other NIC which did not require a firmware to
be loaded.

It was so much hard to get it to work on test platforms that I finally
found how to rebuild the myri10ge driver with the firmware included in
it, and now it has changed my life. Simply modprobe and that's all.
Firmwares are so much annoying that I think that those should just be
optional, and be built into modules *by default*. There should be a
really valid reason for wanting to extract them. Upgradability is not
one, because there are more frequent changes in modules than in firmwares.

Just my 2 cents,
Willy

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Arjan van de Ven]

On Mon, 14 Jul 2008 19:45:57 -0700 (PDT)
David Miller <davem@davemloft.net> wrote:

> From: Linus Torvalds <torvalds@linux-foundation.org>
> Date: Mon, 14 Jul 2008 19:39:03 -0700 (PDT)
> 
> > Put this way: if you do a distro, you _need_ to support firmware
> > loading anyway. And once you do that, it's just annoying how some
> > drivers then do something odd and special for the same thing, for
> > no real good reason.
> 
> In what way is it annoying?
> 
> Most distribution people aren't even aware that drivers like tg3 and
> bnx2 even have firmware.  In fact it's self contained and less for
> them to worry about.

and.. after this patch that still seems to be the case, unless I'm
looking at it really cross eyed.
Nothing in this patch makes it impossible to do so.. or changes the
drivers you mention. in fact it's the opposite almost: it improves
request_firmware() usability significantly. (by allowing vmlinux build
in etc).



-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Willy Tarreau]

On Mon, Jul 14, 2008 at 07:51:55PM -0700, Linus Torvalds wrote:
> 
> 
> On Mon, 14 Jul 2008, David Miller wrote:
> > 
> > Jeff's not against request_firmware() in an of itself.
> > 
> > He's against the fact that it's not possible to build a
> > self-contained foo.ko module file any longer, and that
> > would definitely be possible with using request_firmware().
> 
> .. but is it really that big of a deal?
> 
> If it were about not being able to build a self-contained non-modular 
> "vmlinux", I'd be upset too.
> 
> But once you can load a module, you can load the firmware. You just have 
> to _remember_ to move it along with the module.

In order to transfer it, right now you feed it through a working device.
When that device itself requires firmware to work, you will suddenly
discover that it becomes harder and harder to get any communication device
to work on your target system.

Then, in order to load it, you need to have a properly set up system
(hotplug scripts, firmware at the proper location, etc...). This is
really not convenient on embedded systems or live CD systems. In fact,
among all my machines, only my notebook is currently able to load a
firmware.

Also, there's a versionning problem. It's easy to rename a driver
"foo.ko" => "foo-1.ko" and just load the exact file or hold several
versions there by simply moving a symlink. Having to fiddle around
with firmware files depending on the driver version will get much
harder.

Also, when you're cleaning up a system after installation, how do
you know which firmware files you need to keep and which ones you
can safely remove ?

Just like Jeff and Davem, I really want to be able to continue to
use my drivers like I have done since modules were introduced.
Switching a single-file to a multi-file/multi-dir installation would
be a huge maintenance regression IMHO.

Willy

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Benjamin Herrenschmidt]

On Mon, 2008-07-14 at 17:44 -0700, David Woodhouse wrote:
> On Mon, 2008-07-14 at 17:40 -0700, Linus Torvalds wrote:
> > 
> > On Mon, 14 Jul 2008, David Woodhouse wrote:
> > > 
> > > I see no real point in that. If you have userspace to load modules, then
> > > you have userspace to load firmware.
> > 
> > David, the case is different: the module is loaded _across_ the 
> > suspend/resume, but the device needs to have its firmware reloaded at 
> > resume time because it lost power.
> 
> Either I put my reply in response to the wrong quotation, or you thought
> I did.
> 
> I intended the above to be a response to the "please put firmware _into_
> the .ko file" request. Not to the suspend/resume thing.
> 
> Yes, I agree that suspend/resume is a non-issue.

Besides, request_firmware() from a resume() method is broken. Has always
been and will probably always be.

Ben.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Willy Tarreau]

On Mon, Jul 14, 2008 at 08:29:13PM -0700, Linus Torvalds wrote:
> 
> 
> On Mon, 14 Jul 2008, Jeff Garzik wrote:
> > 
> > Do you really think embedded system and micro distro authors are in 100%
> > agreement that /lib/firmware is best for their situation?  Really?
> 
> Umm. Why are you bringing up these total red herrings?

It's not red herring, Linus. I (as a user, as an embedded system developer
and as a micro distro maintainer) do really care.

> Those embedded systems can just build everything into one image, and are 
> much better off using no modules at all.

While it is sometimes acceptable to have everything built into one image,
sometimes your embedded system is modular in that the same image works
on several hardware variants. In this case, it's largely preferable to
work with modules.

> But how about instead of complaining, help add the infrastructure then, so 
> that you can insert those firmware things into modules and still use just 
> one single interface? 

I think that would make everyone happy. It would be worth taking a
look at myri10ge version 1.4.2 to see how they managed to support
both mechanisms.

> Move _forward_, not backwards.

This is just a matter of point of view. From my POV, having to feed
several unrelated files into a system to be able to get it to support
any device is moving backwards, compared to the currently clean all-in-1
driver model.

BTW, I still did not understand what existing problem we are trying
to solve by this move "forward".

Willy

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Mon, 2008-07-14 at 21:56 -0700, Arjan van de Ven wrote:
> On Mon, 14 Jul 2008 19:45:57 -0700 (PDT)
> David Miller <davem@davemloft.net> wrote:
> 
> > From: Linus Torvalds <torvalds@linux-foundation.org>
> > Date: Mon, 14 Jul 2008 19:39:03 -0700 (PDT)
> > 
> > > Put this way: if you do a distro, you _need_ to support firmware
> > > loading anyway. And once you do that, it's just annoying how some
> > > drivers then do something odd and special for the same thing, for
> > > no real good reason.
> > 
> > In what way is it annoying?
> > 
> > Most distribution people aren't even aware that drivers like tg3 and
> > bnx2 even have firmware.  In fact it's self contained and less for
> > them to worry about.
> 
> and.. after this patch that still seems to be the case, unless I'm
> looking at it really cross eyed.
> Nothing in this patch makes it impossible to do so.. or changes the
> drivers you mention.

Well, the HEAD of my tree (and linux-next) does change tg3. I just
haven't asked Linus to pull that part, because I wanted to keep the
contentious part separate from the simple and obvious parts.

(Even though the tg3 fix _is_ simple and obvious, to me. But that
evidently doesn't stop it from being contentious too. So it can wait.)

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Willy Tarreau]

On Mon, Jul 14, 2008 at 07:48:32PM -0700, David Woodhouse wrote:
> On Mon, 2008-07-14 at 22:22 -0400, Jeff Garzik wrote:
> > You have to look at the build process, second stage image builders, 
> > embedded system image creators, and other "hangers on" that do not look 
> > and behave like a mainstream Linux distro.
> > 
> > It is a mistake to assume that all systems are _already_ prepared to 
> > install and digest new /lib/firmware files that the kernel build now 
> > spits out.
> 
> I didn't just assume it. I checked the major distributions -- and they
> _do_ check the MODULE_FIRMWARE() tags in the modules they include, and
> include the corresponding firmware. The firmware loader has been around
> for a _long_ time now, and they've had to support this for almost as
> long.

David, "being around" and "being heavily used" are not the same. My
experience with firmware is limited to WiFi drivers. I've never seen
that annoying thing on a server yet. Not that it does not exist there,
just that fortunately I've not being facing it.

> Yes, there are occasional bugs -- and I'm sure there _are_ some minor
> distributions or 'image creators' out there which still don't cope.
> 
> But that isn't a new problem -- and it's certainly not _caused_ by us
> updating a small handful of the older drivers to match what modern
> drivers do.

Pardon? If by "updating" you mean you keep backwards compatibility, OK.
But if by "updating" you mean "breaking existing behaviour", yes, it
is your change which will cause the problems.

Willy

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Benjamin Herrenschmidt]

> 
> Thanks for proving something I tried to establish for weeks
> but which Alan Cox, David W., and others vehemently denied.
> 
> They states that it was being done on a technical basis rather
> than being predominantly a legal one.

Despite the risk of going against the current here, I do like David
patches for one major reason: I love my kernels with everything
built-in, that's how I can netboot them etc... without having to bother
with installing modules.

David's patches allow me to do that with drivers that would have
-required- disk based firmwares beforehand. It allows things like
rooting off a block device that needs a firmware without an initrd,
etc....

As long as the firmwares are shipped with the kernel tree, I'm happpy,
and I don't care much about building the firmware inside the .ko's. Now,
if somebody care, I suppose it's a feature that can be added fairly
transparently.

Cheers,
Ben.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Tue, 2008-07-15 at 15:15 +1000, Benjamin Herrenschmidt wrote:
> > 
> > Thanks for proving something I tried to establish for weeks
> > but which Alan Cox, David W., and others vehemently denied.
> > 
> > They states that it was being done on a technical basis rather
> > than being predominantly a legal one.
> 
> Despite the risk of going against the current here, I do like David
> patches for one major reason: I love my kernels with everything
> built-in, that's how I can netboot them etc... without having to bother
> with installing modules.

I think you're definitely _with_ the current on that one.

> David's patches allow me to do that with drivers that would have
> -required- disk based firmwares beforehand. It allows things like
> rooting off a block device that needs a firmware without an initrd,
> etc....
> 
> As long as the firmwares are shipped with the kernel tree, I'm happpy,

They don't even need to be shipped with the kernel tree. You can set
CONFIG_EXTRA_FIRMWARE_DIR to point to wherever you keep them
(like /lib/firmware) and it'll pull them in from there. That's _why_
it's useful for drivers which would have required an initrd before, and
which don't have their firmware shipped in the kernel tree.

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Tue, 2008-07-15 at 07:06 +0200, Willy Tarreau wrote:
> This is just a matter of point of view. From my POV, having to feed
> several unrelated files into a system to be able to get it to support
> any device is moving backwards, compared to the currently clean
> all-in-1 driver model.

So you disapprove of having to load mac80211.ko before your wireless NIC
driver? Or sungem_phy.ko before sungem.ko? Or libphy.ko before various
other drivers?

Or having to install /sbin/modprobe before you can load modules?

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Rene Herman]

On 15-07-08 06:56, Arjan van de Ven wrote:

> On Mon, 14 Jul 2008 19:45:57 -0700 (PDT)
> David Miller <davem@davemloft.net> wrote:
> 
>> From: Linus Torvalds <torvalds@linux-foundation.org>
>> Date: Mon, 14 Jul 2008 19:39:03 -0700 (PDT)
>>
>>> Put this way: if you do a distro, you _need_ to support firmware
>>> loading anyway. And once you do that, it's just annoying how some
>>> drivers then do something odd and special for the same thing, for
>>> no real good reason.
>> In what way is it annoying?
>>
>> Most distribution people aren't even aware that drivers like tg3 and
>> bnx2 even have firmware.  In fact it's self contained and less for
>> them to worry about.
> 
> and.. after this patch that still seems to be the case, unless I'm
> looking at it really cross eyed.
> Nothing in this patch makes it impossible to do so.. or changes the
> drivers you mention. in fact it's the opposite almost: it improves
> request_firmware() usability significantly. (by allowing vmlinux build
> in etc).

Only by virtue of those specific drivers having been skipped for now. 
For those that weren't skipped, the only way to have them still be self 
contained is to have the kernel itself compile in all firmware for all 
modules after which the module will find it there. Distributions will of 
course not do this, not does it make any sense for own kernels.

Rene.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[david]

On Mon, 14 Jul 2008, David Woodhouse wrote:

> On Mon, 2008-07-14 at 18:50 -0700, David Miller wrote:
>> From: Linus Torvalds <torvalds@linux-foundation.org>
>> Date: Mon, 14 Jul 2008 17:33:58 -0700 (PDT)
>>
>>> The only reliable way to handle it is to load the firmware before the
>>> suspend happens - preferably by simply never unloading it in the first
>>> place.
>>
>> That's what we were told was the huge benefit of moving some
>> of these drivers over to this request_firmware() stuff, it would
>> save memory.
>
> There _are_ RAM benefits for a number of drivers, even when you load the
> firmware just once at initialisation time and keep it resident all the
> time the device is active (and no, I won't slap myself for saying it).
>
> The tg3 driver is a prime example -- it contains three separate firmware
> images for different revisions of the card, but you'll almost never
> actually need all three. Mostly you'll only need _one_ of them.

and when a card gets hotplugged into the system for a driver that's 
already in memory handling other cards you intend that it will have to go 
out to userspace to find the firmware?

how much memory are you expecting to save this way?

David Lang

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Willy Tarreau]

On Mon, Jul 14, 2008 at 10:32:10PM -0700, David Woodhouse wrote:
> On Tue, 2008-07-15 at 07:06 +0200, Willy Tarreau wrote:
> > This is just a matter of point of view. From my POV, having to feed
> > several unrelated files into a system to be able to get it to support
> > any device is moving backwards, compared to the currently clean
> > all-in-1 driver model.
> 
> So you disapprove of having to load mac80211.ko before your wireless NIC
> driver?

for this one, I don't know, as it is typically something which might
be installed without me noticing.

> Or sungem_phy.ko before sungem.ko?

that one might be more annoying, however, having the same name and being
in the same directory as its brother makes it quite obvious. The ones
which had already caught me were mii, crc32, etc...

> Or libphy.ko before various other drivers?

cf above.

> Or having to install /sbin/modprobe before you can load modules?

David, please do not exagerate. I hoped you were able to see the
difference between being lazy and encountering difficulties to
set up a system.

Willy

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[david]

On Mon, 14 Jul 2008, David Woodhouse wrote:

> On Tue, 2008-07-15 at 07:06 +0200, Willy Tarreau wrote:
>> This is just a matter of point of view. From my POV, having to feed
>> several unrelated files into a system to be able to get it to support
>> any device is moving backwards, compared to the currently clean
>> all-in-1 driver model.
>
> So you disapprove of having to load mac80211.ko before your wireless NIC
> driver? Or sungem_phy.ko before sungem.ko? Or libphy.ko before various
> other drivers?
>
> Or having to install /sbin/modprobe before you can load modules?

I actually have systems that don't have modprobe (Tivo series 1 boxes, 
they have the insmod/rmmod binary , but not modproble)

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[david]

On Mon, 14 Jul 2008, David Woodhouse wrote:

> On Mon, 2008-07-14 at 21:56 -0700, Arjan van de Ven wrote:
>> On Mon, 14 Jul 2008 19:45:57 -0700 (PDT)
>> David Miller <davem@davemloft.net> wrote:
>>
>>> From: Linus Torvalds <torvalds@linux-foundation.org>
>>> Date: Mon, 14 Jul 2008 19:39:03 -0700 (PDT)
>>>
>>>> Put this way: if you do a distro, you _need_ to support firmware
>>>> loading anyway. And once you do that, it's just annoying how some
>>>> drivers then do something odd and special for the same thing, for
>>>> no real good reason.
>>>
>>> In what way is it annoying?
>>>
>>> Most distribution people aren't even aware that drivers like tg3 and
>>> bnx2 even have firmware.  In fact it's self contained and less for
>>> them to worry about.
>>
>> and.. after this patch that still seems to be the case, unless I'm
>> looking at it really cross eyed.
>> Nothing in this patch makes it impossible to do so.. or changes the
>> drivers you mention.
>
> Well, the HEAD of my tree (and linux-next) does change tg3. I just
> haven't asked Linus to pull that part, because I wanted to keep the
> contentious part separate from the simple and obvious parts.
>
> (Even though the tg3 fix _is_ simple and obvious, to me. But that
> evidently doesn't stop it from being contentious too. So it can wait.)

a techinical question here (I know, how dare I ask a technical question in 
the middle of the flamefest ;-)

linus pointed out that the documentation reccomended the
request_firmware();load_firmware();release() approach and stated that that 
approach was the wrong way to do things, instead doing a request_firmware 
early and release when the module is unloaded.

does this patch series follow the documented reccomendation? or does it 
follow the more concervative approach Linus pointed out? (it's far faster 
to ask then to search Internet archives for the patches)

David Lang

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Mon, 2008-07-14 at 22:45 -0700, david@lang.hm wrote:
> linus pointed out that the documentation reccomended the
> request_firmware();load_firmware();release() approach and stated that that 
> approach was the wrong way to do things, instead doing a request_firmware 
> early and release when the module is unloaded.
> 
> does this patch series follow the documented reccomendation? or does it 
> follow the more concervative approach Linus pointed out? (it's far faster 
> to ask then to search Internet archives for the patches)

Mostly it follows the documented recommendation, since most of the
touched drivers are USB drivers and you end up re-enumerating and
starting from scratch on resume anyway. And the remainder are so old
that they don't have suspend/resume support anyway. Remember, we're only
really updating the older drivers; newer drivers tend to use
request_firmware() already, and have done for years.

The only one I remember offhand that loads the firmware and keeps it
around is tg3 -- and I didn't ask Linus to pull that one. In fact, we
only did it that way for tg3 since that driver seems to doing _all_ its
chip reset and firmware reload (including a boatload of udelays) within
a spinlock. That in itself is probably not optimal, but it wasn't really
within the scope of what we were doing to fix it.

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Oliver Neukum]

Am Dienstag 15 Juli 2008 08:03:15 schrieb David Woodhouse:
> Mostly it follows the documented recommendation, since most of the
> touched drivers are USB drivers and you end up re-enumerating and
> starting from scratch on resume anyway. And the remainder are so old
> that they don't have suspend/resume support anyway. Remember, we're only
> really updating the older drivers; newer drivers tend to use
> request_firmware() already, and have done for years.

As for USB that was before persistance. It now allows you to even
support a loss of power without reprobing, provided

a) the devices does not change IDs
b) you have firmware around when reset_resume() is called

	Regards
		Oliver

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Linus Torvalds wrote:
> 
> On Mon, 14 Jul 2008, Jeff Garzik wrote:
>> Do you really think embedded system and micro distro authors are in 100%
>> agreement that /lib/firmware is best for their situation?  Really?
> 
> Umm. Why are you bringing up these total red herrings?
> 
> Those embedded systems can just build everything into one image, and are 
> much better off using no modules at all.

Oh come on, embedded systems use modules all the time, and surely you 
know that.  My router running OpenWRT says

	root@gw:~# ipkg list | grep -c kmod
	91

Therefore, simple kernel replacement can create a regression by doing

1) build 2.6.26 kernel, including all associated kernel module packages 
("kmod-*")
2) install new kernel and associated kmod packages
3) watch system work as expected
4) build 2.6.27 kernel, including all associated kernel module packages
5) install new kernel and associated kmod packages
6) watch system fail

Why fail?  The package file lists for kmod-* ipkg's are unaware of any 
firmware needs newly added in 2.6.27.

So the driver gets packaged, but ipkg build process is completely 
unaware of the additional firmware requirement until a manifest is 
updated.  The normal build process appears to succeed -- yet at next 
boot you see that it failed.

Similar breakage for driver disks.

Similar breakage for older versions of mainstream distros (won't get 
copied into initrd, with obvious results).


> But how about instead of complaining, help add the infrastructure then, so 
> that you can insert those firmware things into modules and still use just 
> one single interface? 
> 
> Move _forward_, not backwards.

Does that mean you agree it's a regression?  :)  Sure, I'd be happy to 
help fix the damage.

I am still surprised this was merged at all in its state, and am a bit 
disappointed.  This one

* removed choice, by removing ability to build firmware into modules

* forced a flag day build process change upon all distros/builders who 
switch to >= 2.6.27.  no build script updates == non-working drivers.

* the legal subtext of these changes was not mentioned at all

Generally, we don't do that.  Generally, we give distros the ability to 
choose between old-way and new-way during a time of transition. 
Generally we make it easy for newer kernels to work on existing systems, 
maximizing the [test] audience.

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Tue, 2008-07-15 at 03:21 -0400, Jeff Garzik wrote:
> * forced a flag day build process change upon all distros/builders who 
> switch to >= 2.6.27.  no build script updates == non-working drivers.

Jeff, we don't usually call it a 'flag day' when it's just the normal
and uninteresting case that you update the upstream version of the
package, and you need minor tweaks to how you package it. That happens
all the time. A general rule of thumb is to stop drinking at least 4
hours before undertaking a full version upgrade of _any_ package you
maintain.

You could argue that converting a wireless driver to use mac80211 would
break ipkg distributions too. And call that a 'flag day'.

But that would be silly. Wouldn't it?

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Ingo Molnar]

* David Woodhouse <dwmw2@infradead.org> wrote:

> On Mon, 2008-07-14 at 17:31 -0700, Yinghai Lu wrote:
> > make[1]: *** No rule to make target
> > `include/config/builtin/firmware/dir.h', needed by
> > `firmware/ql2100_fw.bin.gen.S'.  Stop.
> > make: *** [firmware] Error 2
> 
> Oh, pants. Sorry, I renamed the CONFIG_BUILTIN_FIRMWARE option to 
> CONFIG_EXTRA_FIRMWARE some time back, but all my own test builds had 
> the include/config/builtin/firmware/dir.h file still lying around so I 
> didn't notice that I'd forgotten to change that manual dependency. And 
> evidently nobody using linux-next actually tried to include arbitrary 
> _other_ firmware into their tree. This patch should fix it...

here's another bugreport, i get this build failure with latest -git 
(which includes your fix above):

   CALL    scripts/checksyscalls.sh
   CHK     include/linux/compile.h
 make[1]: *** No rule to make target `firmware/tigon/tg3.bin', needed by `firmware/tigon/tg3.bin.gen.o'.  Stop.
 make: *** [firmware] Error 2

with:

  http://redhat.com/~mingo/misc/config-Tue_Jul_15_09_38_58_CEST_2008.bad

such configs worked before.

(i'd exclude TIGON3 from my qa builds but i've got tg3 hardware.)

	Ingo

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Linus Torvalds wrote:
> On Mon, 14 Jul 2008, david@lang.hm wrote:
>> On Mon, 14 Jul 2008, Linus Torvalds wrote:
>>> That's a totally bogus argument.
>> you misunderstood me. the people pushing request_firmware() are doing so on
>> the basis that they won't have to use kernel ram to hold the firmware. the
>> people pushing for having the option of building the firmware into the module
>> are acknowleding that this may use a little more ram, but they see it as being
>> more reliable.
> 
> I'm just saying that it's a totally bogus argument to claim that it takes 
> less memory - Either way.

Agreed.


> As to reliability, I don't buy that, especially with a generic interface, 
> and with a way to link the thing in-kernel anyway. Using common 
> infrastructure is going to be more reliable.

A more complex, multi-file inter-dependent system is more reliable than 
a single-file driver with built-in firmware, doing the same thing?

Come _on_.

There are good arguments for request_firmware(), but that ain't it.


> The argument for request_firmware() is that it's a good _single_ interface 
> to the whole firmware issue, allowing us to split up the driver from the 
> firmware without every driver having to do some hack of its own.

Agreed.

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

David Woodhouse wrote:
> On Tue, 2008-07-15 at 03:21 -0400, Jeff Garzik wrote:
>> * forced a flag day build process change upon all distros/builders who 
>> switch to >= 2.6.27.  no build script updates == non-working drivers.
> 
> Jeff, we don't usually call it a 'flag day' when it's just the normal
> and uninteresting case that you update the upstream version of the
> package, and you need minor tweaks to how you package it. That happens
> all the time. A general rule of thumb is to stop drinking at least 4
> hours before undertaking a full version upgrade of _any_ package you
> maintain.
> 
> You could argue that converting a wireless driver to use mac80211 would
> break ipkg distributions too. And call that a 'flag day'.
> 
> But that would be silly. Wouldn't it?

Normally, one can upgrade the kernel, not upgrade userland, and happily 
continue working.

With this change, there are several exceptions where this is not the 
case, across all driver types -- media, wan, char, net, scsi, ...

Package manifests, build scripts and other details MUST be updated, or 
these drivers will not work as they did in kernels <= 2.6.26.

That is not an opinion, that is a simple fact of life with the kernel as 
of 50515af207d410c9f228380e529c56f43c3de0bd.

And yes, for the people who have to clean up the mess, I call that a 
flag day.

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Ingo Molnar]

* David Woodhouse <dwmw2@infradead.org> wrote:

> On Tue, 2008-07-15 at 03:21 -0400, Jeff Garzik wrote:
> > * forced a flag day build process change upon all distros/builders 
> > who switch to >= 2.6.27.  no build script updates == non-working 
> > drivers.
> 
> Jeff, we don't usually call it a 'flag day' when it's just the normal 
> and uninteresting case that you update the upstream version of the 
> package, and you need minor tweaks to how you package it. [...]

well, i have no opinion one way or another about the firmware issue, but 
breaking the build and breaking previously working setups is not a 
"minor tweak".

> [...] That happens all the time. [...]

No, you are quite wrong, it does not happen all the time.

It happens occasionally out of 10,000 changes per kernel cycle. It 
happens unintentionally most of the time and it's called a regression.

The difference here is that you seem to have broken my setup fully 
intentionally.

I have no problem with going forward, but i have a problem with breaking 
working setups and i have a problem with forced migration to something 
that doesnt work by default.

If there's a fix i can apply and test i'm a happy camper, but all this 
arguing in these threads with zero patches posted makes me quite 
nervous.

	Ingo

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> If there's a fix i can apply and test i'm a happy camper, but all this 
> arguing in these threads with zero patches posted makes me quite 
> nervous.

ah, Yinghai forwarded me the fix below - i am a happy camper :-)

btw.:

| from the old tree to omit the drivers/net patches. Again, it didn't 
| show up in my own testing because I already _had_ the file in 
| question.

6 out of 7 test-distros i have didnt have the tg3.bin file.

	Ingo

---------- Forwarded message ----------
From: David Woodhouse <dwmw2@infradead.org>
Date: Mon, Jul 14, 2008 at 6:13 PM
Subject: Re: [GIT *] Allow request_firmware() to be satisfied from
in-kernel, use it in more drivers.
To: Yinghai Lu <yhlu.kernel@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>,
torvalds@linux-foundation.org, alan@lxorguk.ukuu.org.uk,
linux-kernel@vger.kernel.org


On Mon, 2008-07-14 at 17:50 -0700, Yinghai Lu wrote:
>   MK_FW   firmware/tigon/tg3.bin.gen.S
> make[1]: *** No rule to make target `firmware/tigon/tg3.bin', needed
> by `firmware/tigon/tg3.bin.gen.o'.  Stop.
> make: *** [firmware] Error 2

I apologise; a line intended for the tg3 conversion accidentally slipped
into firmware/Makefile along with the dabusb patch, when I cherry-picked
from the old tree to omit the drivers/net patches. Again, it didn't show
up in my own testing because I already _had_ the file in question.

(Sorry, Linus. This and the previous fix are now in the for-2.6.27
branch of the same repo as before:
       git://git.infradead.org/users/dwmw2/firmware-2.6.git for-2.6.27
)

----
Subject: Fix accidental reference to tg3 firmware

We're not updating the tg3 driver to use request_firmware() yet, but a
reference to its firmware accidentally slipped in as part of commit
c4667746 ("dabusb: use request_firmware()"). Remove it again.

Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>

diff --git a/firmware/Makefile b/firmware/Makefile
index 10028ac..8dbdd3f 100644
--- a/firmware/Makefile
+++ b/firmware/Makefile
@@ -34,8 +34,6 @@ fw-shipped-$(CONFIG_SND_SB16_CSP) +=
sb16/mulaw_main.csp sb16/alaw_main.csp \
                                    sb16/ima_adpcm_capture.csp
 fw-shipped-$(CONFIG_SND_YMFPCI) += yamaha/ds1_ctrl.fw yamaha/ds1_dsp.fw \
                                  yamaha/ds1e_ctrl.fw
-fw-shipped-$(CONFIG_TIGON3) += tigon/tg3.bin tigon/tg3_tso.bin \
-                              tigon/tg3_tso5.bin
 fw-shipped-$(CONFIG_USB_DABUSB) += dabusb/firmware.fw dabusb/bitstream.bin
 fw-shipped-$(CONFIG_USB_EMI26) += emi26/loader.fw emi26/firmware.fw \
                                 emi26/bitstream.fw


--
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Marcel Holtmann]

Hi David,

> I'm not happy for network drivers, although I'm happy to see that
> David respected out NACKs on tg3/bnx2/etc. and didn't include those
> bits.
> 
> I still haven't seen an answer to the chip reset issues.  The argument
> has been that this stuff is going to save kernel memory when the
> driver isn't loading firmware, but that's not really possible.
> 
> When a lot of these network drivers reset, due to a transmit timeout
> or other exception, we need to load the firmware again.
> 
> So this means, that if request_firmware() dies or fails for some
> reason in these scenerios, we can't reset the network card.  Something
> as simple as a memory or other allocation failure deep inside of
> request_firmware() means we lose the networking interface.
> 
> To me this seems like a very non-resilient setup.  It makes sense
> to just keep the firmware in-core so we can load it without having
> to even think about possible failures.

actually when we wrote request_firmware() a long time ago, the idea was
to support various kind of setups (although until now only the on-demand
got used).

So one idea was to allow minimal inclusion of firmware as binary blobs
in the driver and allow overwriting them by loading newer version from
the filesystem. This was intended to ensure minimal working drivers. The
main background was that some devices work with a small firmware and
provide basic features and only need extra firmware for advanced
features. Everything suppose to be hidden behind request_firmware() and
a method to register binary blobs as firmware from within a driver. So
the driver developer really doesn't have to invent anything by himself.

We are not there yet and at the moment, I am not sure if this is still a
valid requirement, but David's work goes into the right direction and is
a good start.

Regards

Marcel

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Alan Cox]

> * the legal subtext of these changes was not mentioned at all

The only legal subtext has been some mad fantasy proposed by Dave Miller
that David Woodhouse current employer and previous one were both in some
conspiracy to push firmware elsewhere. It would be very helpful if you
stuck to the real world.

The only legal agenda people I know actually don't like request_firmware
because it makes it easier to have non-free firmware files. In their
strange world they'd rather delete the driver

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Tue, 2008-07-15 at 08:23 +0200, Oliver Neukum wrote:
> Am Dienstag 15 Juli 2008 08:03:15 schrieb David Woodhouse:
> > Mostly it follows the documented recommendation, since most of the
> > touched drivers are USB drivers and you end up re-enumerating and
> > starting from scratch on resume anyway. And the remainder are so old
> > that they don't have suspend/resume support anyway. Remember, we're only
> > really updating the older drivers; newer drivers tend to use
> > request_firmware() already, and have done for years.
> 
> As for USB that was before persistance. It now allows you to even
> support a loss of power without reprobing, provided
> 
> a) the devices does not change IDs
> b) you have firmware around when reset_resume() is called

c) the driver in question actually _has_ a reset_resume() method

The old, unloved drivers which I've touched as part of this patch set
did not. In general the newer, loved drivers use request_firmware()
already.

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Willy Tarreau wrote:
> > 
> > But once you can load a module, you can load the firmware. You just have 
> > to _remember_ to move it along with the module.
> 
> In order to transfer it, right now you feed it through a working device.
> When that device itself requires firmware to work, you will suddenly
> discover that it becomes harder and harder to get any communication device
> to work on your target system.

Willy, stop this blathering.

I suspect I will have to just delete this thread unread because it's so 
full of total crap.

This whole "working device" argument is total bullshit.

If the driver was a module before, it needed a module load to become 
working. If you could load the module, you could load the firmware. Thus 
the transfer is non-issue.

And if the driver was built-in, you can still build in the firmware.

And by proof of induction - your "now you feed it through a working 
device" - this also holds for that "a working device" driver. The device 
you load the module off has exactly the same rules: you can build in the 
firmware.

So it didn't get any harder at all. Except for the fact that you need to 
remember to install the firmware. 

Which is just about the same thing as asking people to remember to do 
"make modules_install" to get a working system. Yes, if you have a driver 
as a module, you need to install that module for the device to work. Yes, 
it's definitely "harder", but seriously..

			Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Jeff Garzik wrote:
> 
> A more complex, multi-file inter-dependent system is more reliable than a
> single-file driver with built-in firmware, doing the same thing?

I'm not interested in your made-up arguments.

Please just build everything statically. Don't use modules. They are 
clearly too complex for you, adn everybody is happy.

		Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[david]

On Tue, 15 Jul 2008, Linus Torvalds wrote:

> On Tue, 15 Jul 2008, Willy Tarreau wrote:
>>>
>>> But once you can load a module, you can load the firmware. You just have
>>> to _remember_ to move it along with the module.
>>
>> In order to transfer it, right now you feed it through a working device.
>> When that device itself requires firmware to work, you will suddenly
>> discover that it becomes harder and harder to get any communication device
>> to work on your target system.
>
> Willy, stop this blathering.
>
> I suspect I will have to just delete this thread unread because it's so
> full of total crap.
>
> This whole "working device" argument is total bullshit.
>
> If the driver was a module before, it needed a module load to become
> working. If you could load the module, you could load the firmware. Thus
> the transfer is non-issue.
>
> And if the driver was built-in, you can still build in the firmware.
>
> And by proof of induction - your "now you feed it through a working
> device" - this also holds for that "a working device" driver. The device
> you load the module off has exactly the same rules: you can build in the
> firmware.
>
> So it didn't get any harder at all. Except for the fact that you need to
> remember to install the firmware.
>
> Which is just about the same thing as asking people to remember to do
> "make modules_install" to get a working system. Yes, if you have a driver
> as a module, you need to install that module for the device to work. Yes,
> it's definitely "harder", but seriously..

has a standard been defined for how to maintain different versions of 
firmware for different versions of drivers yet? driver maintainers do not 
test drivers with all different firmware versions, just the one(s) that 
ship with that kernel version. there are also reports from maintainers 
that some driver/firmware combos do not have full backwards compatability 
so you can't just use the new firmware with the old drivers (causing 
unexpected glitches when switching between kernel versions)

that was one of the things that was identified in the last flamewar that 
was handwaved away but nothing was defined.

yes it's something that other drivers that use request_firmware() have not 
run into problems with, but the maintainers of drivers that have embedded 
the firmware into the driver up until now have stated that they see this 
as a problem.

it's also not that hard to do (it's just someone making a statement of 
what the standard will be that others can agree with), but it needs to be 
done.

the trivial way out is to define a firmware tree per kernel version 
(similar to how modules are handled), but that would offend some people (I 
think it was Alan Cox who was arguing in the last flamewar about how much 
bandwidth it would save to not distribute firmware with each kernel 
version), and it would definantly be a waste of disk space since the 
firmware changes far less frequently then the modules do.

David Lang

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Jeff Garzik wrote:
> 
> With this change, there are several exceptions where this is not the case,
> across all driver types -- media, wan, char, net, scsi, ...
> 
> Package manifests, build scripts and other details MUST be updated, or these
> drivers will not work as they did in kernels <= 2.6.26.

Umm. I complained six months ago when the e1000e driver took over, and the 
e1000 driver stopped working for me, and you suddenly had to know which 
version of the e1000 you had - even though there was not any sane way to 
tell them apart.

Ingo _still_ complains about that occasionally.

What did you do? You didn't care.

Now you suddenly do care that some driver needs additional love and 
learning.

I call bullshit. You don't like the change - ok. But stop making bogus 
arguments about it.

			Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Frans Pop]

Linus Torvalds wrote:
> On Mon, 14 Jul 2008, Jeff Garzik wrote:
>> IMO the newly added /inability/ to build firmware into kernel modules
>> is a clear regression.
> 
> IMO you're being stupid.
> 
> How about explainign why it makes any difference what-so-ever?
> 
> If you can load the module, you can load the firmware. Claiming anything
> else is just _stupid_.

Sure, in theory it's that simple. Here's a concrete example that shows how 
things are harder in practice.

I use the 'make deb-pkg' target (from scripts/package) to build my Debian 
kernel packages from git. So that needs to be adapted to include
/lib/firmware. No real problem so far.

So then I build 2.6.27-rc1 and install it. Great.

You release 2.6.27-rc2 and I build it. Ouch! It fails to install, at least 
if I want to install it _alongside_ 2.6.27-rc1 or other kernels (which I 
do!). Why does it fail? Because dpkg's package management does not allow 
one package to overwrite files already "owned" by another package.

So, how is this solved by Debian for already existing firmware packages? 
Basically by making a separate package for each firmware file (or 
driver). This works because there are not too many of them, but having a 
huge number of tiny packages is a nightmare by itself.

But anyway, the dep-pkg target will have to be made smarter than it is now 
if it's to deal with this [1]. And at least currently it is broken.

If I were able to compile firmware into the modules, the problem would be 
solved in one go.

I don't know how the Debian kernel team plans to deal with this for distro 
kernel packages. They probably _do_ want to keep them separate [2]. Maybe 
by grouping firmware for really common drivers into 
firmware-basic-drivers or something along those lines.

Cheers,
FJP

[1] Only quick solution I see is to have it install the firmware in a 
versioned directory and have the postinst copy things from there to
/lib/firmware.
[2] As one of the developers for Debian Installer I'm not looking forward 
to the complications that is going to cause for us and users.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, david@lang.hm wrote:
> 
> has a standard been defined for how to maintain different versions of firmware
> for different versions of drivers yet?

No. We haven't had that possibility, probably largely because we haven't 
even really had the ability (Davem and Jeff talk about "choice" and how 
request_firmware() supposedly removes it, but they ignore the fact that it 
very much cuts the other way too - request_firmware() _adds_ choice and 
makes it easier for people to try to upgrade firmware without upgrading 
the kernel).

> that was one of the things that was identified in the last flamewar that was
> handwaved away but nothing was defined.

I don't think we can - or want to - define things until we've seen the 
problems and advantages in practice over a longer time.

Do we want a versioning scheme? Probably not - exactly because people may 
well want to update (or downgrade) firmware from vendors independently of 
the kernel.

But maybe people will eventually want some kind of blacklisting scheme 
where the driver can say things like "I know I won't work with versions 
that match this checksum" and cause a nice big warning at load-time.

And I think we can all _imagine_ scenarios in all kinds of possible ways 
for things to work better or fail spectaclarly, and even come up with 
anecdotal evidence where scheme X (out of any number of random variations) 
is the "only possible one to work". I'd argue that we simply don't have 
any realistic way to decide which ones make sense.

For example, we can be in the situation where we have some case where
"only this particular firmware is known to work, and others are known to 
break", and people can still take two totally different mental approaches 
to the problem.

One approach is the "strict control one", where people argue that the 
above problem means that we absolutely *must* have very strict versioning, 
and we just always pair things up exactly.

But the other argument is that firmware is part of hardware, and the 
driver should strive to work with different versions of vendor firmware, 
the same way it should strive to work with different versions of chip 
hardware.

Which one is right?  Probably neither. Sometimes one, sometimes the other, 
and in both cases it should probably not be black-and-white.

IOW, I suspect that in the long run we want to say

  "Ok, there's no way we can support _all_ firmware versions, but on the 
   other hand, we also don't want to force particular versions and not let 
   people experiment.

   So drivers should strive to be flexible, and driver writers should not 
   always just say 'there is only one "blessed" firmware', but on the 
   other hand we cannot require drivers to work around horrible known bugs 
   in firmware when a fixed firmware is available"

but the exact details will have to depend on particular drivers.

And we should obviously have _a_ version of the firmware available with 
the kernel when that is possible. But I'd hate for it to be 1:1 with a 
particular driver version - because at that point it smells of being a 
single work, and if it is more than mere aggregation it's no longer viable 
with most of our firmware (I don't think we have source for more than one 
or two cases).

			Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Arjan van de Ven]

On Tue, 15 Jul 2008 17:57:09 +0200
Frans Pop <elendil@planet.nl> wrote:

> Linus Torvalds wrote:
> > On Mon, 14 Jul 2008, Jeff Garzik wrote:
> >> IMO the newly added /inability/ to build firmware into kernel
> >> modules is a clear regression.
> > 
> > IMO you're being stupid.
> > 
> > How about explainign why it makes any difference what-so-ever?
> > 
> > If you can load the module, you can load the firmware. Claiming
> > anything else is just _stupid_.
> 
> Sure, in theory it's that simple. Here's a concrete example that
> shows how things are harder in practice.
> 
> I use the 'make deb-pkg' target (from scripts/package) to build my
> Debian kernel packages from git. So that needs to be adapted to
> include /lib/firmware. No real problem so far.
> 
> So then I build 2.6.27-rc1 and install it. Great.
> 
> You release 2.6.27-rc2 and I build it. Ouch! It fails to install, at
> least if I want to install it _alongside_ 2.6.27-rc1 or other kernels
> (which I do!). Why does it fail? Because dpkg's package management
> does not allow one package to overwrite files already "owned" by
> another package.
> 

question: how do you deal with this with the dozens of drivers that use
request_firmware() even before yesterday ?
the answer is critical in how to deal with this "new" situation as
well ;-)

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Frans Pop wrote:
> 
> So, how is this solved by Debian for already existing firmware packages? 
> Basically by making a separate package for each firmware file (or 
> driver). This works because there are not too many of them, but having a 
> huge number of tiny packages is a nightmare by itself.

Why don't you just take the kernel-supplied firmware and make it part of 
the kernel package? The same way the kernel-supplied modules are part of 
it?

You're sounding like

  "Mommy, mommy, it hurts when I repeatedly hit my head against the wall 
   and stick forks in my eyes!"

and quite frankly, it doesn't make me go "Oh, poor baby" at all.

			Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Tue, 2008-07-15 at 09:01 -0700, Linus Torvalds wrote:
> (I don't think we have source for more than one or two cases).

We don't. As part of the cleanup I've been trying to document what we
know about the origins of the firmware, in the firmware/WHENCE file. 
Unfortunately, mostly that's just "unknown".

We have managed to dig out the original source for one or two hex arrays
though.

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Frans Pop]

On Tuesday 15 July 2008, Arjan van de Ven wrote:
> Frans Pop <elendil@planet.nl> wrote:
> > You release 2.6.27-rc2 and I build it. Ouch! It fails to install, at
> > least if I want to install it _alongside_ 2.6.27-rc1 or other kernels
> > (which I do!). Why does it fail? Because dpkg's package management
> > does not allow one package to overwrite files already "owned" by
> > another package.
>
> question: how do you deal with this with the dozens of drivers that use
> request_firmware() even before yesterday ?
> the answer is critical in how to deal with this "new" situation as
> well ;-)

For Debian packages that was in the part of the mail you snipped.

The simple answer for my current use of dep-pkg for custom kernels is:
I don't _have_ any hardware that requires firmware that's split out with 
current kernels.
Maybe I'll be lucky and will find that I don't have any with 2.6.27, but 
I'm not betting on that.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[david]

On Tue, 15 Jul 2008, Linus Torvalds wrote:

> On Tue, 15 Jul 2008, Frans Pop wrote:
>>
>> So, how is this solved by Debian for already existing firmware packages?
>> Basically by making a separate package for each firmware file (or
>> driver). This works because there are not too many of them, but having a
>> huge number of tiny packages is a nightmare by itself.
>
> Why don't you just take the kernel-supplied firmware and make it part of
> the kernel package? The same way the kernel-supplied modules are part of
> it?

David W has made it very clear that the eventual intent of this patch 
series is to remove all firmware from the kernel tree and have it shipped 
as a seperate package.

David Lang

> You're sounding like
>
>  "Mommy, mommy, it hurts when I repeatedly hit my head against the wall
>   and stick forks in my eyes!"
>
> and quite frankly, it doesn't make me go "Oh, poor baby" at all.
>
> 			Linus
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Linus Torvalds wrote:
> 
> Why don't you just take the kernel-supplied firmware and make it part of 
> the kernel package? The same way the kernel-supplied modules are part of 
> it?

And, btw, if there is something we can do to help, we obviously should.

So ask yourself (and DavidW) things like

 - Is "make firmware_install" not enough for the package build system?

 - Does it need some other way to figure out what firmware is needed for 
   just the configured set of drivers, eg "make firmware_list"?

 - Do we need to make the firmware loader have multiple directories it 
   looks at? Do we want first a kernel-version-specific one, then a 
   generic fallback one, so that we can easily have per-kernel firmware, 
   the same way we have per-kernel modules?

 - Add your interesting question here.

There are obviously all these kinds of _productive_ questions you can ask. 
But people seem to be mostly in an unproductive "Mommy, it hurts when I do 
something stupid" mode.

			Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Frans Pop]

On Tuesday 15 July 2008, you wrote:
> > You release 2.6.27-rc2 and I build it. Ouch! It fails to install, at
> > least if I want to install it _alongside_ 2.6.27-rc1 or other kernels
> > (which I do!). Why does it fail? Because dpkg's package management
> > does not allow one package to overwrite files already "owned" by
> > another package.
>
> question: how do you deal with this with the dozens of drivers that use
> request_firmware() even before yesterday ?
> the answer is critical in how to deal with this "new" situation as
> well ;-)

For Debian packages that's in the part of the mail you snipped.

The simple answer for my current use of dep-pkg is: I don't _have_ any 
hardware that requires firmware that's split out with current kernels.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Frans Pop]

On Tuesday 15 July 2008, you wrote:
> On Tue, 15 Jul 2008, Frans Pop wrote:
> > So, how is this solved by Debian for already existing firmware
> > packages? Basically by making a separate package for each firmware
> > file (or driver). This works because there are not too many of them,
> > but having a huge number of tiny packages is a nightmare by itself.
>
> Why don't you just take the kernel-supplied firmware and make it part
> of the kernel package? The same way the kernel-supplied modules are
> part of it?

Important note: this is not me, this is the Debian kernel team based on 
Debian Free Software Guidelines (DFSG). If it were purely up to me I'd be 
a lot more pragmatic.

1) Because Debian _wants_ them split out for DFSG compliance reasons.
Most of the current firmware packages are kept in the "non-free" section 
of the Debian archive while the kernel itself lives in "main".
As long as firmware could not be split out, the compliance problem was the 
source of many discussions, but for the most part ignored in practice, 
with the exception of a few drivers with really problematic firmware 
licences.

2) Because of the overwrite and version management problems.
/lib/firmware as a single dumping ground for firmware for all kernel 
versions really sucks from that PoV. One of the huge strengths of Debian 
is its ability to clean up after itself when packages are removed.
This is partially solved by giving each firmware file it's own package 
because then you can use the versioning of the firmware itself in the 
package versions, which allows proper file management by the packaging 
system. But as I said, I'm not sure that still works if their number 
suddenly explodes.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, david@lang.hm wrote:
>
> David W has made it very clear that the eventual intent of this patch series
> is to remove all firmware from the kernel tree and have it shipped as a
> seperate package.

I don't think that kind of black-and-white model is necessarily 
attainable. Nor do I think it's even _good_.

It's probably the right thign for _some_ firmware, assuming we have models 
that just make it easy enough - for drivers where it's generally better 
for distros to get the firmware from vendors rather than the kernel.

But there is certainly nothing inherently good in forcing a split.

I think both sides here need to calm down and stop being so extreme.

I do want request_firmware() to be the way to load firmware (and possibly 
other configuration data too, for that matter - I don't think it's 
necessarily wrong for people to "misuse" it for other long-term data 
despite the name), and I do want people to be able to update firmware 
easily without having to force a kernel bump, but I also don't think it 
means that firmware has to go away from the kernel sources.

So I think the _infrastructure_ is important. But the extreme "you have to 
split it off entirely" mindset is equally as stupid as the "you must 
compile it into the module" mindset.

			Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Frans Pop]

On Tuesday 15 July 2008, Linus Torvalds wrote:
> There are obviously all these kinds of _productive_ questions you can
> ask. But people seem to be mostly in an unproductive "Mommy, it hurts
> when I do something stupid" mode.

Of course, but the main reason I replied in the first place was because 
your "there are no real problems" attitude in this discussion was 
annoying me.

And even in your first reply to me you basically ignored that fact that at 
least one kernel packaging tool and one that is *part of the kernel tree 
itself* is now broken.

There *are* real problems and some of them *could* be solved by allowing 
firmware to be compiled into modules. Sure, other solutions are possible 
too and if I actually run into the problems I foresee with dpk-deb you 
can be certain I will try to help solve them.

But if you want to invite people to hold a productive discussion, it helps 
if you don't completely ignore their concerns out of hand.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Oliver Neukum]

Am Dienstag 15 Juli 2008 16:40:54 schrieb David Woodhouse:
> On Tue, 2008-07-15 at 08:23 +0200, Oliver Neukum wrote:
> > Am Dienstag 15 Juli 2008 08:03:15 schrieb David Woodhouse:
> > > Mostly it follows the documented recommendation, since most of the
> > > touched drivers are USB drivers and you end up re-enumerating and
> > > starting from scratch on resume anyway. And the remainder are so old
> > > that they don't have suspend/resume support anyway. Remember, we're only
> > > really updating the older drivers; newer drivers tend to use
> > > request_firmware() already, and have done for years.
> > 
> > As for USB that was before persistance. It now allows you to even
> > support a loss of power without reprobing, provided
> > 
> > a) the devices does not change IDs
> > b) you have firmware around when reset_resume() is called
> 
> c) the driver in question actually _has_ a reset_resume() method
> 
> The old, unloved drivers which I've touched as part of this patch set
> did not. In general the newer, loved drivers use request_firmware()
> already.
> 

Generally we love all drivers and welcome new additions.
Which did you touch? I lost track.

	Regards
		Oliver

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Linus Torvalds wrote:
> 
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>> With this change, there are several exceptions where this is not the case,
>> across all driver types -- media, wan, char, net, scsi, ...
>>
>> Package manifests, build scripts and other details MUST be updated, or these
>> drivers will not work as they did in kernels <= 2.6.26.
> 
> Umm. I complained six months ago when the e1000e driver took over, and the 
> e1000 driver stopped working for me, and you suddenly had to know which 
> version of the e1000 you had - even though there was not any sane way to 
> tell them apart.
> 
> Ingo _still_ complains about that occasionally.
> 
> What did you do? You didn't care.

Completely false -- what was the outcome?  The breakage complained about 
was fixed, we made sure Ingo had proper defconfig entries, the changes 
to e1000e were pushed back.

That's a lot of taking-feedback-from-others that you define as "didn't 
care."

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Linus Torvalds wrote:
> 
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>> A more complex, multi-file inter-dependent system is more reliable than a
>> single-file driver with built-in firmware, doing the same thing?
> 
> I'm not interested in your made-up arguments.

It's a simple question...  you claimed A was more reliable than B, even 
though A is a more complex system than B.

I'm curious how that works, especially given that you have claimed the 
_exact opposite_ in years past, by pointing out how firmware separation 
could mean no-boot.


> Please just build everything statically. Don't use modules. They are 
> clearly too complex for you, adn everybody is happy.

Can't you make a simple, objective, technical comparison between two 
systems?

firmware-outside-driver has more utility and flexibility, but more 
things can go wrong, like driver present/firmware absent case that 
happens every day with today's wireless drivers.

firmware-inside-driver guarantees without _any_ doubt that the firmware 
is present, if the driver is guarantee.  There is no guarantee more 
solid than that.

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Frans Pop wrote:
> 
> Of course, but the main reason I replied in the first place was because 
> your "there are no real problems" attitude in this discussion was 
> annoying me.

I really don't think there are any real problems.

The problems are all due to inertia - people (and scripts) are used to one 
way of doing things. Those aren't really technical issues, and more 
importantly, they are not issues that _can_ be solved any other way than 
just doing it, and fixing the fallout.

Trying to be "proactive" doesn't work. Waiting for people (and scripts) to 
get used to a new model before merging it also doesn't work. And 
complaining about a change that pretty obviously is needed also doesn't 
work.

So let's just do it. And then, when specific scripts are encountered, fix 
them. And yes, fixing the scripts may _well_ imply adding more support for 
it in the kernel. It could be either in build and installation scripts, or 
in udev infrastructure, or in request_firmware() capability itself.

For example, some people hate udev, and they may have some really good 
reason why they really don't want to rely on that in their setup. So maybe 
we should teach request_firmware() to try to look up (binary) firmware 
files directly from the filesystem before it does a udev callback (in 
fact, maybe it even already does - I really didn't check very closely).

None of these are "problems". 

But people who don't even want to look for solutions - _that_ is a 
problem.

			Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Jeff Garzik wrote:
> 
> I'm curious how that works, especially given that you have claimed the _exact
> opposite_ in years past, by pointing out how firmware separation could mean
> no-boot.

Umm. Firmware separation _could_ mean non-boot.

DavidW &co fixed it. You can now build it in.

You seem to ignore that things change, and _have_ changed.

		Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Linus Torvalds wrote:
> I call bullshit. You don't like the change - ok.

SIGH.  Wrong, wrong, wrong.  I have always thought request_firmware() 
was a good idea, even if it decreased reliability.  Because we need that 
ability and that API.

The changes are good,

They just don't need to break stuff.

Which they do.


> But stop making bogus 
> arguments about it.

How many more real world examples of breakage do you need?

* This breaks newer kernel / older userland
* This breaks anywhere with a file list for modules' contents (ipkg 
kernel module packages)
* This breaks driver disk build packages
* This breaks for the oddball, not just me, who
* This breaks for an unknown amount of people who have been assuming 
that their driver will always work when modprobe'd, even though they are 
completely unaware of its new firmware needs

All these are cases in use today.  Real users, Linus.  Real distros.

	Jeff


P.S.  And as I stated in another email, I'm happy to help fix this 
regression.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Frans Pop]

On Tuesday 15 July 2008, you wrote:
> So I think the _infrastructure_ is important. But the extreme "you have
> to split it off entirely" mindset is equally as stupid as the "you must
> compile it into the module" mindset.

Agreed. All I've wanted to say is that having the _option_ to continue to 
compile it into the module can simplify certain use cases, including my 
current use of the kernel's own deb-pkg target for private builds of the 
kernel for testing purposes.

One thing I would dearly like to see is the ability to choose a different 
location for firmware for my own custom built kernels than the location 
used for firmware from official Debian kernel packages so that at least 
those two don't clash.
Doesn't look like that is possible with current git head.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Jeff Garzik wrote:
> 
> The changes are good,
> 
> They just don't need to break stuff.
> 
> Which they do.

If you were serious about that, then you'd be suggesting _fixes_.

I'm not seeing you do that. I'm just seeing non-productive "we should 
never have changed" emails.

See my emails about what can be _productive_ avenues to explore. As it is, 
I have yet to see a _single_ productive email from you. 

Really, Jeff.

		Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Linus Torvalds wrote:
> 
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>> I'm curious how that works, especially given that you have claimed the _exact
>> opposite_ in years past, by pointing out how firmware separation could mean
>> no-boot.
> 
> Umm. Firmware separation _could_ mean non-boot.
> 
> DavidW &co fixed it. You can now build it in.
> 
> You seem to ignore that things change, and _have_ changed.

Already addressed -- you and David seem to be ignoring the common case 
where you may update the driver but not the kernel, making that feature 
far less useful than it appears.

Inability to update the kernel image (vmlinuz) is the case with just 
about every enterprise distro out there, including ones to be based on 
 >= 2.6.27.

Driver disks and our own in-tree build-module-out-of-tree support are 
built precisely for this purpose:  to build a new driver against an 
existing kernel, because so often you cannot update the core vmlinuz image.

(rather, you _can_ update vmlinuz, but that would fail on the next 
automated update, so its not a realistic option)

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Arjan van de Ven]

On Tue, 15 Jul 2008 18:36:56 +0200
Frans Pop <elendil@planet.nl> wrote:

> On Tuesday 15 July 2008, you wrote:
> > > You release 2.6.27-rc2 and I build it. Ouch! It fails to install,
> > > at least if I want to install it _alongside_ 2.6.27-rc1 or other
> > > kernels (which I do!). Why does it fail? Because dpkg's package
> > > management does not allow one package to overwrite files already
> > > "owned" by another package.
> >
> > question: how do you deal with this with the dozens of drivers that
> > use request_firmware() even before yesterday ?
> > the answer is critical in how to deal with this "new" situation as
> > well ;-)
> 
> For Debian packages that's in the part of the mail you snipped.
> 
> The simple answer for my current use of dep-pkg is: I don't _have_
> any hardware that requires firmware that's split out with current
> kernels.

... and do you have any with current -git ?


-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Willy Tarreau]

On Tue, Jul 15, 2008 at 07:55:37AM -0700, Linus Torvalds wrote:
> Willy, stop this blathering.
> 
> I suspect I will have to just delete this thread unread because it's so 
> full of total crap.
> 
> This whole "working device" argument is total bullshit.
> 
> If the driver was a module before, it needed a module load to become 
> working. If you could load the module, you could load the firmware. Thus 
> the transfer is non-issue.

Sometimes your responses leave me speechless! It's not only a problem of
transfer, but also a problem of selection of what to copy overthere. It
has happened to me more than once when installing a machine to proceed this
way :
  - leave the data center
  - go back to the office to find the driver somewhere
  - burn a CD with it (I once even had to feed it through a 9600 bauds Annex
    port concentrator).
  - go back to the DC and copy it from the CD. I must say that at this stage
    when you see warnings because it's not the exact same kernel version, you
    don't mind anymore and force the load in order to be able to stick an IP
    address on your machine.

Now when you load it and you see it requires a firmware, you're back to #1,
without even the certainty that the firmware you get will :
  a) be the right one
  b) work
  c) be enough
  
So please don't say that the transfer is a non-issue. It is what magnifies
the added complexity.

Instead, I would have loved to see multiboot-compatible module loading.
When your kernel does not support your fresh new RAID card and panics at
boot, you simply put the driver on a floppy (or USB stick), pre-load the
driver from GRUB and boot the kernel. I don't see how this would be possible
anymore with separate firmwares.

Don't get me wrong: I'm not against the ability to separate firmware from
modules, I'm against making that mandatory when it obviously complexifies
a lot of things without any obvious advantage in return.

> Which is just about the same thing as asking people to remember to do 
> "make modules_install" to get a working system. Yes, if you have a driver 
> as a module, you need to install that module for the device to work. Yes, 
> it's definitely "harder", but seriously..

Please don't compare an upgrade of a working desktop or developer machine with
an installation from scratch of a server in an uncomfortable location. Those
are two completely different things.

Willy

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Tue, 2008-07-15 at 09:41 -0700, Linus Torvalds wrote:
> So I think the _infrastructure_ is important. But the extreme "you
> have to split it off entirely" mindset is equally as stupid as the
> "you must compile it into the module" mindset.

I wouldn't necessarily argue that we _have_ to split it off, but I'd
certainly like our infrastructure to be mature enough that it could cope
with doing so.

I plan to make a 'linux-firmware-from-linus' git tree which is generated
automatically from yours by a script, much like the kernel-headers git
tree is, commit by commit with an automatic transform on the tree
objects.

Once we have that tree being created, we could pull from it into a
canonical 'linux-firmware' tree, and _that_ tree could also start to
include firmware belonging to companies who have said that we may
distribute their firmware, but _not_ as part of the Linux kernel.

We can see how that tree works out in practice, and how/whether people
start using it. And then maybe we'll be in a position to have a
_discussion_ about how best to ship the firmware. But for now it's
premature.
 
-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Jeff Garzik wrote:
> 
> Already addressed -- you and David seem to be ignoring the common case where
> you may update the driver but not the kernel, making that feature far less
> useful than it appears.

This is the "I'm too stupid to live" argument. I've already answered that. 
What's so hard with "make firmware_install"?

		Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Tue, 2008-07-15 at 19:35 +0200, Willy Tarreau wrote:
> It's not only a problem of transfer, but also a problem of selection
> of what to copy overthere.

This is why the MODULE_FIRMWARE() tag exists.

Most scripts tend to get this right already -- they _have_ to, since
modern drivers have been using request_firmware() for some time already.

> Please don't compare an upgrade of a working desktop or developer
> machine with an installation from scratch of a server in an
> uncomfortable location. Those are two completely different things.

If your setup doesn't cope with MODULE_FIRMWARE(), it's likely to break
when you next deploy a new server anyway -- because your new server will
have new hardware, and your new hardware will have a modern driver. And
modern drivers use request_firmware() _anyway_.

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Linus Torvalds wrote:
> 
> This is the "I'm too stupid to live" argument. I've already answered that. 
> What's so hard with "make firmware_install"?

Btw, that is by no means the only way to handle it either. As I said in 
other emails - about trying to find _productive_ questions to answer - 
maybe we could also just install it in the modules directory or some 
similar per-kernel location, and just teach request_firmware() to load 
binary blobs directly.

Then - horror of horrors - we could even make "make modules_install" do 
things like that automatically for you with some Makefile rule trickery.

See? There are _productive_ things you can do or ask others to do.

But singing the same "I'm so stupid that I cannot install the firmware" 
song over and over again doesn't make me think you are worth listening to.

			Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Linus Torvalds wrote:
> 
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>> The changes are good,
>>
>> They just don't need to break stuff.
>>
>> Which they do.
> 
> If you were serious about that, then you'd be suggesting _fixes_.

The fix was suggested from day one:  permit firmware-in-module, which 
I've already said I will work on...  provided its not dead on arrival.

Other fixes I've stated in these threads I want to tackle:

* newly separated firmware should live alongside the driver

* driver author should be able to deliver binary firmware direct from 
vendor, without any ihex or C-source-code mangling.


> I'm not seeing you do that. I'm just seeing non-productive "we should 
> never have changed" emails.

The emails are "this change should not contain obvious breakages"

To review the parts you've apparently missed, the problems and 
regressions I have found during review were

- Kconfig produced non-working drivers.  10+ skilled kernel hackers hit 
this.  I complained, was called silly, pointless, and having a tantrum. 
  Then, the issue was fixed, and kernel hackers could boot again.

- the normal build process produced non-working drivers.  A couple more 
skilled kernel hackers hit this one.  Build process regression was 
OBVIOUS.  Again, fixing this was called pointless and silly.  Again, it 
got fixed.  Again, kernel hackers started working again.

Now we have one last major regression hole to fix -- firmware-in-module.


> See my emails about what can be _productive_ avenues to explore. As it is, 
> I have yet to see a _single_ productive email from you. 

Strange, considering that it has been me who pointed out problems that 
would -- and did -- hit other kernel hackers, and those problems after 
much complaining and insulting got fixed.

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[david]

On Tue, 15 Jul 2008, Linus Torvalds wrote:

> On Tue, 15 Jul 2008, david@lang.hm wrote:
>>
>> has a standard been defined for how to maintain different versions of firmware
>> for different versions of drivers yet?
>
> No. We haven't had that possibility, probably largely because we haven't
> even really had the ability (Davem and Jeff talk about "choice" and how
> request_firmware() supposedly removes it, but they ignore the fact that it
> very much cuts the other way too - request_firmware() _adds_ choice and
> makes it easier for people to try to upgrade firmware without upgrading
> the kernel).

and they aren't objecting to this choice being added. they just want to 
keep the choice of working the other way (although they implicitly are 
saying that they won't be doing a lot of testing of different firmware 
versions becouse they do want to work the other way)

David Lang

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Frans Pop]

On Tuesday 15 July 2008, Arjan van de Ven wrote:
> > The simple answer for my current use of dep-pkg is: I don't _have_
> > any hardware that requires firmware that's split out with current
> > kernels.
>
> ... and do you have any with current -git ?

Haven't tried that yet. Only just finished building .26 for my boxes and 
have other priorities ATM anyway.
You'll hear about it on the list when I do :-)

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Henrique de Moraes Holschuh]

On Tue, 15 Jul 2008, Frans Pop wrote:
> So then I build 2.6.27-rc1 and install it. Great.
> 
> You release 2.6.27-rc2 and I build it. Ouch! It fails to install, at least 
> if I want to install it _alongside_ 2.6.27-rc1 or other kernels (which I 
> do!). Why does it fail? Because dpkg's package management does not allow 
> one package to overwrite files already "owned" by another package.

I believe I read in the previous thread that some distros are already using
/lib/firmware/<kernel version>/.

There was also the suggestion of moving the entire set of kernel-packaged
firmware to /lib/modules/<kernel version>/firmware, probably while keeping
/lib/firmware as a second place to look for firmware so that we don't hose
any system.

> If I were able to compile firmware into the modules, the problem would be 
> solved in one go.

And this thread would have been shorter, even. I hope someone decides to
write that support instead of complaining ;-)

But I do feel we still need a smarter userspace firmware loader to make
firmware packaging less insane.  The "current aproach" you described, with
one firmware per package, is not good.  It doesn't allow for multiple
versions of the firmware to be installed should you need it.

> I don't know how the Debian kernel team plans to deal with this for distro 
> kernel packages. They probably _do_ want to keep them separate [2]. Maybe 
> by grouping firmware for really common drivers into 
> firmware-basic-drivers or something along those lines.

I sure hope we go with the more proper fix (a version-enabled firmware
loader that can do the unversioned /lib/firmware as well).  It is far more
resilient in the long run.

> [1] Only quick solution I see is to have it install the firmware in a 
> versioned directory and have the postinst copy things from there to
> /lib/firmware.

Yuck.  That would work only if you never needed two active copies of the
kernel [with different firmware files] active at the same time.

> [2] As one of the developers for Debian Installer I'm not looking forward 
> to the complications that is going to cause for us and users.

That was my point.  These firmware loading changes are good, but there is a
lot of crap missing (most of it NOT in the kernel) before it can be exposed
to ordinary users.

And without firmware-in-the-module support (which is the ONLY scenario where
the entire userspace will not notice anything different), this WILL cause
problem to distros, we will need to scramble up and fix it all before we can
package 2.6.26.  I don't know if this is a big problem, though.  The work
will need to be done sooner or later anyway, and we should have enough time
to do so as long as we don't care for packaging the early -rc.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Tue, 2008-07-15 at 18:52 +0200, Oliver Neukum wrote:
> Am Dienstag 15 Juli 2008 16:40:54 schrieb David Woodhouse:
> > On Tue, 2008-07-15 at 08:23 +0200, Oliver Neukum wrote:
> > > Am Dienstag 15 Juli 2008 08:03:15 schrieb David Woodhouse:
> > > > Mostly it follows the documented recommendation, since most of the
> > > > touched drivers are USB drivers and you end up re-enumerating and
> > > > starting from scratch on resume anyway. And the remainder are so old
> > > > that they don't have suspend/resume support anyway. Remember, we're only
> > > > really updating the older drivers; newer drivers tend to use
> > > > request_firmware() already, and have done for years.
> > > 
> > > As for USB that was before persistance. It now allows you to even
> > > support a loss of power without reprobing, provided
> > > 
> > > a) the devices does not change IDs
> > > b) you have firmware around when reset_resume() is called
> > 
> > c) the driver in question actually _has_ a reset_resume() method
> > 
> > The old, unloved drivers which I've touched as part of this patch set
> > did not. In general the newer, loved drivers use request_firmware()
> > already.
> > 
> 
> Generally we love all drivers and welcome new additions.
> Which did you touch? I lost track.

Ignoring the simple bits to make fw->data const? See
http://git.infradead.org/users/dwmw2/firmware-2.6.git?a=shortlog&h=d71792ac
or    git-diff --stat 59890f74..d71792ac | grep -v firmware/

21 drivers, mostly older USB drivers and including four sound drivers
which were already using request_firmware() optionally. The HEAD of my
tree contains a few more updates in drivers/scsi and drivers/net, but
out of respect for Dave's and Jeff's objections I left those out of the
pull request for now. Hopefully the noise will die down when the sky
doesn't _actually_ fall on our heads, and we can merge the remaining
changes into 2.6.28.

 drivers/atm/Makefile                               |    6 +-
 drivers/atm/ambassador.c                           |  140 +-
 drivers/atm/ambassador.h                           |   11 -
 drivers/atm/atmsar11.data                          | 2063 -----
 drivers/atm/atmsar11.regions                       |    6 -
 drivers/atm/atmsar11.start                         |    4 -
 drivers/char/dsp56k.c                              |   84 +-
 drivers/char/ip2/fip_firm.h                        | 2149 -----
 drivers/char/ip2/ip2base.c                         |    5 +-
 drivers/char/ip2/ip2main.c                         |   47 +-
 drivers/media/dvb/ttusb-budget/dvb-ttusb-budget.c  |   16 +-
 .../media/dvb/ttusb-budget/dvb-ttusb-dspbootcode.h | 1644 ----
 drivers/media/video/cpia2/cpia2_core.c             |   46 +-
 drivers/media/video/cpia2/cpia2patch.h             |  233 -
 drivers/media/video/dabfirmware.h                  | 1415 ----
 drivers/media/video/dabusb.c                       |   44 +-
 drivers/media/video/usbvideo/vicam.c               |  317 +-
 drivers/net/tokenring/smctr.c                      |   56 +-
 drivers/net/tokenring/smctr.h                      |    2 -
 drivers/net/tokenring/smctr_firmware.h             |  978 ---
 drivers/net/usb/kaweth.c                           |   43 +-
 drivers/net/usb/kawethfw.h                         |  557 --
 drivers/usb/misc/emi26.c                           |   96 +-
 drivers/usb/misc/emi26_fw.h                        | 5779 -------------
 drivers/usb/misc/emi62.c                           |  130 +-
 drivers/usb/misc/emi62_fw_m.h                      | 8853 --------------------
 drivers/usb/misc/emi62_fw_s.h                      | 8837 -------------------
 drivers/usb/serial/Kconfig                         |   24 +-
 drivers/usb/serial/io_edgeport.c                   |  183 +-
 drivers/usb/serial/io_fw_boot.h                    |  556 --
 drivers/usb/serial/io_fw_boot2.h                   |  546 --
 drivers/usb/serial/io_fw_down.h                    | 1229 ---
 drivers/usb/serial/io_fw_down2.h                   | 1133 ---
 drivers/usb/serial/io_fw_down3.h                   |  847 --
 drivers/usb/serial/io_ti.c                         |   73 +-
 drivers/usb/serial/keyspan.c                       |   78 +-
 drivers/usb/serial/keyspan.h                       |   84 -
 drivers/usb/serial/keyspan_mpr_fw.h                |  286 -
 drivers/usb/serial/keyspan_pda.S                   | 1124 ---
 drivers/usb/serial/keyspan_pda.c                   |   51 +-
 drivers/usb/serial/keyspan_pda_fw.h                |   99 -
 drivers/usb/serial/keyspan_usa18x_fw.h             |  447 -
 drivers/usb/serial/keyspan_usa19_fw.h              |  285 -
 drivers/usb/serial/keyspan_usa19qi_fw.h            |  284 -
 drivers/usb/serial/keyspan_usa19qw_fw.h            |  448 -
 drivers/usb/serial/keyspan_usa19w_fw.h             |  446 -
 drivers/usb/serial/keyspan_usa28_fw.h              |  466 -
 drivers/usb/serial/keyspan_usa28x_fw.h             |  447 -
 drivers/usb/serial/keyspan_usa28xa_fw.h            |  449 -
 drivers/usb/serial/keyspan_usa28xb_fw.h            |  448 -
 drivers/usb/serial/keyspan_usa49w_fw.h             |  464 -
 drivers/usb/serial/keyspan_usa49wlc_fw.h           |  476 --
 drivers/usb/serial/ti_fw_3410.h                    |  885 --
 drivers/usb/serial/ti_fw_5052.h                    |  885 --
 drivers/usb/serial/ti_usb_3410_5052.c              |   40 +-
 drivers/usb/serial/whiteheat.c                     |   77 +-
 drivers/usb/serial/whiteheat_fw.h                  | 1669 ----
 drivers/usb/serial/xircom_pgs.S                    | 1192 ---
 drivers/usb/serial/xircom_pgs_fw.h                 |  103 -
 sound/isa/Kconfig                                  |    9 -
 sound/isa/sb/sb16_csp.c                            |   22 -
 sound/isa/sb/sb16_csp_codecs.h                     |  949 ---
 sound/pci/Kconfig                                  |   30 -
 sound/pci/korg1212/korg1212-firmware.h             |  987 ---
 sound/pci/korg1212/korg1212.c                      |   18 -
 sound/pci/maestro3.c                               |  171 -
 sound/pci/ymfpci/ymfpci_image.h                    | 1565 ----
 sound/pci/ymfpci/ymfpci_main.c                     |   63 -

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Linus Torvalds wrote:
> 
> On Tue, 15 Jul 2008, Linus Torvalds wrote:
>> This is the "I'm too stupid to live" argument. I've already answered that. 
>> What's so hard with "make firmware_install"?
> 
> Btw, that is by no means the only way to handle it either. As I said in 
> other emails - about trying to find _productive_ questions to answer - 
> maybe we could also just install it in the modules directory or some 
> similar per-kernel location, and just teach request_firmware() to load 
> binary blobs directly.
> 
> Then - horror of horrors - we could even make "make modules_install" do 
> things like that automatically for you with some Makefile rule trickery.
> 
> See? There are _productive_ things you can do or ask others to do.
> 
> But singing the same "I'm so stupid that I cannot install the firmware" 
> song over and over again doesn't make me think you are worth listening to.

Can't you see that changing build processes takes time?  Fixing driver 
disk creation tools take time?  Validating that each driver is packaged 
with its correct firmware takes time?

With each driver converted, it is entirely possible that all users are 
completely unaware that it even had a firmware, much less that it now 
needs 'make firmware_install' to continue working.

If one is completely unaware of a driver's firmware need, (a) you don't 
know you need to install it and (b) you don't find out you need it until 
you boot and the driver fails to work.

I am trying to work to minimize (a) and (b) here, because as deployed, 
there are large chances for failure here, across a random selection of 
drivers.

	Jeff


P.S.  AFAIK, 'make modules_install' installs firmware, unless that 
change was not pushed upstream.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Henrique de Moraes Holschuh wrote:
> On Tue, 15 Jul 2008, Frans Pop wrote:
>> If I were able to compile firmware into the modules, the problem would be 
>> solved in one go.
> 
> And this thread would have been shorter, even. I hope someone decides to
> write that support instead of complaining ;-)

I've volunteered -- but if Linus and David will reject my code out of 
hand, there is not much point in writing it.


>> [2] As one of the developers for Debian Installer I'm not looking forward 
>> to the complications that is going to cause for us and users.
> 
> That was my point.  These firmware loading changes are good, but there is a
> lot of crap missing (most of it NOT in the kernel) before it can be exposed
> to ordinary users.

100% agreed.

That's the point I've been trying to make all along.

The end goal is good, but it's not at all ready for normal users.


> And without firmware-in-the-module support (which is the ONLY scenario where
> the entire userspace will not notice anything different), this WILL cause
> problem to distros, we will need to scramble up and fix it all before we can
> package 2.6.26.  I don't know if this is a big problem, though.  The work
> will need to be done sooner or later anyway, and we should have enough time
> to do so as long as we don't care for packaging the early -rc.

Agreed.

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Linus Torvalds wrote:
> 
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>> Already addressed -- you and David seem to be ignoring the common case where
>> you may update the driver but not the kernel, making that feature far less
>> useful than it appears.
> 
> This is the "I'm too stupid to live" argument. I've already answered that. 
> What's so hard with "make firmware_install"?

So hard that several kernel hackers wound up with non-working drivers 
(read the pre-push bug reports).

Which is why 'make modules_install' installs the firmware, or at least 
it did before David W pushed upstream.

That was a key build process regression fix (thanks David!).

Sorry Linus, the end change is a good idea, but you cannot hand-wave 
away breakage just by calling people stupid.

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Tue, 2008-07-15 at 15:05 -0300, Henrique de Moraes Holschuh wrote:
> 
> I believe I read in the previous thread that some distros are already using
> /lib/firmware/<kernel version>/.
> 
> There was also the suggestion of moving the entire set of kernel-packaged
> firmware to /lib/modules/<kernel version>/firmware, probably while keeping
> /lib/firmware as a second place to look for firmware so that we don't hose
> any system.

You can do that right now; I expect Ubuntu will, since their userspace
already looks like that. You just need to set $(INSTALL_FW_PATH) when
you install the firmware (or modules).

The Ubuntu folks have chosen not to share their changes with upstream,
though -- and if we were to _impose_ that policy, then we would be
requiring changes to userspace which are not yet widespread. That's a
bad idea. What we're doing right now is _entirely_ compatible with what
userspace already has to deal with for modern drivers, and that's very
important.

There may be some merit in Ubuntu's policy, but we need userspace to
change first, long before we can change the default in the kernel.

But certainly the kernel _allows_ that, if you want it.

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Jeff Garzik wrote:
> 
> Can't you see that changing build processes takes time?  Fixing driver disk
> creation tools take time?  Validating that each driver is packaged with its
> correct firmware takes time?

Umm. Can't you see that if we don't start doing it, it will never get done 
AT ALL?

Yes, this will take time.

But that's not an argument for not merging it. Quite the reverse.

			Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Jeff Garzik wrote:

> Henrique de Moraes Holschuh wrote:
> > On Tue, 15 Jul 2008, Frans Pop wrote:
> > > If I were able to compile firmware into the modules, the problem would be
> > > solved in one go.
> > 
> > And this thread would have been shorter, even. I hope someone decides to
> > write that support instead of complaining ;-)
> 
> I've volunteered -- but if Linus and David will reject my code out of hand,
> there is not much point in writing it.

Quite frankly, I don't think it's something to reject out of hand, but it 
does sound pretty damn stupid.

Why compile them into the module, when that's just the _inflexible_ model?

Why not just install them with the module (perhaps in the module 
directory, perhaps not), and just load them? 

Really. You _do_ have do to "make modules_install" or similar. That has 
_already_ got inter-file dependencies (ie some modules simply depend on 
other modules). I'm not seeing why you cannot accept change.

			Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Jeff Garzik wrote:
> 
> Which is why 'make modules_install' installs the firmware, or at least it did
> before David W pushed upstream.

So you're literally just about making this be "make modules_install" 
rather than "make firmware_install"

Ok. Are you going to be happy if "make modules_install" just copies the 
firmware files of the affected modules too?

Or are you just whining because you want to whine, and will make up some 
new argument then?

			Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Tue, 2008-07-15 at 11:25 -0700, Linus Torvalds wrote:
> 
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
> > 
> > Which is why 'make modules_install' installs the firmware, or at least it did
> > before David W pushed upstream.
> 
> So you're literally just about making this be "make modules_install" 
> rather than "make firmware_install"
> 
> Ok. Are you going to be happy if "make modules_install" just copies the 
> firmware files of the affected modules too?

I did that already. I'm not _entirely_ sure it was the right thing to
do, but it does help to reduce surprises when people don't realise that
a driver they're using now has separate firmware.

I don't believe it was the result of a constructive request from Jeff; I
think someone else suggested it in response to his whining. It may even
have been my idea, but I don't recall.

This is the current situation:

'make modules_install' will install the firmware required by the modules
it installs. It'll install that to $(INSTALL_MOD_PATH)/lib/firmware/
unless you override that by setting $(INSTALL_FW_PATH). So you can put
it into a kernel-specific directory if you want to.

'make firmware_install' will install _all_ available firmware files
to /lib/firmware (or $(INSTALL_FW_PATH)), regardless of architecture or
configuration. That's what Fedora is using for its 'kernel-firmware'
package, which is arch-independent and required by the various kernel
binary packages.

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Linus Torvalds wrote:
> 
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>> Can't you see that changing build processes takes time?  Fixing driver disk
>> creation tools take time?  Validating that each driver is packaged with its
>> correct firmware takes time?
> 
> Umm. Can't you see that if we don't start doing it, it will never get done 
> AT ALL?
> 
> Yes, this will take time.

Which, by implication, means that userland is not -already- prepared.

Which means that if you don't have a recent userland of a mainstream 
distro, the result is non-working drivers.

Which is the type of regression I thought we did not want.


> But that's not an argument for not merging it. Quite the reverse.

This is not an either/or proposition, and that is what you are missing.

The change should go in _while at the same time_ not creating regressions.

The change is not the problem.

That the change exists without the ability to recreate similar outputs 
(i.e. firmware in module) is the problem.

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Chris Friesen]

Linus Torvalds wrote:

> Why compile them into the module, when that's just the _inflexible_ model?
> 
> Why not just install them with the module (perhaps in the module 
> directory, perhaps not), and just load them? 
> 
> Really. You _do_ have do to "make modules_install" or similar. That has 
> _already_ got inter-file dependencies (ie some modules simply depend on 
> other modules). I'm not seeing why you cannot accept change.

Even if the end goal is to support more flexible firmware loading, it 
would seem to make sense to allow building the firmware into the 
module--if only as an interim step to avoid breaking existing setups.

Chris

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Linus Torvalds wrote:
> 
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>> Which is why 'make modules_install' installs the firmware, or at least it did
>> before David W pushed upstream.
> 
> So you're literally just about making this be "make modules_install" 
> rather than "make firmware_install"
> 
> Ok. Are you going to be happy if "make modules_install" just copies the 
> firmware files of the affected modules too?

Re-read the above, and what you snipped.  My statement was

* upstream _already_ does this (installs firmware via modules_install)

* it is a good thing, because it means existing build scripts and 
processes will Do The Right Thing and produce a working driver

* thus eliminating the most common problem likely to be encountered

That does not eliminate all problems, because build processes still have 
hardcoded assumptions about outputs.

Restoring firmware-in-module ability closes the last logical hole, last 
fundamental flaw in the scheme, by permitting distros to reproduce 
precisely what they built in 2.6.26.

All of the regressions examples I am citing are cured by 
firmware-in-module, because they are all examples of problems that occur 
when you remove the ability to reproduce the same functional pieces as 
2.6.26.

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Jeff Garzik wrote:
> 
> That does not eliminate all problems, because build processes still have
> hardcoded assumptions about outputs.

Tough. They need to get fixed.

What's the downside of fixing them, again?

		Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Linus Torvalds wrote:
> 
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
> 
>> Henrique de Moraes Holschuh wrote:
>>> On Tue, 15 Jul 2008, Frans Pop wrote:
>>>> If I were able to compile firmware into the modules, the problem would be
>>>> solved in one go.
>>> And this thread would have been shorter, even. I hope someone decides to
>>> write that support instead of complaining ;-)
>> I've volunteered -- but if Linus and David will reject my code out of hand,
>> there is not much point in writing it.
> 
> Quite frankly, I don't think it's something to reject out of hand, but it 
> does sound pretty damn stupid.
> 
> Why compile them into the module, when that's just the _inflexible_ model?

Because it works today.

Because that is what people and their scripts expect, today.

Because people should not be forced to fix all their firmware-related 
breakage immediately, just to boot 2.6.27.


> I'm not seeing why you cannot accept change.

It is highly feasible to do the change without breaking stuff and 
removing choice.

Beer can be great tasting AND less filling.

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Theodore Tso]

On Tue, Jul 15, 2008 at 02:44:46PM -0400, Jeff Garzik wrote:
> All of the regressions examples I am citing are cured by  
> firmware-in-module, because they are all examples of problems that occur  
> when you remove the ability to reproduce the same functional pieces as  
> 2.6.26.

Jeff,

	I think you've said this before, but let me try to help you
cut to the chase.  You are willing to *implement*
firmware-in-the-module for request_firmware(), but you want a
commitment that David Woodhouse and Linus will accept such a patch
before you go off and write it.  Is that right?

	If so, may I suggest you start implementing, instead of
sending e-mails?  For all the time people have spent arguing about it,
maybe it could have been implemented already.

	Once it has been implemented, do you have any further
objections aka ideas about how request_firmware() could be improved?

					- Ted
					(who is hoping we can guide
					 this thread onto slightly more
					 productive avenues)

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Jeff Garzik wrote:
> 
> Because people should not be forced to fix all their firmware-related breakage
> immediately, just to boot 2.6.27.

You're continuing to make an argument that doesn't seem to backed up with 
any actual real problems.

Can you point to real breakage? For real people? 

It sounds like your whole argument literally boils down to "one or two 
people doing something really stupid or odd cannot just fix their setup".

What is the real-life situation where copying the firmware with the 
modules (but still as separate files) actually breaks?

IOW, what _is_ this theoretical breakage? And why is it so deadly 
suddenly? Give us real examples that somehow cannot be fixed?

		Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Tue, 2008-07-15 at 14:55 -0400, Jeff Garzik wrote:
> Linus Torvalds wrote:
> > 
> > On Tue, 15 Jul 2008, Jeff Garzik wrote:
> > 
> >> Henrique de Moraes Holschuh wrote:
> >>> On Tue, 15 Jul 2008, Frans Pop wrote:
> >>>> If I were able to compile firmware into the modules, the problem would be
> >>>> solved in one go.
> >>> And this thread would have been shorter, even. I hope someone decides to
> >>> write that support instead of complaining ;-)
> >> I've volunteered -- but if Linus and David will reject my code out of hand,
> >> there is not much point in writing it.
> > 
> > Quite frankly, I don't think it's something to reject out of hand, but it 
> > does sound pretty damn stupid.
> > 
> > Why compile them into the module, when that's just the _inflexible_ model?
> 
> Because it works today.
> 
> Because that is what people and their scripts expect, today.
> 
> Because people should not be forced to fix all their firmware-related 
> breakage immediately, just to boot 2.6.27.

On one hand, they don't -- the drivers I've updated in 2.6.27 aren't the
ones they need to boot. I deliberately left drivers/net and drivers/scsi
till later. I don't _expect_ any problems, but there's no rush -- we can
iron out any kinks with the drivers I've done already, then merge the
remaining updates into 2.6.28.

One the other hand, they _do_ need to fix such breakage immediately, if
they have it -- because what I'm doing here is nothing new. Most modern
drivers use request_firmware() already.

So if your userspace has this hypothetical "firmware-related breakage"
of which you speak, then it already won't boot on machines with qla2xxx,
which was converted to use request_firmware() in 2005. To pick but one
example out of many.

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Linus Torvalds wrote:
> 
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>> That does not eliminate all problems, because build processes still have
>> hardcoded assumptions about outputs.
> 
> Tough. They need to get fixed.
> 
> What's the downside of fixing them, again?

Yes, they need fixing.

But without firmware-in-module this would be akin to adding kernel 
module support in version 1.0.1, and forcing all distros to fix their 
stuff immediately in order to upgrade from version 1.0.0.

Permitting firmware-in-module means you do not force each distro, build 
script, and user immediately into the new regime, with the associated 
long list of breakage examples.

It's just sad because it feels "not the Linux way".

The way this was rolled out just feels "harder", far less friendly on 
the distros -- our customers -- than most other transitions.  Not as 
hard as vmlinuz->kmodule and IDE->libata transitions, I grant, but still 
  not the way we usually do things.

Nobody was ever objecting to request_firmware().  It was always about 
the removal of firmware-in-module and associated forced breakage on 
distros and build processes.  The fallout of _that_ is the rich source 
of breakage examples I've been pulling from.

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Frans Pop]

On Tuesday 15 July 2008, David Woodhouse wrote:
> On Tue, 2008-07-15 at 15:05 -0300, Henrique de Moraes Holschuh wrote:
> > I believe I read in the previous thread that some distros are already
> > using /lib/firmware/<kernel version>/.
> >
> > There was also the suggestion of moving the entire set of
> > kernel-packaged firmware to /lib/modules/<kernel version>/firmware,
> > probably while keeping /lib/firmware as a second place to look for
> > firmware so that we don't hose any system.
>
> You can do that right now; I expect Ubuntu will, since their userspace
> already looks like that. You just need to set $(INSTALL_FW_PATH) when
> you install the firmware (or modules).

With the disclaimer that I really haven't researched this all that much I 
think we'll end up finding that a multi-level model is needed.

Something like:
- /lib/firmware: intended for externally provided firmware and local
  overrides; try to load from there first (version checks may fail)
- /lib/modules/firmware: not sure if this level is necessary, but
  could be useful for e.g. firmware for out-of-tree drivers
- /lib/modules/$kver/firmware or
  /lib/modules/$kver/$modulepath: firmware built (and possibly packaged
  together with the kernel

Some scheme like this would introduce at least some sanity when it comes 
to keeping control of what is installed and provided (and may therefore 
be removed) by what and to avoid the question "how the hell did that file 
get there and do I still actually need it with my current kernels?".

It sounds to me like $(INSTALL_FW_PATH) is a good start, but maybe even 
too flexible as it does not define a standard: it allows user spaces of 
different distros to diverge which at some point will create its own 
nightmares.
It's also not flexible enough as it only supports a single location for 
firmware while clearly there can be different sources of firmware.

Cheers,
FJP

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Theodore Tso wrote:
> On Tue, Jul 15, 2008 at 02:44:46PM -0400, Jeff Garzik wrote:
>> All of the regressions examples I am citing are cured by  
>> firmware-in-module, because they are all examples of problems that occur  
>> when you remove the ability to reproduce the same functional pieces as  
>> 2.6.26.
> 
> Jeff,
> 
> 	I think you've said this before, but let me try to help you
> cut to the chase.  You are willing to *implement*
> firmware-in-the-module for request_firmware(), but you want a
> commitment that David Woodhouse and Linus will accept such a patch
> before you go off and write it.  Is that right?
> 
> 	If so, may I suggest you start implementing, instead of
> sending e-mails?  For all the time people have spent arguing about it,
> maybe it could have been implemented already.

Already started, in fact, since Linus said he would not reject it out of 
hands.

Obviously that is not acceptance; I know as well as any that acceptance 
does not occur until the moment of upstream merge.


> 	Once it has been implemented, do you have any further
> objections aka ideas about how request_firmware() could be improved?

Further objections?  None major.

The two other minor problems I feel need solving, but are not related to 
breakage or regressions, are:

1) firmware should be able to live alongside the driver.  We don't need 
to start growing firmware/net alongside drivers/net, firmware/scsi 
alongside drivers/scsi, firmware/char alongside drivers/char, etc.

2) firmware should be able to be shipped in final format (binary), 
rather than ihex.


I feel strongly that 2.6.27 should not be released without 
firmware-in-module, for all the reasons mentioned in these threads. 
It's a major regression, IMO, without firmware-in-module ability.

The other stuff (#1, #2 in list above) is small potatoes.

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Jeff Garzik wrote:
> 
> But without firmware-in-module this would be akin to adding kernel module
> support in version 1.0.1, and forcing all distros to fix their stuff
> immediately in order to upgrade from version 1.0.0.

You're just making that up. The distro's can fix it long before they use 
the kernel. 

And the users we can just fix by making request_firmware() look up the 
files itself, if it doesn't already.

You're just making up your bogus arguments.

			Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Jeff Garzik wrote:
> 
> Already started, in fact, since Linus said he would not reject it out of
> hands.

Btw, I need to see it actually solve a real problem. As it is, I really 
think you've just painted yourself into a corner where you can't just 
admit that we can just load the firmware directly.

Could you _please_ entertain at least thinking about that option?

		Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Marcel Holtmann]

Hi Frans,

> > On Mon, 14 Jul 2008, Jeff Garzik wrote:
> >> IMO the newly added /inability/ to build firmware into kernel modules
> >> is a clear regression.
> > 
> > IMO you're being stupid.
> > 
> > How about explainign why it makes any difference what-so-ever?
> > 
> > If you can load the module, you can load the firmware. Claiming anything
> > else is just _stupid_.
> 
> Sure, in theory it's that simple. Here's a concrete example that shows how 
> things are harder in practice.
> 
> I use the 'make deb-pkg' target (from scripts/package) to build my Debian 
> kernel packages from git. So that needs to be adapted to include
> /lib/firmware. No real problem so far.
> 
> So then I build 2.6.27-rc1 and install it. Great.
> 
> You release 2.6.27-rc2 and I build it. Ouch! It fails to install, at least 
> if I want to install it _alongside_ 2.6.27-rc1 or other kernels (which I 
> do!). Why does it fail? Because dpkg's package management does not allow 
> one package to overwrite files already "owned" by another package.
> 
> So, how is this solved by Debian for already existing firmware packages? 
> Basically by making a separate package for each firmware file (or 
> driver). This works because there are not too many of them, but having a 
> huge number of tiny packages is a nightmare by itself.
> 
> But anyway, the dep-pkg target will have to be made smarter than it is now 
> if it's to deal with this [1]. And at least currently it is broken.
> 
> If I were able to compile firmware into the modules, the problem would be 
> solved in one go.
> 
> I don't know how the Debian kernel team plans to deal with this for distro 
> kernel packages. They probably _do_ want to keep them separate [2]. Maybe 
> by grouping firmware for really common drivers into 
> firmware-basic-drivers or something along those lines.
> 
> Cheers,
> FJP
> 
> [1] Only quick solution I see is to have it install the firmware in a 
> versioned directory and have the postinst copy things from there to
> /lib/firmware.

using /lib/firmware/`uname -r`/ is actually not a bad idea. You only
have to fix udev to actually include this in the list of directories to
look for firmware files. Also Ubuntu is already doing this.

Regards

Marcel

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Marcel Holtmann]

Hi Linus,

> > David W has made it very clear that the eventual intent of this patch series
> > is to remove all firmware from the kernel tree and have it shipped as a
> > seperate package.
> 
> I don't think that kind of black-and-white model is necessarily 
> attainable. Nor do I think it's even _good_.
> 
> It's probably the right thign for _some_ firmware, assuming we have models 
> that just make it easy enough - for drivers where it's generally better 
> for distros to get the firmware from vendors rather than the kernel.
> 
> But there is certainly nothing inherently good in forcing a split.
> 
> I think both sides here need to calm down and stop being so extreme.
> 
> I do want request_firmware() to be the way to load firmware (and possibly 
> other configuration data too, for that matter - I don't think it's 
> necessarily wrong for people to "misuse" it for other long-term data 
> despite the name), and I do want people to be able to update firmware 
> easily without having to force a kernel bump, but I also don't think it 
> means that firmware has to go away from the kernel sources.

actually the PCMCIA subsystem is already using request_firmware() to
load external CIS data to overwrite broken ones.

Regards

Marcel

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Linus Torvalds wrote:
> 
> Could you _please_ entertain at least thinking about that option?

.. and the reason I ask you to do that is quite frankly, you seem to want 
to just hide the thing under a rug - and _encouraging_ people to not even 
try to work with a new and more flexible model.

For example, there's nothing wrong with having people replace their 
firmware when they want to. Equally, there's nothing wrong in having 
distros put stable firmware into another package, so that you don't need 
to have <n> different versions of the firmware in <n> different kernel 
module directories just because you have <n> different versions of the 
module itself.

So why do you want to work so hard at not just encouraging people to get 
the firmware loaded dynamically from the filesystem?

		Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Marcel Holtmann]

Hi Linus,

> > Why don't you just take the kernel-supplied firmware and make it part of 
> > the kernel package? The same way the kernel-supplied modules are part of 
> > it?
> 
> And, btw, if there is something we can do to help, we obviously should.
> 
> So ask yourself (and DavidW) things like
> 
>  - Is "make firmware_install" not enough for the package build system?
> 
>  - Does it need some other way to figure out what firmware is needed for 
>    just the configured set of drivers, eg "make firmware_list"?
> 
>  - Do we need to make the firmware loader have multiple directories it 
>    looks at? Do we want first a kernel-version-specific one, then a 
>    generic fallback one, so that we can easily have per-kernel firmware, 
>    the same way we have per-kernel modules?

as just mentioned in another post. If you look at Ubuntu, their udev
helper that actually looks for the firmware and loads it into the kernel
does exactly this. Historically we have also old paths since the
agreement on /lib/firmware came later, but that is userspace policy
anyway and not part of the kernel.

Regards

Marcel

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Marcel Holtmann wrote:
> 
> using /lib/firmware/`uname -r`/ is actually not a bad idea. You only
> have to fix udev to actually include this in the list of directories to
> look for firmware files. Also Ubuntu is already doing this.

I really don't think we need to even "fix udev".

Why don't we just load it ourselves? Esepcially as there are probably 
places that try to avoid udev entirely, or at least use a very 
cut-down-version.

We should be fairly trivially able to be _entirely_ backwards compatible 
with any sane setup (not the _sane_ part! It implies that people don't 
copy individual modules around by hand!), with no actual breakage or need 
for distros to even update anything at all - just make the kernel able to 
look up binary blobs in the same place it installed them.

That sounds like the RightThing(tm) to do _regardless_ of any other 
issues, doesn't it? If the kernel installs it in some known place, why 
should it not just read them from that known place?

		Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Tue, 2008-07-15 at 14:58 -0400, Theodore Tso wrote:
> On Tue, Jul 15, 2008 at 02:44:46PM -0400, Jeff Garzik wrote:
> > All of the regressions examples I am citing are cured by  
> > firmware-in-module, because they are all examples of problems that occur  
> > when you remove the ability to reproduce the same functional pieces as  
> > 2.6.26.
> 
> Jeff,
> 
> 	I think you've said this before, but let me try to help you
> cut to the chase.  You are willing to *implement*
> firmware-in-the-module for request_firmware(), but you want a
> commitment that David Woodhouse and Linus will accept such a patch
> before you go off and write it.  Is that right?

The more important question is whether _Linus_ will take it, and of
course I can't answer for him.

Personally I think it's pointless. But since it's only _mildly_
counterproductive, if you really want to do it then I wouldn't actively
oppose it unless it turns out to be a mess to implement, or unless it's
limited to working only with the in-tree firmware.

If you implement it cleanly and optionally (and defaulting off), and if
it works properly with out-of-tree firmware too (so I can build a
libertas usb8xxx.ko module with the firmware included, for example),
then I certainly wouldn't throw my toys out of the pram over it.
Ideally, it would work for out-of-tree drivers (make M=... modules) too.

I was thinking that we could maybe post-process .ko files (which we do
already for other reasons), and add a special section in them containing
the firmware which is mentioned in MODULE_FIRMWARE() tags. The special
section is how we do it for the kernel too. That would take a little bit
of code in module.c to keep track of those sections and add them to a
list somewhere that firmware_class.c can see them, and then you have to
think about lifetime and locking issues.

But those were just preliminary thoughts -- I wasn't really planning to
implement it so I didn't go through it in detail.

An alternative approach might be to turn firmware blobs into actual
modules, which just register themselves with the firmware loader in
module_init(). I'm not sure I like that approach though, and depmod
wouldn't 'notice' them automatically so it wouldn't work with current
userspace. I've tried to be very careful to ensure that anything we do
here is entirely compatible with existing userspace.

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Peter T. Breuer]

In article <alpine.LFD.1.10.0807151122010.3017@woody.linux-foundation.org> Linus wrote:
> Why not just install them with the module (perhaps in the module 
> directory, perhaps not), and just load them? 

I also don't get why one can't put them in a module wrapper in some OTHER
dir as

__init__ fwFOO = something_huge_made_from_a_string ;
EXPORT_SYMBOL(fwFOO);

and have the real driver module just reference fwFOO in its init sequence.
Modprobe can deal with where to look for it. No kernel source need be
tainted.

Peter

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Marcel Holtmann]

Hi Linus,

> > using /lib/firmware/`uname -r`/ is actually not a bad idea. You only
> > have to fix udev to actually include this in the list of directories to
> > look for firmware files. Also Ubuntu is already doing this.
> 
> I really don't think we need to even "fix udev".

fixing udev means adding an extra path to it :)

> Why don't we just load it ourselves? Esepcially as there are probably 
> places that try to avoid udev entirely, or at least use a very 
> cut-down-version.
> 
> We should be fairly trivially able to be _entirely_ backwards compatible 
> with any sane setup (not the _sane_ part! It implies that people don't 
> copy individual modules around by hand!), with no actual breakage or need 
> for distros to even update anything at all - just make the kernel able to 
> look up binary blobs in the same place it installed them.
> 
> That sounds like the RightThing(tm) to do _regardless_ of any other 
> issues, doesn't it? If the kernel installs it in some known place, why 
> should it not just read them from that known place?

Personally I am against hacking file loading from within the kernel.
This is why we involved udev (or hotplug in the beginning). It allows
you to have userspace policy for the search paths etc.

Regards

Marcel

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Linus Torvalds wrote:
>
> [..] just make the kernel able to look up binary blobs in the same place 
> it installed them.

Obviously, we'd want to make it dynamically configurable, ie the same way 
we specify where to find 'modprobe', let people specify (in 
/proc/kernel/firmware-dir or something) where to find the firmware. Just 
default it to something sane (that obviously matches where we installed 
them).

I'm not a huge udevd fan, obviously. I think we can short-circuit this all 
in the name of expediency, with udevd as a fallback, not as the only way.

			Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Tue, 2008-07-15 at 12:27 -0700, Linus Torvalds wrote:
> 
> On Tue, 15 Jul 2008, Marcel Holtmann wrote:
> > 
> > using /lib/firmware/`uname -r`/ is actually not a bad idea. You only
> > have to fix udev to actually include this in the list of directories to
> > look for firmware files. Also Ubuntu is already doing this.
> 
> I really don't think we need to even "fix udev".
> 
> Why don't we just load it ourselves? Esepcially as there are probably 
> places that try to avoid udev entirely, or at least use a very 
> cut-down-version.
> 
> We should be fairly trivially able to be _entirely_ backwards compatible 
> with any sane setup (not the _sane_ part! It implies that people don't 
> copy individual modules around by hand!), with no actual breakage or need 
> for distros to even update anything at all - just make the kernel able to 
> look up binary blobs in the same place it installed them.
> 
> That sounds like the RightThing(tm) to do _regardless_ of any other 
> issues, doesn't it? If the kernel installs it in some known place, why 
> should it not just read them from that known place?

I'm unconvinced that the kernel should be setting this kind of policy.

But I suppose if you make it tunable in sysfs and just switch to calling
do_filp_open() directly from firmware_class.c instead of punting to
userspace, that might work.

It leaves you with less flexibility -- it would prevent stuff like the
udev trick I posted a week or so back to fix the Intel microcode loader
by automatically generating the needed binary blob on the fly from
microcode.dat, for example.

We also have a long tradition of pointing and laughing at people who
want to call open() from within the kernel. But it _could_ work,
certainly.

I'm not really convinced it helps though. The script which creates an
initrd still needs to look at the MODULE_FIRMWARE() tag and include the
right firmware file. If that's broken, you're screwed anyway. And that
was true in 2.6.25 too.

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Marcel Holtmann wrote:
> 
> Personally I am against hacking file loading from within the kernel.
> This is why we involved udev (or hotplug in the beginning). It allows
> you to have userspace policy for the search paths etc.

Well, I'm personally against _forcing_ people to use udev. 

Yes, desktop distros generally will do so, since there are lots of complex 
issues etc. But I don't think we've generally forced it on people if they 
don't want it (ie I haven't tried it personally since the distro I used 
started using udev, but I think you can still just ignore udev events and 
set everything up statically).

And I don't think that's wrong. Which implies that setup things should 
still generally at least allow us to avoid udev.

(But maybe I'm wrong, and everybody already uses udev just because they 
couldn't be bothered not to).

			Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Marcel Holtmann]

Hi Henrique,

> > You release 2.6.27-rc2 and I build it. Ouch! It fails to install, at least 
> > if I want to install it _alongside_ 2.6.27-rc1 or other kernels (which I 
> > do!). Why does it fail? Because dpkg's package management does not allow 
> > one package to overwrite files already "owned" by another package.
> 
> I believe I read in the previous thread that some distros are already using
> /lib/firmware/<kernel version>/.
> 
> There was also the suggestion of moving the entire set of kernel-packaged
> firmware to /lib/modules/<kernel version>/firmware, probably while keeping
> /lib/firmware as a second place to look for firmware so that we don't hose
> any system.

can we please leave the /lib/modules/*/firmware idea out of it. While
this thread is about the kernel drivers that need firmware loading, but
it is not the whole story. We do have hardware that needs a firmware
loading process that takes place entirely in userspace. In some USB and
UART cases we can do the firmware loading step before any kernel driver
takes control of that device and currently these firmware loading
helpers look into /lib/firmware/ where every firmware file should be
placed.

Also the /lib/firmware directory was chosen a long time ago and you find
the discussion on the linux-hotplug mailing list archive if you are
interested.

That said, using /lib/firmware/`uname -r`/ like Ubuntu supports right
now and then fall back to /lib/firmware looks like a sane approach to
me. However as David mentioned, this needs patches to udev first before
we go any step further.

Regards

Marcel

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[david]

On Tue, 15 Jul 2008, Linus Torvalds wrote:

> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>>
>> Because people should not be forced to fix all their firmware-related breakage
>> immediately, just to boot 2.6.27.
>
> You're continuing to make an argument that doesn't seem to backed up with
> any actual real problems.
>
> Can you point to real breakage? For real people?

several kernel devs ran into this with -next (Jeff can name names if 
needed)

> It sounds like your whole argument literally boils down to "one or two
> people doing something really stupid or odd cannot just fix their setup".
>
> What is the real-life situation where copying the firmware with the
> modules (but still as separate files) actually breaks?
>
> IOW, what _is_ this theoretical breakage? And why is it so deadly
> suddenly? Give us real examples that somehow cannot be fixed?

there are no cases that cannot somehow be fixed. the desire is to make it 
possible to avoid having to find and fix everything before booting 2.6.27, 
while still making it possible to compile a kernel with all the new 
capabilities for those who want to fix everything now.

David Lang

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Marcel Holtmann]

Hi Linus,

> > Personally I am against hacking file loading from within the kernel.
> > This is why we involved udev (or hotplug in the beginning). It allows
> > you to have userspace policy for the search paths etc.
> 
> Well, I'm personally against _forcing_ people to use udev. 
> 
> Yes, desktop distros generally will do so, since there are lots of complex 
> issues etc. But I don't think we've generally forced it on people if they 
> don't want it (ie I haven't tried it personally since the distro I used 
> started using udev, but I think you can still just ignore udev events and 
> set everything up statically).
> 
> And I don't think that's wrong. Which implies that setup things should 
> still generally at least allow us to avoid udev.
> 
> (But maybe I'm wrong, and everybody already uses udev just because they 
> couldn't be bothered not to).

I actually think that is the case here. udev is so lightweight and
really slim nowadays that not using it is a big drawback. There might be
some system out there that are still using a static /dev directory, but
for sure there are not many and even the embedded world doesn't really
mind.

Also I think the /sbin/hotplug thing is still available (not verified
this) and thus calling a firmware loading helper script with a udev
replacement would be possible. Someone correct me here if I am wrong.

Regards

Marcel

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, David Woodhouse wrote:
> 
> I'm unconvinced that the kernel should be setting this kind of policy.

I do think it should probably be conditional, but I think that's true of 
udevd itself too, so hey, it cuts both ways.

> But I suppose if you make it tunable in sysfs and just switch to calling
> do_filp_open() directly from firmware_class.c instead of punting to
> userspace, that might work.

Yup. And then you can disable it either statically (config option) or by 
writing an invalid path into /proc/sys/kernel/firmware-dir or whatever.

Or you can just decide that if you find something in the kernel-specific 
firmware directory, then it should always take precedence over whatever 
udev rules. Which sounds good to me anyway.

Maybe you really do have some very kernel-specific issue (ie you're trying 
a new driver that can handle a new experimental firmware, but you don't 
want your old fall-back kernels to use it because you just fixed the bug 
that makes it work again).

Requiring you to write udevd scripts for that sounds insane. I wouldn't 
even know where to start.

So kernel-specific directories do make sense. As does the whole "I don't 
want to handle the pain that is udev scripts".

			Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Marcel Holtmann]

Hi Linus,

> Requiring you to write udevd scripts for that sounds insane. I wouldn't 
> even know where to start.
> 
> So kernel-specific directories do make sense. As does the whole "I don't 
> want to handle the pain that is udev scripts".

you don't have to. We extend udev once and then it will always work. As
mentioned before, Ubuntu is already doing this. They never bothered to
submit this upstream and I didn't really care since I have all my
firmware files in /lib/firmware and had no need to add versions to it.

Regards

Marcel

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Tue, 2008-07-15 at 12:49 -0700, Linus Torvalds wrote:
> So kernel-specific directories do make sense. As does the whole "I
> don't want to handle the pain that is udev scripts".

I'm not violently opposed to kernel-specific directories. Ubuntu does it
that way already, and it may well make sense. I'm not sure it's really
necessary either, but it's an option.

Unless we really are going to make the kernel open files for itself,
though, I think it's too early for us to think about making that change
in the kernel. We'd need userspace to cope with it that way _first_.

(When I say 'in the kernel', I think the only place this 'policy' is yet
implemented is the default for $(INSTALL_FW_PATH) in modules_install and
firmware_install make targets, right?)

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Linus Torvalds wrote:
> 
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>> Because people should not be forced to fix all their firmware-related breakage
>> immediately, just to boot 2.6.27.
> 
> You're continuing to make an argument that doesn't seem to backed up with 
> any actual real problems.
> 
> Can you point to real breakage? For real people? 

Already posted a list.


> It sounds like your whole argument literally boils down to "one or two 
> people doing something really stupid or odd cannot just fix their setup".
> 
> What is the real-life situation where copying the firmware with the 
> modules (but still as separate files) actually breaks?

The breakage comes from all the existing-at-this-point-today processes 
that will not copy the firmware with the module.

We're not talking about one or two people, we are talking about projects 
being actively used.

Which in turn means not only those projects need to immediately fix 
stuff to make 2.6.27 work, but users who build and test kernels are left 
to pick up the pieces when their drivers break too.


> IOW, what _is_ this theoretical breakage? And why is it so deadly 
> suddenly? Give us real examples that somehow cannot be fixed?

It's not theoretical, we have already run into non-working drivers due 
to build process changes.  And those were the smart guys, the kernel 
hackers.

All of the examples I have listed can certainly be fixed -- well, except 
the "new kernel, old system" case -- sometimes easily.

* the consequences of the breakage -- 100% non-working driver, possibly 
unbootable system

* the likelihood of breakage -- anything that is not a recent version of 
a mainstream distro WILL have problems, because it simply does not know 
about the firmware that must follow the module whereever it goes.

* the need to immediately add/fix userland and build processes, just to 
keep the same driver set working.

* the need for time, to figure out if you are even affected by this change

* the need for time, to plan the best method to implement firmware 
deployment in your distro


Without firmware-in-module, it is a basic fact that you are forcing 
everyone to fix their stuff, simply to be able to use 2.6.27 like they 
did 2.6.26.

That's unfriendly to distros, users, and kernel testers.

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Marcel Holtmann]

Hi David,

> > > Which is why 'make modules_install' installs the firmware, or at least it did
> > > before David W pushed upstream.
> > 
> > So you're literally just about making this be "make modules_install" 
> > rather than "make firmware_install"
> > 
> > Ok. Are you going to be happy if "make modules_install" just copies the 
> > firmware files of the affected modules too?
> 
> I did that already. I'm not _entirely_ sure it was the right thing to
> do, but it does help to reduce surprises when people don't realise that
> a driver they're using now has separate firmware.
> 
> I don't believe it was the result of a constructive request from Jeff; I
> think someone else suggested it in response to his whining. It may even
> have been my idea, but I don't recall.
> 
> This is the current situation:
> 
> 'make modules_install' will install the firmware required by the modules
> it installs. It'll install that to $(INSTALL_MOD_PATH)/lib/firmware/
> unless you override that by setting $(INSTALL_FW_PATH). So you can put
> it into a kernel-specific directory if you want to.
> 
> 'make firmware_install' will install _all_ available firmware files
> to /lib/firmware (or $(INSTALL_FW_PATH)), regardless of architecture or
> configuration. That's what Fedora is using for its 'kernel-firmware'
> package, which is arch-independent and required by the various kernel
> binary packages.

I really think we should use /lib/firmware/`uname -r`/. I do see the
point here that I don't wanna overwrite existing firmware from other
installed kernels. Especially if modules_install will install the
firmware files.

So in case of non-Ubuntu distros we have to push a fix to udev, but that
is not a big deal. It should be a one-line change if I am not mistaken.

Regards

Marcel

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Tue, 2008-07-15 at 22:00 +0200, Marcel Holtmann wrote:
> I really think we should use /lib/firmware/`uname -r`/. I do see the
> point here that I don't wanna overwrite existing firmware from other
> installed kernels. Especially if modules_install will install the
> firmware files.
> 
> So in case of non-Ubuntu distros we have to push a fix to udev, but that
> is not a big deal. It should be a one-line change if I am not mistaken.

My concern with that is that even though it's a one-line change, it's a
one-line change which people don't already _have_. We really do need to
remain compatible with existing setups.

For _now_, I think it's much better to leave it in /lib/firmware where
existing upstream udev will find it. Let's change udev ASAP and then we
can talk about when it makes sense to change the default setting of
$(INSTALL_FW_PATH) to match.

I'm not too worried about overwriting existing firmware
in /lib/firmware. We very rarely change firmware anyway, when we _do_
change it it's almost always compatible rather than having ABI changes.
On the rare occasions that remain, the incompatibly changed firmware
really ought to have a new filename so that older drivers don't pick it
up when they want the old one.

It's not a _complete_ non-problem, but it's fairly close. And the
incompatibility with existing upstream udev would be much more of an
issue, I think.

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Marcel Holtmann wrote:
> 
> you don't have to. We extend udev once and then it will always work.

Umm. The thing is, people running new kernels with old user land is not 
just supposed to work, it's _really_ supposed to work.

It's what I do. Something that breaks that has to have damn good reasons 
to break it.

So I do not disagree with Jeff on that point _at_all_. I'm in violent 
agreement with Jeff on the fact that we should not require system updates 
for the kernel to do the right thing.

The thing I disagree with Jeff on is that he then seems to turn that into 
something very negative ("let's not separate the firmware at all").

And I'd much rather just fix it. And that means that if people can point 
to udevd's that get confused - or lack of udevd's entirely - both of which 
sound very likely to me, then we should have a graceful fallback position.

And just supporting the notion of loading the firmware directly sounds 
like an obvious such case. It may not be the _only_ solution, for example, 
which is why I'd actually like to see people point to the _actual_ 
reported problems.

IOW, the problems shouldn't a "don't do this" thing. They should be a "ok, 
that problem happened, we can solve it by doing X".

			Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Marcel Holtmann]

Hi David,

> > So kernel-specific directories do make sense. As does the whole "I
> > don't want to handle the pain that is udev scripts".
> 
> I'm not violently opposed to kernel-specific directories. Ubuntu does it
> that way already, and it may well make sense. I'm not sure it's really
> necessary either, but it's an option.

then lets get the udev extension merged upstream (since it doesn't do
any harm anyway) and then install everything in /lib/firmware/`uname -r`
and see how that works out.

So Ubuntu will obviously not break since it does this already. For
Fedora deploying new packages for 8 and 9 should not be a problem at all
once the maintainer is convinced that it is a good idea. They update
packages all the time anyway. For the other distros, I don't know.

Regards

Marcel

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Jeff Garzik wrote:
> 
> Already posted a list.

No you didn't.

You posted a list of problems you have from doing odd and stupid things. 
Not the reports, not the explanations of what those odd and stupid 
things actually were in practice, and why they had to be that stupid.

Quite frankly, I suspect that it's all a matter of "people compiled their 
own kernels, and didn't copy the new files, because they just didn't 
realize they needed to".

Which we really don't even need kernel fixes fot, it just needs some 
education. BUT. If we knew what the details were, we could make the need 
for education even less, so that it would just work with whatever they do.

But yes, if it fundamentally boils down do "you had a script that copied 
named files by hand", then that script needs fixing. But that's somethign 
that somebody who compiles his own kernel (especially in an odd 
environment, apparently) just needs to do.

Sometimes build updates need people to fix things. We find gcc bugs, we 
require people to update. The whole "new kernels should work with old 
distributions" does not mean that people building the kernels might not 
have to do _something_ special.

		Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Marcel Holtmann wrote:
> 
> So Ubuntu will obviously not break since it does this already. For
> Fedora deploying new packages for 8 and 9 should not be a problem at all
> once the maintainer is convinced that it is a good idea. They update
> packages all the time anyway. For the other distros, I don't know.

Umm. Again, Marcel. We don't require new user land to work.

But just loading the thing on our own does solve this problem. But udev we 
simply cannot require people to magically have a new version of.

		Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Marcel Holtmann]

Hi Linus,

> > you don't have to. We extend udev once and then it will always work.
> 
> Umm. The thing is, people running new kernels with old user land is not 
> just supposed to work, it's _really_ supposed to work.
> 
> It's what I do. Something that breaks that has to have damn good reasons 
> to break it.

so request_firmware() is well established and works. It is used by a
bunch of drivers at the moment.

If we do wanna make more heavy use of it (which seems everybody is in
agreement) then we need to make some userspace changes like adding the
kernel version to keep it sane. This can be done step by step to make
sure we keep the current status quo.

And as David mentioned multiple times, he left drivers/{net,scsi} out of
it for now.

> So I do not disagree with Jeff on that point _at_all_. I'm in violent 
> agreement with Jeff on the fact that we should not require system updates 
> for the kernel to do the right thing.
> 
> The thing I disagree with Jeff on is that he then seems to turn that into 
> something very negative ("let's not separate the firmware at all").
> 
> And I'd much rather just fix it. And that means that if people can point 
> to udevd's that get confused - or lack of udevd's entirely - both of which 
> sound very likely to me, then we should have a graceful fallback position.
> 
> And just supporting the notion of loading the firmware directly sounds 
> like an obvious such case. It may not be the _only_ solution, for example, 
> which is why I'd actually like to see people point to the _actual_ 
> reported problems.

I really don't see that problem here since request_firmware() has been
around for so long now and so many drivers are using it nowadays that I
don't see your point here on how it will break.

However if you wanna make the kernel do the loading from the filesystem
directly, I am actually fine with it. We were opposed to that idea,
because it was always kinda hackish to have the kernel load files and it
felt wrong to do it this way. Policy decisions on directory layouts
belong into the userspace and not the kernel. At least that is my take
on this.

Regards

Marcel

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Marcel Holtmann]

Hi Linus,

> > So Ubuntu will obviously not break since it does this already. For
> > Fedora deploying new packages for 8 and 9 should not be a problem at all
> > once the maintainer is convinced that it is a good idea. They update
> > packages all the time anyway. For the other distros, I don't know.
> 
> Umm. Again, Marcel. We don't require new user land to work.
> 
> But just loading the thing on our own does solve this problem. But udev we 
> simply cannot require people to magically have a new version of.

we might have to do a step by step upgrade for this. Something like
install everything in /lib/firmware/`uname -r`/ and create symlinks for
a grace period of the next sixt month or something like this.

These are details that have be sorted out, but should not stop us to get
this right.

And before you say anything. You can expect that firmware loading
from /lib/firmware works almost everywhere. At least this has been for
the last 3 years or so. Remember that we do have request_firmware() even
in the latest 2.4 kernels.

Regards

Marcel

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Linus Torvalds wrote:
> 
> On Tue, 15 Jul 2008, Marcel Holtmann wrote:
>> you don't have to. We extend udev once and then it will always work.
> 
> Umm. The thing is, people running new kernels with old user land is not 
> just supposed to work, it's _really_ supposed to work.
> 
> It's what I do. Something that breaks that has to have damn good reasons 
> to break it.
> 
> So I do not disagree with Jeff on that point _at_all_. I'm in violent 
> agreement with Jeff on the fact that we should not require system updates 
> for the kernel to do the right thing.
> 
> The thing I disagree with Jeff on is that he then seems to turn that into 
> something very negative ("let's not separate the firmware at all").

How many times do I have to say you misunderstand?

I'm happy with separating the firmware at the source code level.  I 
never objected to that.

All I want is the _option_ to compile firmware into a module.  We are 
talking about compiled output, here.

That _option_ has the desireable properties of

* working on older userland, even with DavidW's tg3 and scsi patches

* retaining an option several kernel hackers and users find useful

* permitting use of 2.6.27 without _immediately_ having to fix stuff


> And I'd much rather just fix it. And that means that if people can point 
> to udevd's that get confused - or lack of udevd's entirely - both of which 
> sound very likely to me, then we should have a graceful fallback position.
> 
> And just supporting the notion of loading the firmware directly sounds 
> like an obvious such case. It may not be the _only_ solution, for example, 
> which is why I'd actually like to see people point to the _actual_ 
> reported problems.
> 
> IOW, the problems shouldn't a "don't do this" thing. They should be a "ok, 
> that problem happened, we can solve it by doing X".

Stop using strawmen, please.

I never said "don't do that".  Ever.

I said "when you do that, don't remove ability to compile firmware into 
the driver."

And as has been stated repeatedly, the compiling firmware into the 
driver clearly DOES NOT exclude the more flexible, load-from-userland 
request_firmware() model.



To be extremely concrete, firmware-in-module is

	* add Kconfig option (kernel-wide or per-driver, dunno) asking
	  "build firmware into drivers, as before?"

	* tweak build process to build firmware into foo.ko output,
	  probably in a specially marked ELF section

	* get request_firmware() to automatically notice that the
	  MODULE_FIRMWARE() was built into this driver, and to
	  look at the special ELF section for its data

See?  That is not a "don't do that."  That has never been "don't do that."

That is the one major request here, one that enables us to avoid all 
those potential breakages I've been listing, and the many more breakages 
of which I'm unaware.  It enables use of older userlands.

And yes, it enables "weirdos" like me to continue building the firmware 
into the driver, because it has a proven track record of high 
reliability and simplicity.

All without stomping on the flexibility of request_firmware().

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Marcel Holtmann]

Hi David,

> > I really think we should use /lib/firmware/`uname -r`/. I do see the
> > point here that I don't wanna overwrite existing firmware from other
> > installed kernels. Especially if modules_install will install the
> > firmware files.
> > 
> > So in case of non-Ubuntu distros we have to push a fix to udev, but that
> > is not a big deal. It should be a one-line change if I am not mistaken.
> 
> My concern with that is that even though it's a one-line change, it's a
> one-line change which people don't already _have_. We really do need to
> remain compatible with existing setups.
> 
> For _now_, I think it's much better to leave it in /lib/firmware where
> existing upstream udev will find it. Let's change udev ASAP and then we
> can talk about when it makes sense to change the default setting of
> $(INSTALL_FW_PATH) to match.

fair enough. What about keeping symlink in /lib/firmware, but actually
doing the install in a versioned directory?

> I'm not too worried about overwriting existing firmware
> in /lib/firmware. We very rarely change firmware anyway, when we _do_
> change it it's almost always compatible rather than having ABI changes.
> On the rare occasions that remain, the incompatibly changed firmware
> really ought to have a new filename so that older drivers don't pick it
> up when they want the old one.

Actually in some case we do. I think about the case where I have an two
years old kernel that came with the distro and would need that firmware
to make the driver work. However now we have the new firmware and it
falls over.

Breaking a distro install is worse then breaking a new kernel. At least
this is how I see it. I do like to have the distro setup as fallback or
rescue when I mess up.

Regards

Marcel

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Marcel Holtmann wrote:
> 
> so request_firmware() is well established and works. It is used by a
> bunch of drivers at the moment.

That's not the question.

The question is whether it works with the *new* users.

Which it would apparently NOT do for everybody if we put the new drivers 
in any other place than /lib/firmware, very much including a kernel- 
specific subdirectory there.

		Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Marcel Holtmann wrote:
> I really don't see that problem here since request_firmware() has been
> around for so long now and so many drivers are using it nowadays that I
> don't see your point here on how it will break.


There is no fundamental breakage in the new request_firmware() system.

However, it is misleading to state

	what I'm doing here is nothing new. Most modern
	drivers use request_firmware() already.

because we are not talking about new drivers, we are talking about 
existing drivers changing en masse from "not requiring firmware" state 
to "requiring firmware" state.

We are also talking about a totally new source of critically important 
data (the firmware, coming from the kernel tree), and scripts must be 
updated to take that into account.

Sure, in many cases the fix may even be a one-liner...  but you still 
need that fix, in order to continue to use drivers that were working in 
2.6.26.

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Rafael J. Wysocki]

On Tuesday, 15 of July 2008, David Woodhouse wrote:
> On Mon, 2008-07-14 at 16:41 -0700, Andrew Morton wrote:
> > On Mon, 14 Jul 2008 16:23:26 -0700
> > David Woodhouse <dwmw2@infradead.org> wrote:
> > 
> > > Linus, please pull from the for-2.6.27 branch of:
> > > 	git://git.infradead.org/users/dwmw2/firmware-2.6.git for-2.6.27
> > 
> > The firmware flamewars seem to have subsided lately.  Is everyone happy
> > with this now?
> 
> I've deliberately left drivers/net out of this pull request... all
> _other_ maintainers seemed quite happy to see their drivers updated to
> use request_firmware().

Have you fixed the issue with FORCE in line 117 of scripts/Makefile.host?

Rafael

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Willy Tarreau]

On Tue, Jul 15, 2008 at 04:26:41PM -0400, Jeff Garzik wrote:
> To be extremely concrete, firmware-in-module is
> 
> 	* add Kconfig option (kernel-wide or per-driver, dunno) asking
> 	  "build firmware into drivers, as before?"
> 
> 	* tweak build process to build firmware into foo.ko output,
> 	  probably in a specially marked ELF section
> 
> 	* get request_firmware() to automatically notice that the
> 	  MODULE_FIRMWARE() was built into this driver, and to
> 	  look at the special ELF section for its data

Jeff, just thinking, wouldn't it be slightly easier to move the firmware
in a separate module on its own and just add a dependency, so that foo.ko
automatically loads foo-fw.ko ? I know it will be slightly differente, but
would not change in-site deployment workflows nor installed scripts.

Regards,
Willy

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Tue, 2008-07-15 at 22:45 +0200, Rafael J. Wysocki wrote:
> Have you fixed the issue with FORCE in line 117 of
> scripts/Makefile.host?

I need to hassle Sam about that.

Sam, why do we have FORCE on the rules for $(hostprogs-y)? It forces the
ihex2fw tool to be rebuilt, which in turn forces all .HEX->.fw
conversions to happen again, which means that 'make modules_install'
wants to write to the source tree. That's a problem in some strange
situations like Rafael's, where he builds on another host and exports
the tree read-only by NFS.

Can we do this? ...

diff --git a/scripts/Makefile.host b/scripts/Makefile.host
index 1ac414f..c0b6a86 100644
--- a/scripts/Makefile.host
+++ b/scripts/Makefile.host
@@ -114,7 +114,7 @@ hostcxx_flags  = -Wp,-MD,$(depfile) $(__hostcxx_flags)
 quiet_cmd_host-csingle 	= HOSTCC  $@
       cmd_host-csingle	= $(HOSTCC) $(hostc_flags) -o $@ $< \
 	  	$(HOST_LOADLIBES) $(HOSTLOADLIBES_$(@F))
-$(host-csingle): $(obj)/%: $(src)/%.c FORCE
+$(host-csingle): $(obj)/%: $(src)/%.c
 	$(call if_changed_dep,host-csingle)
 
 # Link an executable based on list of .o files, all plain c

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Linus Torvalds wrote:
> 
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>> Already posted a list.
> 
> No you didn't.
> 
> You posted a list of problems you have from doing odd and stupid things. 
> Not the reports, not the explanations of what those odd and stupid 
> things actually were in practice, and why they had to be that stupid.
> 
> Quite frankly, I suspect that it's all a matter of "people compiled their 
> own kernels, and didn't copy the new files, because they just didn't 
> realize they needed to".

No, I spelled out a list of projects, what would break, and how it would 
break.

Again,

* Red Hat driver disks.  The build process is unaware of the need to 
bundle firmware.  Any use with Fedora (or, later, RHEL) will result in a 
non-working driver, for the simple reason that firmware is not copied 
onto the image.

* Embedded projects like OpenWRT, that divide up kernel modules into 
multiple packages.  Since it's not just one big package, you have to 
update each package's list of files to include the newly split-out 
firmwares.  Failure to do so results in a successful build set of 
packages... of drivers that lack their firmware.

* Router and embedded system image builders like tomsrtbt, with similar 
problem to RH driver disks:  they simply do not know that a firmware 
needs to be copied into the image, alongside the tg3 driver.

* Older userlands where, axiomatically, the mkinitrd and other firmware 
fixes have not been made, and may never be made.

Yes, it is a matter of education.

Yes, it is a matter of fixes.

That does not change the fact that 2.6.27 will mean non-working drivers 
for these situations, and there is a simple remedy for this entire class 
of problems:  retain ability to compile firmware into the module.

That does not change the fact that useful testers may be turned away 
from 2.6.2[78] while waiting for Red Hat or OpenWRT or whomever to fix 
their stuff.

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Tue, 2008-07-15 at 13:49 -0700, David Woodhouse wrote:
> Sam, why do we have FORCE on the rules for $(hostprogs-y)? It forces the
> ihex2fw tool to be rebuilt, which in turn forces all .HEX->.fw
> conversions to happen again, which means that 'make modules_install'
> wants to write to the source tree. That's a problem in some strange
> situations like Rafael's, where he builds on another host and exports
> the tree read-only by NFS.
> 
> Can we do this? ...

Alternatively, Rafael, does this do it for you?

--- a/firmware/Makefile
+++ b/firmware/Makefile
@@ -150,11 +150,11 @@ $(obj)/%: $(obj)/%.ihex | $(objtree)/$(obj)/$$(dir %)
 # is actually meaningful, because the firmware has to be loaded in a certain
 # order rather than as a single binary blob. Thus, we convert them into our
 # more compact binary representation of ihex records (<linux/ihex.h>)
-$(obj)/%.fw: $(obj)/%.HEX $(obj)/ihex2fw | $(objtree)/$(obj)/$$(dir %)
+$(obj)/%.fw: $(obj)/%.HEX | $(obj)/ihex2fw $(objtree)/$(obj)/$$(dir %)
 	$(call cmd,ihex2fw)
 
 # .H16 is our own modified form of Intel HEX, with 16-bit length for records.
-$(obj)/%.fw: $(obj)/%.H16 $(obj)/ihex2fw | $(objtree)/$(obj)/$$(dir %)
+$(obj)/%.fw: $(obj)/%.H16 | $(obj)/ihex2fw $(objtree)/$(obj)/$$(dir %)
 	$(call cmd,h16tofw)
 
 $(firmware-dirs):

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Linus Torvalds wrote:
> 
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>> Already started, in fact, since Linus said he would not reject it out of
>> hands.
> 
> Btw, I need to see it actually solve a real problem. As it is, I really 
> think you've just painted yourself into a corner where you can't just 
> admit that we can just load the firmware directly.

Kernel 2.6.26.  Older userland w/ initrd requirement, plus module w/ 
compiled-in firmware.

Simply dropping 2.6.27 into that situation will produce an unbootable 
system, since the firmware would not get copied into the initrd.

That's not just a kernel hacker situation.  Plenty of sites, even a few 
large Red Hat deployments, run newer kernels on a known-stable, older 
userland.  RH does not actively support swap-in-your-own-kernel, 
preferring our own tested kernel, but new-kernel/older-userland is not 
uncommon.

With upstream as it is today, there are just too many little ways the 
firmware can fail to follow the driver it once lived inside.  And the 
consequences of failure are delayed (until next boot), silent (until 
next boot), and notable (may produce non-booting system, certainly a 
non-working driver).

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Marcel Holtmann wrote:
> And as David mentioned multiple times, he left drivers/{net,scsi} out of
> it for now.

True, but we obviously must consider them -- and booting-related 
implications -- since that is the obvious direction everybody is headed 
towards.

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Tue, 2008-07-15 at 17:14 -0400, Jeff Garzik wrote:
> Linus Torvalds wrote:
> > 
> > On Tue, 15 Jul 2008, Jeff Garzik wrote:
> >> Already started, in fact, since Linus said he would not reject it out of
> >> hands.
> > 
> > Btw, I need to see it actually solve a real problem. As it is, I really 
> > think you've just painted yourself into a corner where you can't just 
> > admit that we can just load the firmware directly.
> 
> Kernel 2.6.26.  Older userland w/ initrd requirement, plus module w/ 
> compiled-in firmware.
> 
> Simply dropping 2.6.27 into that situation will produce an unbootable 
> system, since the firmware would not get copied into the initrd.

Kernel 2.6.15 with ancient userland. You update to 2.6.16, which
includes commit 5433383e ('convert qla2xxx to request_firmware()').

Same problem? Only three years ago. And not really a problem back then
either, because _even_ then you had to have a fairly ancient userland
for it not to work.

You're far too late to start whining about it now, Jeff. We're just
cleaning up the last drivers now. The rest have been where we want them
to be for _years_.

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Willy Tarreau wrote:
> On Tue, Jul 15, 2008 at 04:26:41PM -0400, Jeff Garzik wrote:
>> To be extremely concrete, firmware-in-module is
>>
>> 	* add Kconfig option (kernel-wide or per-driver, dunno) asking
>> 	  "build firmware into drivers, as before?"
>>
>> 	* tweak build process to build firmware into foo.ko output,
>> 	  probably in a specially marked ELF section
>>
>> 	* get request_firmware() to automatically notice that the
>> 	  MODULE_FIRMWARE() was built into this driver, and to
>> 	  look at the special ELF section for its data
> 
> Jeff, just thinking, wouldn't it be slightly easier to move the firmware
> in a separate module on its own and just add a dependency, so that foo.ko
> automatically loads foo-fw.ko ? I know it will be slightly differente, but
> would not change in-site deployment workflows nor installed scripts.

Quite true.  That's definitely an option, but I feel that building the 
firmware into the driver module itself would be about the same level of 
difficulty, but carry with it additional benefits:

Users and distros can be _certain_ their driver setup will not break due 
to these changes, if you have Kconfig options available to reproduce 
exactly what 2.6.26 produced [firmware compiled into the driver itself].

If Kconfig options are set such that the outputs are the same in 2.6.26 
and 2.6.27 -- vmlinuz and kernel modules -- then that should close 
windows of regression both real and theoretical, by producing _exactly_ 
the same outputs.

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Jeff Garzik wrote:
> 
> Kernel 2.6.26.  Older userland w/ initrd requirement, plus module w/
> compiled-in firmware.
> 
> Simply dropping 2.6.27 into that situation will produce an unbootable system,
> since the firmware would not get copied into the initrd.

Umm. Neither would the module.

So I assume you are doing the module load from the disk, not the initrd?

Or are you rebuilding the module into the initrd? If so, why didn't you do 
the firmware too and my "load directly from file" idea which is pretty 
simple? IOW, what's the fundamnetal problem here?

But hey, if you _do_ the "link it together", go ahead. I don't see why 
it's so magically superior, but I wanted it for the compiled-in driver 
case, so it's not fundamentally different from that either. And I can't 
really find the energy to care any more.

		Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[david]

On Tue, 15 Jul 2008, David Woodhouse wrote:

> On Tue, 2008-07-15 at 22:00 +0200, Marcel Holtmann wrote:
>> I really think we should use /lib/firmware/`uname -r`/. I do see the
>> point here that I don't wanna overwrite existing firmware from other
>> installed kernels. Especially if modules_install will install the
>> firmware files.
>>
>> So in case of non-Ubuntu distros we have to push a fix to udev, but that
>> is not a big deal. It should be a one-line change if I am not mistaken.
>
> My concern with that is that even though it's a one-line change, it's a
> one-line change which people don't already _have_. We really do need to
> remain compatible with existing setups.

when I read this I had to double check who the mail was from. this sounds 
like exactly the statement that Jeff has been making in his arguments to 
allow for the firmware-in-the-modules approach.

David Lang

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[david]

On Tue, 15 Jul 2008, Linus Torvalds wrote:

> On Tue, 15 Jul 2008, Marcel Holtmann wrote:
>>
>> you don't have to. We extend udev once and then it will always work.
>
> Umm. The thing is, people running new kernels with old user land is not
> just supposed to work, it's _really_ supposed to work.
>
> It's what I do. Something that breaks that has to have damn good reasons
> to break it.
>
> So I do not disagree with Jeff on that point _at_all_. I'm in violent
> agreement with Jeff on the fact that we should not require system updates
> for the kernel to do the right thing.
>
> The thing I disagree with Jeff on is that he then seems to turn that into
> something very negative ("let's not separate the firmware at all").

he's not saying not to seperate the firmware at all, he's asking for the 
_option_ of compiling a kernel that doesn't have the firmware seperate 
from the modules.

a kernel compiled with this option would just drop-in to an older distro 
with zero impact. newer distros that have updated their userspace tools 
could compile with different options and have the firmware seperate.

David Lang

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Linus Torvalds wrote:
> 
> On Tue, 15 Jul 2008, Jeff Garzik wrote:
>> Kernel 2.6.26.  Older userland w/ initrd requirement, plus module w/
>> compiled-in firmware.
>>
>> Simply dropping 2.6.27 into that situation will produce an unbootable system,
>> since the firmware would not get copied into the initrd.
> 
> Umm. Neither would the module.

Sure it would.

Normal kernel build with modules, older userland (with older initrd).

'make modules_install && make install' will
	* install modules in /lib
	* install firmware in /lib	[2.6.27 only]
	* install kernel
	* update grub
	* rebuild initrd

Do this, under both 2.6.26 and 2.6.27.

2.6.26 will put a working driver into initrd.

2.6.27 will put a dead driver into initrd (because no firmware got 
copied into initrd image).


> Or are you rebuilding the module into the initrd? If so, why didn't you do 
> the firmware too and my "load directly from file" idea which is pretty 
> simple? IOW, what's the fundamnetal problem here?

The firmware doesn't get copied into the initrd, because it is an 
automated process (older mkinitrd) that is unaware of the need to copy 
the firmware into the initrd.

Do the same build process on 2.6.26 and 2.6.27, and you WILL get 
different results on older userlands, for the same simple reason:  the 
scripts do not know to copy the firmware when they copy the kernel module.


> But hey, if you _do_ the "link it together", go ahead. I don't see why 

Great!  I will :)


> it's so magically superior, but I wanted it for the compiled-in driver 
> case, so it's not fundamentally different from that either.

It's not magically superior, just a necessary tool that helps avoid a 
whole class of regressions, keeps us working on older userland, etc., etc.

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

david@lang.hm wrote:
> he's not saying not to seperate the firmware at all, he's asking for the 
> _option_ of compiling a kernel that doesn't have the firmware seperate 
> from the modules.
> 
> a kernel compiled with this option would just drop-in to an older distro 
> with zero impact. newer distros that have updated their userspace tools 
> could compile with different options and have the firmware seperate.


Correct.

Thanks :)

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, david@lang.hm wrote:
> 
> a kernel compiled with this option would just drop-in to an older distro with
> zero impact. newer distros that have updated their userspace tools could
> compile with different options and have the firmware seperate.

The 'zero impact' is what doesn't make sense here.

You are supposed to be able to run ol distributions, yes.

But that doesn't mean that you can necessarily just plop things in the 
same way as you always did before.

For example, you have to rewrite your distro's initrd if you are using 
modules. You cannot just re-use the modules in the distro initrd. So doing 
a new kernel has _never_ been 'zero impact' in the sense that you could 
just switch vmlinux files around.

(Btw, I personally actually want my kernel to be _truly_ zero impact, but 
that also means that I don't use modules - because that way I really can 
avoid changing even the initrd image too. But that also already works)

Why is it suddenly so important that a kernel be 'zero impact' for that 
module case, when it's never been zero impact for that case before? You 
had to rewrite the initrd to begin with, but now you're not willing to do 
it again, just because you have to rewrite it slightly _differently_?

THAT is what I find so odd. The inability to accept just a slight change 
in kernel build.

But whatever. This really isn't worth it. The request_firmware() thing 
will clearly happen regardless, and as long as the backwards compat code 
is small and Jeff writes it, what do I care? Even if I think it looks 
largely pointless..

			Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Miller]

From: David Woodhouse <dwmw2@infradead.org>
Date: Tue, 15 Jul 2008 14:24:21 -0700

> Kernel 2.6.15 with ancient userland. You update to 2.6.16, which
> includes commit 5433383e ('convert qla2xxx to request_firmware()').
> 
> Same problem?

For the record I did get burnt by that one and I did make a huge stink
about it.

I used to be able to avoid initrd entirely on every single one of my
boxes, even my machines with qla2xxx based root filesystems, and that
changeset basically meant the end of that.

I simply don't do much hacking on that particular machine any more,
it's not worth the pain just to get a kernel that boots on it.

Yes, I know I can do stupid things like use the dist's initrd image
and just ignore the module version warnings, which is what Linus
happens to do, but that's just too gross for my tastes.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Tue, 2008-07-15 at 17:39 -0400, Jeff Garzik wrote:
> 
> 'make modules_install && make install' will
>         * install modules in /lib
>         * install firmware in /lib      [2.6.27 only]
>         * install kernel
>         * update grub
>         * rebuild initrd
> 
> Do this, under both 2.6.26 and 2.6.27.
> 
> 2.6.26 will put a working driver into initrd.
> 
> 2.6.27 will put a dead driver into initrd (because no firmware got 
> copied into initrd image).

I believe your claim is untrue. It uses the distribution's tool to
create the initrd, doesn't it? And those tools _do_ handle the
MODULE_FIRMWARE() tags and pull in appropriate firmware. They've had to
get that right for _years_ already.

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Adrian Bunk]

On Tue, Jul 15, 2008 at 09:44:45PM +0200, Marcel Holtmann wrote:
> Hi Linus,
> 
> > > Personally I am against hacking file loading from within the kernel.
> > > This is why we involved udev (or hotplug in the beginning). It allows
> > > you to have userspace policy for the search paths etc.
> > 
> > Well, I'm personally against _forcing_ people to use udev. 
> > 
> > Yes, desktop distros generally will do so, since there are lots of complex 
> > issues etc. But I don't think we've generally forced it on people if they 
> > don't want it (ie I haven't tried it personally since the distro I used 
> > started using udev, but I think you can still just ignore udev events and 
> > set everything up statically).
> > 
> > And I don't think that's wrong. Which implies that setup things should 
> > still generally at least allow us to avoid udev.
> > 
> > (But maybe I'm wrong, and everybody already uses udev just because they 
> > couldn't be bothered not to).
> 
> I actually think that is the case here. udev is so lightweight and
> really slim nowadays that not using it is a big drawback. There might be
> some system out there that are still using a static /dev directory, but
> for sure there are not many and even the embedded world doesn't really
> mind.
>...

Depends on which embedded systems you look at.

For MID devices that's definitely true, but there are also today still 
cases where customers want fast startup times on < 200 MHz embedded 
devices, and when every second counts you don't want to add a
"Waiting for /dev to be fully populated" from udev to the bootup.

(And on my computer (whose Linux installation predates the inclusion of 
 devfsd (sic) into the kernel) the static /dev also still works fine.)

> Regards
> 
> Marcel

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Frans Pop]

On Tuesday 15 July 2008, you wrote:
> Why is it suddenly so important that a kernel be 'zero impact' for that
> module case, when it's never been zero impact for that case before? You
> had to rewrite the initrd to begin with, but now you're not willing to
> do it again, just because you have to rewrite it slightly
> _differently_?

Because in most cases users do not write and maintain the scripts used to 
build custom kernels themselves. They use (huge and way too) tools like 
Debian's 'make-kpkg' for that.

Because users may want to build a custom 2.6.27 in for example a Debian 
Etch environment using the make-kpkg package that is available in that 
release of Debian. Or even a Debian Lenny environment (Lenny is about to 
be frozen so is very unlikely to include whatever may be needed for 
this).

Why is it good to extend the lifetime of the i386 and amd64 symlinks for 
x86 images by an extra few years so that older versions of kernel 
packaging tools - like make-kpkg - don't break, but is breaking the exact 
same tools with this firmware change suddenly OK?

Users _want_ to be able to build new kernels in older environments. Why 
take that option away from them? Why force them through the pain of 
partial upgrades of their userspace if it can so easily be avoided?

>From my PoV adding this option is a no-brainer: if Jeff is willing to do 
the work, then what harm can it do? The benefits to me are obvious.
I really don't get what you have against it.

Cheers,
FJP

P.S. Sorry to keep coming up with Debian examples, but that's the 
environment I know.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[david]

On Tue, 15 Jul 2008, Linus Torvalds wrote:

> On Tue, 15 Jul 2008, david@lang.hm wrote:
>>
>> a kernel compiled with this option would just drop-in to an older distro with
>> zero impact. newer distros that have updated their userspace tools could
>> compile with different options and have the firmware seperate.
>
> The 'zero impact' is what doesn't make sense here.
>
> You are supposed to be able to run ol distributions, yes.
>
> But that doesn't mean that you can necessarily just plop things in the
> same way as you always did before.
>
> For example, you have to rewrite your distro's initrd if you are using
> modules. You cannot just re-use the modules in the distro initrd. So doing
> a new kernel has _never_ been 'zero impact' in the sense that you could
> just switch vmlinux files around.

this time I will call 'strawman'. we've been required to put in the 
appropriate modules ever since modules were introduced. as Jeff gave 
examples of the userspace tools have been taking care of that for many 
years (and on distros where they don't take care of that you go with 
monolithic kernels), but they don't take care of firmware in many cases.

> (Btw, I personally actually want my kernel to be _truly_ zero impact, but
> that also means that I don't use modules - because that way I really can
> avoid changing even the initrd image too. But that also already works)

I do the same thing 99% of the time.

> Why is it suddenly so important that a kernel be 'zero impact' for that
> module case, when it's never been zero impact for that case before? You
> had to rewrite the initrd to begin with, but now you're not willing to do
> it again, just because you have to rewrite it slightly _differently_?

becouse the tools that wrote the initrd already put the modules in. I 
don't maintain those tools, they came with the distro. we're just asking 
to not require those tools to be updated immediatly.

David Lang

> THAT is what I find so odd. The inability to accept just a slight change
> in kernel build.
>
> But whatever. This really isn't worth it. The request_firmware() thing
> will clearly happen regardless, and as long as the backwards compat code
> is small and Jeff writes it, what do I care? Even if I think it looks
> largely pointless..
>
> 			Linus
>
>
>

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Tue, 2008-07-15 at 15:24 -0700, david@lang.hm wrote:
> 
> > Why is it suddenly so important that a kernel be 'zero impact' for that
> > module case, when it's never been zero impact for that case before? You
> > had to rewrite the initrd to begin with, but now you're not willing to do
> > it again, just because you have to rewrite it slightly _differently_?
> 
> becouse the tools that wrote the initrd already put the modules in. I 
> don't maintain those tools, they came with the distro. we're just asking 
> to not require those tools to be updated immediatly.

The tools that write the initrd already put the firmware in too.

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Tue, 2008-07-15 at 14:47 -0700, David Miller wrote:
> From: David Woodhouse <dwmw2@infradead.org>
> Date: Tue, 15 Jul 2008 14:24:21 -0700
> 
> > Kernel 2.6.15 with ancient userland. You update to 2.6.16, which
> > includes commit 5433383e ('convert qla2xxx to request_firmware()').
> > 
> > Same problem?
> 
> For the record I did get burnt by that one and I did make a huge stink
> about it.
> 
> I used to be able to avoid initrd entirely on every single one of my
> boxes, even my machines with qla2xxx based root filesystems, and that
> changeset basically meant the end of that.

Until now. The only significant _generic_ change that just went into
Linus' kernel is the fact that you can now build that firmware _into_
your kernel, and you can boot qla2xxx without an initrd again.

HTH.

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Miller]

From: David Woodhouse <dwmw2@infradead.org>
Date: Tue, 15 Jul 2008 15:42:32 -0700

> On Tue, 2008-07-15 at 14:47 -0700, David Miller wrote:
> > From: David Woodhouse <dwmw2@infradead.org>
> > Date: Tue, 15 Jul 2008 14:24:21 -0700
> > 
> > > Kernel 2.6.15 with ancient userland. You update to 2.6.16, which
> > > includes commit 5433383e ('convert qla2xxx to request_firmware()').
> > > 
> > > Same problem?
> > 
> > For the record I did get burnt by that one and I did make a huge stink
> > about it.
> > 
> > I used to be able to avoid initrd entirely on every single one of my
> > boxes, even my machines with qla2xxx based root filesystems, and that
> > changeset basically meant the end of that.
> 
> Until now. The only significant _generic_ change that just went into
> Linus' kernel is the fact that you can now build that firmware _into_
> your kernel, and you can boot qla2xxx without an initrd again.

That's great.

Even better would be if the qla2xxx firmware was the tree like it used
to be too.  Then it truly would be fully back to how things used to be.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

David Woodhouse wrote:
> On Tue, 2008-07-15 at 17:39 -0400, Jeff Garzik wrote:
>> 'make modules_install && make install' will
>>         * install modules in /lib
>>         * install firmware in /lib      [2.6.27 only]
>>         * install kernel
>>         * update grub
>>         * rebuild initrd
>>
>> Do this, under both 2.6.26 and 2.6.27.
>>
>> 2.6.26 will put a working driver into initrd.
>>
>> 2.6.27 will put a dead driver into initrd (because no firmware got 
>> copied into initrd image).
> 
> I believe your claim is untrue. It uses the distribution's tool to
> create the initrd, doesn't it? And those tools _do_ handle the
> MODULE_FIRMWARE() tags and pull in appropriate firmware.

slackware's CURRENT mkinitrd and kernel-modules scripts appear to 
disprove your claims.  And yes, they use a modern kernel, 2.6.24.


> They've had to
> get that right for _years_ already.

You should not just /assume/ that everything is ok, simply because time 
passed since request_firmware() was added.  Or let impatience blind you 
to the breakage that will result from your change.

Further, there are several factors that clearly differentiate a driver 
that has been using request_firmware() for a while, and newly converting 
a large block of drivers en masse.

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[david]

On Tue, 15 Jul 2008, David Woodhouse wrote:

> On Tue, 2008-07-15 at 15:24 -0700, david@lang.hm wrote:
>>
>>> Why is it suddenly so important that a kernel be 'zero impact' for that
>>> module case, when it's never been zero impact for that case before? You
>>> had to rewrite the initrd to begin with, but now you're not willing to do
>>> it again, just because you have to rewrite it slightly _differently_?
>>
>> becouse the tools that wrote the initrd already put the modules in. I
>> don't maintain those tools, they came with the distro. we're just asking
>> to not require those tools to be updated immediatly.
>
> The tools that write the initrd already put the firmware in too.

not always, as Jeff showed.

David Lang

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Dave Jones]

On Tue, Jul 15, 2008 at 04:55:45PM -0400, Jeff Garzik wrote:

 > No, I spelled out a list of projects, what would break, and how it would 
 > break.
 > 
 > Again,
 > 
 > * Red Hat driver disks.  The build process is unaware of the need to 
 > bundle firmware.  Any use with Fedora (or, later, RHEL) will result in a 
 > non-working driver, for the simple reason that firmware is not copied 
 > onto the image.

For old RHEL - they won't support 2.6.27, so moot.
For next-gen RHEL - they need fixing anyway.
For Fedora, they don't (and hopefully won't) exist.

	Dave

-- 
http://www.codemonkey.org.uk

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Dave Jones]

On Tue, Jul 15, 2008 at 12:57:46PM -0700, David Woodhouse wrote:
 > On Tue, 2008-07-15 at 12:49 -0700, Linus Torvalds wrote:
 > > So kernel-specific directories do make sense. As does the whole "I
 > > don't want to handle the pain that is udev scripts".
 > 
 > I'm not violently opposed to kernel-specific directories. Ubuntu does it
 > that way already, and it may well make sense. I'm not sure it's really
 > necessary either, but it's an option.

Some of the wireless drivers have an issue where firmware x works with
driver y, so it's useful for that case even if nothing else uses it.

	Dave

-- 
http://www.codemonkey.org.uk

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

David Woodhouse wrote:
> Until now. The only significant _generic_ change that just went into
> Linus' kernel is the fact that you can now build that firmware _into_
> your kernel, and you can boot qla2xxx without an initrd again.

Yes, that is quite nice...

It will be nice to be able to do the same with modules too :)

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Alexandre Oliva]

On Jul 14, 2008, Linus Torvalds <torvalds@linux-foundation.org> wrote:

> [...] I like just making everybody do request_firmware().

> Do you call that a legal reason? Or a technical one?

\o.

Hmm, let's se...

How about legal?

If they don't budge in, you can terminate their license to Linux,
because they are distributing your GPLed work combined with software
that is not under a GPL-compatible license.

:-P :-D

-- 
Alexandre Oliva         http://www.lsd.ic.unicamp.br/~oliva/
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}
FSFLA Board Member       ¡Sé Libre! => http://www.fsfla.org/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, david@lang.hm wrote:
>
> becouse the tools that wrote the initrd already put the modules in. I don't
> maintain those tools, they came with the distro. we're just asking to not
> require those tools to be updated immediatly.

But mkinitrd (which is the _only_ thing that people tend to use to write 
initrd's - is there even anything else) has already been doing this for 
years, as has been pointed out several times.

This is why I harped on the fact that you already rebuilt your initrd 
image.

So this really isn't a "updated immediately" issue, afaik. Googling for 

	mkinitrd MODULE_FIRMWARE

shows a patch from two years ago as the #1 hit in order to make the 
aic94xx driver work.

Loking down a bit, there's a hotplug discussion from early 2005 (gmane 
says "3 years, 20 weeks, 3 days, 6 hours and 39 minutes ago" in the 
header).

Quite frankly, if it's still a problem, there's simply something _wrong_ 
with the distribution. And yeah, maybe people need to update their kernel 
building tools.

I already pointed out how the kernel development team quite often says "we 
will no longer build with gcc-.xyz because it's too old and buggy". The 
build tools requirements are simply *different* from the runtime tools.

If it's literally just an issue of an mkinitrd that is too damn old, why 
don't we just make that test at kernel build time. *EXACTLY* the same way 
we test for compilers that are too old, and refuse to build with them?

(And no, I have no idea which version we should test for. In fact, 
mkinitrd seems to be singularly hard to test versions for, in that it 
seems to want the user to be root even just to give the version output. 
Ooh).

				Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Rafael J. Wysocki]

On Tuesday, 15 of July 2008, David Woodhouse wrote:
> On Tue, 2008-07-15 at 13:49 -0700, David Woodhouse wrote:
> > Sam, why do we have FORCE on the rules for $(hostprogs-y)? It forces the
> > ihex2fw tool to be rebuilt, which in turn forces all .HEX->.fw
> > conversions to happen again, which means that 'make modules_install'
> > wants to write to the source tree. That's a problem in some strange
> > situations like Rafael's, where he builds on another host and exports
> > the tree read-only by NFS.
> > 
> > Can we do this? ...
> 
> Alternatively, Rafael, does this do it for you?

No, it doesn't.

> --- a/firmware/Makefile
> +++ b/firmware/Makefile
> @@ -150,11 +150,11 @@ $(obj)/%: $(obj)/%.ihex | $(objtree)/$(obj)/$$(dir %)
>  # is actually meaningful, because the firmware has to be loaded in a certain
>  # order rather than as a single binary blob. Thus, we convert them into our
>  # more compact binary representation of ihex records (<linux/ihex.h>)
> -$(obj)/%.fw: $(obj)/%.HEX $(obj)/ihex2fw | $(objtree)/$(obj)/$$(dir %)
> +$(obj)/%.fw: $(obj)/%.HEX | $(obj)/ihex2fw $(objtree)/$(obj)/$$(dir %)
>  	$(call cmd,ihex2fw)
>  
>  # .H16 is our own modified form of Intel HEX, with 16-bit length for records.
> -$(obj)/%.fw: $(obj)/%.H16 $(obj)/ihex2fw | $(objtree)/$(obj)/$$(dir %)
> +$(obj)/%.fw: $(obj)/%.H16 | $(obj)/ihex2fw $(objtree)/$(obj)/$$(dir %)
>  	$(call cmd,h16tofw)
>  
>  $(firmware-dirs):
> 

Rafael

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Dave Jones wrote:
> On Tue, Jul 15, 2008 at 04:55:45PM -0400, Jeff Garzik wrote:
> 
>  > No, I spelled out a list of projects, what would break, and how it would 
>  > break.
>  > 
>  > Again,
>  > 
>  > * Red Hat driver disks.  The build process is unaware of the need to 
>  > bundle firmware.  Any use with Fedora (or, later, RHEL) will result in a 
>  > non-working driver, for the simple reason that firmware is not copied 
>  > onto the image.
> 
> For old RHEL - they won't support 2.6.27, so moot.
> For next-gen RHEL - they need fixing anyway.
> For Fedora, they don't (and hopefully won't) exist.

Grin,
http://docs.fedoraproject.org/fedora-install-guide-en/fc5/sn-bootoptions-hardware.html

Or DKMS, which should package firmware:
http://linux.dell.com/projects.shtml

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Dave Jones]

On Tue, Jul 15, 2008 at 07:46:33PM -0400, Jeff Garzik wrote:
 > Dave Jones wrote:
 > > On Tue, Jul 15, 2008 at 04:55:45PM -0400, Jeff Garzik wrote:
 > > 
 > >  > No, I spelled out a list of projects, what would break, and how it would 
 > >  > break.
 > >  > 
 > >  > Again,
 > >  > 
 > >  > * Red Hat driver disks.  The build process is unaware of the need to 
 > >  > bundle firmware.  Any use with Fedora (or, later, RHEL) will result in a 
 > >  > non-working driver, for the simple reason that firmware is not copied 
 > >  > onto the image.
 > > 
 > > For old RHEL - they won't support 2.6.27, so moot.
 > > For next-gen RHEL - they need fixing anyway.
 > > For Fedora, they don't (and hopefully won't) exist.
 > 
 > Grin,
 > http://docs.fedoraproject.org/fedora-install-guide-en/fc5/sn-bootoptions-hardware.html

I don't think any of this stuff has ever been actually used in a Fedora context.
We certainly haven't released any official ones ttbomk.

 > Or DKMS, which should package firmware:
 > http://linux.dell.com/projects.shtml

Not quite the same thing as a driver disk, but I'll humour you..
That handles MODULE_FIRMWARE fine afaik.  Though again, this doesn't see much
actual use in a Fedora context.

	Dave

-- 
http://www.codemonkey.org.uk

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Chris Adams]

Once upon a time, Jeff Garzik  <jeff@garzik.org> said:
>* Red Hat driver disks.  The build process is unaware of the need to 
>bundle firmware.  Any use with Fedora (or, later, RHEL) will result in a 
>non-working driver, for the simple reason that firmware is not copied 
>onto the image.

Then Red Hat driver disks are already broken, as there are current
modules (e.g. qla2xxx) that require external firmware.

>* Embedded projects like OpenWRT, that divide up kernel modules into 
>multiple packages.  Since it's not just one big package, you have to 
>update each package's list of files to include the newly split-out 
>firmwares.  Failure to do so results in a successful build set of 
>packages... of drivers that lack their firmware.

AFAIK OpenWRT and the like don't just throw in random new kernel
releases without at least reading release notes and testing.
-- 
Chris Adams <cmadams@hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Alexandre Oliva]

On Jul 15, 2008, David Woodhouse <dwmw2@infradead.org> wrote:

> On Tue, 2008-07-15 at 07:06 +0200, Willy Tarreau wrote:
>> This is just a matter of point of view. From my POV, having to feed
>> several unrelated files into a system to be able to get it to support
>> any device is moving backwards, compared to the currently clean
>> all-in-1 driver model.

> So you disapprove of having to load mac80211.ko before your wireless NIC
> driver? Or sungem_phy.ko before sungem.ko? Or libphy.ko before various
> other drivers?

Hey, how about a config option that turns firwmare files into
stand-alone kernel modules that register the blob into the
request_firmware internal list, and MODULE_FIRMWARE into a module
dependency?

-- 
Alexandre Oliva         http://www.lsd.ic.unicamp.br/~oliva/
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}
FSFLA Board Member       ¡Sé Libre! => http://www.fsfla.org/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Frans Pop]

On Wednesday 16 July 2008, Linus Torvalds wrote:
> On Tue, 15 Jul 2008, david@lang.hm wrote:
> > becouse the tools that wrote the initrd already put the modules in. I
> > don't maintain those tools, they came with the distro. we're just
> > asking to not require those tools to be updated immediatly.
>
> But mkinitrd (which is the _only_ thing that people tend to use to
> write initrd's - is there even anything else) has already been doing
> this for years, as has been pointed out several times.

Debian doe not have mkinitrd but initramfs-tools [1] and yaird [2]. The 
status of the last is uncertain (not currently included for Lenny).

Where did you get this strange notion that mkinitrd is the _only_ (your 
emphasis) thing people use?
Point is that I find it really strange that - in the extremely rich and 
varied world that is based on Linux - you seem so certain that every tool 
out there already does the right thing for all cases, or can be trivially 
updated to do so. I just don't believe that.

I even _know_ that yaird is broken wrt. firmware [3]. It's one of the 
reasons it's unlikely to be included in Lenny.

But hell, I'll see if I can give this a try tomorrow. Get out of this 
crazy make-belief discussion and provide some facts. Who knows, maybe 
I'll be surprised and the existing Debian toolsets will do the right 
thing. I seriously doubt it though.

[1] http://packages.debian.org/sid/initramfs-tools
[2] http://packages.debian.org/sid/yaird
[3] Only works if config files are edited manually which makes it useless
    during e.g. new installs.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Tue, 15 Jul 2008, Alexandre Oliva wrote:
> 
> Hey, how about a config option that turns firwmare files into
> stand-alone kernel modules that register the blob into the
> request_firmware internal list, and MODULE_FIRMWARE into a module
> dependency?

I think that should work fine from a technical angle, and I don't disagree 
per se, but it really is very ugly.

Wouldn't it be much simpler to just require people to have a reasonably 
up-to-date mkinitrd. Apparently it's not available _everywhere_, but if we 
check for this up-front (even if we can apparently only do so at "make 
install" time due to the root requirements), wouldn't that really solve 
the issue with much less work?

			Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Linus Torvalds]

On Wed, 16 Jul 2008, Frans Pop wrote:
> 
> Where did you get this strange notion that mkinitrd is the _only_ (your 
> emphasis) thing people use?

Heh. Maybe from the fact that I don't personally use modules or initrd at 
all, and have no interest in doing so ;)

Ok, so people use other things. Grr. I guess we can't just require updated 
tools, because we don't know what the tools are.

			Linus

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Linus Torvalds wrote:
> 
> On Wed, 16 Jul 2008, Frans Pop wrote:
>> Where did you get this strange notion that mkinitrd is the _only_ (your 
>> emphasis) thing people use?
> 
> Heh. Maybe from the fact that I don't personally use modules or initrd at 
> all, and have no interest in doing so ;)
> 
> Ok, so people use other things. Grr. I guess we can't just require updated 
> tools, because we don't know what the tools are.

Yes, that's part of the problem :/

Everybody seems to love rolling their own stuff in this area, partly 
necessitated I suppose by the wide variety of ways to boot on various 
platforms.

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Theodore Tso]

On Tue, Jul 15, 2008 at 04:42:52PM -0700, Linus Torvalds wrote:
> On Tue, 15 Jul 2008, david@lang.hm wrote:
> >
> > becouse the tools that wrote the initrd already put the modules in. I don't
> > maintain those tools, they came with the distro. we're just asking to not
> > require those tools to be updated immediatly.
> 
> But mkinitrd (which is the _only_ thing that people tend to use to write 
> initrd's - is there even anything else) has already been doing this for 
> years, as has been pointed out several times.

Actually, there is also mkinitramfs (which is what Debian and Ubuntu
use), and Yaird (Yet Another mkinitrd).  Also, the distro's tend to
heavily customize mkinitrd in general, so there is no guarantee that
just because two distro's use some tool that happens to be named
mkinitrd that the two tools have much in common with each other....

This is highly unfortunate, and I've often argued that we probably
would have been better off if we had included an mkinitrd tool as part
of the kernel sources to make life easier instead of having every
single distro do their own thing; it would also make it easier to run
a newer kernel on an older enterprise distro --- I remember the pain
trying to make a 2.6.16 kernel boot on a RHEL4 base system two years
ago; I don't want to even think about what would happen with something
more modern at this point.

> If it's literally just an issue of an mkinitrd that is too damn old, why 
> don't we just make that test at kernel build time. *EXACTLY* the same way 
> we test for compilers that are too old, and refuse to build with them?

For one reason, because there's more than one mkinitrd.  FC9 ships
with mkinitrd 6.0.52; OpenSuSE ships with mkinitrd 2.1, and the
sources don't look even vaguely similar to one another.  For example,
Red Hat's mkinitrd uses nash, which is a maddingly useless shell that
made it near-impossible to debug why a 2.6.16 kernel wasn't booting on
a RHEL-4 system on a LS-21 blade.  OpenSuSE's mkinitrd ships with
bash, and Debian/Ubuntu will give you a busybox shell, which makes
life *much* easier to debug when you're trying to figure out why your
latest bleeding edge kernel isn't able to mount the root partition out
of initrd.

						- Ted

P.S.  For bonus points, it would be nice if initrd's included fsck,
since it then it would be possible to safely fsck the root partition
and not require a reboot if the root partition was modified.  But
given that every single distro seems to ship their own initrd, not to
mention their own init scripts, it's hard to try to push for this sort
of change.  The amount of distro-specific engineering work probably
makes it not worth it.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Sven-Haegar Koch]

On Tue, 15 Jul 2008, David Woodhouse wrote:

> On Tue, 2008-07-15 at 17:39 -0400, Jeff Garzik wrote:
> > 
> > 'make modules_install && make install' will
> >         * install modules in /lib
> >         * install firmware in /lib      [2.6.27 only]
> >         * install kernel
> >         * update grub
> >         * rebuild initrd
> > 
> > Do this, under both 2.6.26 and 2.6.27.
> > 
> > 2.6.26 will put a working driver into initrd.
> > 
> > 2.6.27 will put a dead driver into initrd (because no firmware got 
> > copied into initrd image).
> 
> I believe your claim is untrue. It uses the distribution's tool to
> create the initrd, doesn't it? And those tools _do_ handle the
> MODULE_FIRMWARE() tags and pull in appropriate firmware. They've had to
> get that right for _years_ already.

Something from the non-development world:

At least the initramfs-tools (which contains mkinitrd) of Debian Etch do 
not do this (the current testing/development version Lenny does) - no 
firmware support in the initrd at all - and not needed for booting until 
now.
(It has /lib/firmware support after the initrd for the not-so-important
stuff like usb/sound/video/wireless)

Yes, it may be nearly ancient by now, but for servers it contains all we 
need, and most certainly would not want to use something that changes and 
needs checking/hand-fixing every month - I expect this version to be still 
in use a year from now, then just gradually being updated to the next 
debian version.

Currently we use 2.6.24, and I expect to use some newer kernel to support 
newer replacement hardware before all installs are upgraded. So having the 
firmware-inside-module support would be a help for us when modules are 
starting to be needed for disk or network access, keeping from the need to 
try backporting huge initrd/make-kpkg changes.

c'ya
sven

-- 

The Internet treats censorship as a routing problem, and routes around
it. (John Gilmore on http://www.cygnus.com/~gnu/)

Prosaic firmware issues

[Alexey Dobriyan]

Could somebody from firmware people fix boring stuff, please:

1) after allyesconfig build, git-status show plenty of crap:

#	firmware/atmsar11.fw
#	firmware/atmsar11.fw.gen.S
#	firmware/cpia2/stv0672_vp4.bin.gen.S
#	firmware/dabusb/bitstream.bin.gen.S
#	firmware/dabusb/firmware.fw
#	firmware/dabusb/firmware.fw.gen.S
#	firmware/edgeport/boot.fw
#	firmware/edgeport/boot.fw.gen.S
#	firmware/edgeport/boot2.fw
#	firmware/edgeport/boot2.fw.gen.S
#	firmware/edgeport/down.fw
#	firmware/edgeport/down.fw.gen.S
#	firmware/edgeport/down2.fw
#	firmware/edgeport/down2.fw.gen.S
#	firmware/edgeport/down3.bin.gen.S
#	firmware/emi26/bitstream.fw
#	firmware/emi26/bitstream.fw.gen.S
#	firmware/emi26/firmware.fw
#	firmware/emi26/firmware.fw.gen.S
#	firmware/emi26/loader.fw
#	firmware/emi26/loader.fw.gen.S
#	firmware/emi62/bitstream.fw
#	firmware/emi62/bitstream.fw.gen.S
#	firmware/emi62/loader.fw
#	firmware/emi62/loader.fw.gen.S
#	firmware/emi62/midi.fw
#	firmware/emi62/midi.fw.gen.S
#	firmware/emi62/spdif.fw
#	firmware/emi62/spdif.fw.gen.S
#	firmware/ess/maestro3_assp_kernel.fw
#	firmware/ess/maestro3_assp_kernel.fw.gen.S
#	firmware/ess/maestro3_assp_minisrc.fw
#	firmware/ess/maestro3_assp_minisrc.fw.gen.S
#	firmware/ihex2fw
#	firmware/intelliport2.bin.gen.S
#	firmware/kaweth/new_code.bin.gen.S
#	firmware/kaweth/new_code_fix.bin.gen.S
#	firmware/kaweth/trigger_code.bin.gen.S
#	firmware/kaweth/trigger_code_fix.bin.gen.S
#	firmware/keyspan/mpr.fw
#	firmware/keyspan/mpr.fw.gen.S
#	firmware/keyspan/usa18x.fw
#	firmware/keyspan/usa18x.fw.gen.S
#	firmware/keyspan/usa19.fw
#	firmware/keyspan/usa19.fw.gen.S
#	firmware/keyspan/usa19qi.fw
#	firmware/keyspan/usa19qi.fw.gen.S
#	firmware/keyspan/usa19qw.fw
#	firmware/keyspan/usa19qw.fw.gen.S
#	firmware/keyspan/usa19w.fw
#	firmware/keyspan/usa19w.fw.gen.S
#	firmware/keyspan/usa28.fw
#	firmware/keyspan/usa28.fw.gen.S
#	firmware/keyspan/usa28x.fw
#	firmware/keyspan/usa28x.fw.gen.S
#	firmware/keyspan/usa28xa.fw
#	firmware/keyspan/usa28xa.fw.gen.S
#	firmware/keyspan/usa28xb.fw
#	firmware/keyspan/usa28xb.fw.gen.S
#	firmware/keyspan/usa49w.fw
#	firmware/keyspan/usa49w.fw.gen.S
#	firmware/keyspan/usa49wlc.fw
#	firmware/keyspan/usa49wlc.fw.gen.S
#	firmware/keyspan_pda/keyspan_pda.fw
#	firmware/keyspan_pda/keyspan_pda.fw.gen.S
#	firmware/keyspan_pda/xircom_pgs.fw
#	firmware/keyspan_pda/xircom_pgs.fw.gen.S
#	firmware/korg/k1212.dsp
#	firmware/korg/k1212.dsp.gen.S
#	firmware/ti_3410.fw
#	firmware/ti_3410.fw.gen.S
#	firmware/ti_5052.fw
#	firmware/ti_5052.fw.gen.S
#	firmware/ttusb-budget/dspbootcode.bin.gen.S
#	firmware/vicam/firmware.fw
#	firmware/vicam/firmware.fw.gen.S
#	firmware/whiteheat.fw
#	firmware/whiteheat.fw.gen.S
#	firmware/whiteheat_loader.fw
#	firmware/whiteheat_loader.fw.gen.S
#	firmware/yamaha/ds1_ctrl.fw
#	firmware/yamaha/ds1_ctrl.fw.gen.S
#	firmware/yamaha/ds1_dsp.fw
#	firmware/yamaha/ds1_dsp.fw.gen.S
#	firmware/yamaha/ds1e_ctrl.fw
#	firmware/yamaha/ds1e_ctrl.fw.gen.S

2) after "make mrproper" after allyesconfig, some files aren't removed,
   use "git-ls-files -o" to see them.

#	firmware/atmsar11.fw
#	firmware/dabusb/firmware.fw
#	firmware/emi26/bitstream.fw
#	firmware/emi26/firmware.fw
#	firmware/emi26/loader.fw
#	firmware/emi62/bitstream.fw
#	firmware/emi62/loader.fw
#	firmware/emi62/midi.fw
#	firmware/emi62/spdif.fw
#	firmware/ess/maestro3_assp_kernel.fw
#	firmware/ess/maestro3_assp_minisrc.fw
#	firmware/korg/k1212.dsp
#	firmware/yamaha/ds1_ctrl.fw
#	firmware/yamaha/ds1_dsp.fw
#	firmware/yamaha/ds1e_ctrl.fw

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Dave Jones]

On Tue, Jul 15, 2008 at 08:51:33PM -0400, Theodore Tso wrote:
 > For one reason, because there's more than one mkinitrd.  FC9 ships
 > with mkinitrd 6.0.52; OpenSuSE ships with mkinitrd 2.1, and the
 > sources don't look even vaguely similar to one another.

Right. Other than name, they've historically shared nothing.
Both have grown through the evolution of multiple distros, requiring
different workarounds in each due to differences in CONFIG_ options
in the kernel between vendors for eg.

Whilst it would be great for unified development on the tools that
create the early boot process, I think it's a non-starter due to the
fact that you can't really do it without throwing out everything you
already have today.  The same reason imo, that hpa's klibc work hasn't
gained mass-appeal from vendors.

Even if we had a 'reference' mkinitrd in the kernel, it would be pretty much
useless until it reached feature parity with every distros existing tools,
and if everyone uses those instead of furthering the reference implementation,
it fails on the starting blocks.

	Dave

-- 
http://www.codemonkey.org.uk

Re: Prosaic firmware issues

[Linus Torvalds]

On Wed, 16 Jul 2008, Alexey Dobriyan wrote:
>
> Could somebody from firmware people fix boring stuff, please:
> 
> 1) after allyesconfig build, git-status show plenty of crap:

Something like the appended, perhaps?

(I'm not going to commit it, I want some firmware person to double-check 
it).

> 2) after "make mrproper" after allyesconfig, some files aren't removed,
>    use "git-ls-files -o" to see them.

Ok, that needs some Makefile magic. git magic I can do.

		Linus

---
 firmware/.gitignore |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
 create mode 100644 firmware/.gitignore

diff --git a/firmware/.gitignore b/firmware/.gitignore
new file mode 100644
index 0000000..9ade43a
--- /dev/null
+++ b/firmware/.gitignore
@@ -0,0 +1,2 @@
+*.gen.S
+*.fw

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Theodore Tso]

On Tue, Jul 15, 2008 at 09:36:19PM -0400, Dave Jones wrote:
> Whilst it would be great for unified development on the tools that
> create the early boot process, I think it's a non-starter due to the
> fact that you can't really do it without throwing out everything you
> already have today.  The same reason imo, that hpa's klibc work hasn't
> gained mass-appeal from vendors.

Yeah, it would have been much better if we had included a reference
mkinitrd from the beginning, instead of saying, "Kernel doesn't set
policy", and hoping the distro's would converge on something similar.
The reality though is to the extent that we want to make changes like
request_firmware() we are actually better off setting policy for stuff
which is this low-level.

> Even if we had a 'reference' mkinitrd in the kernel, it would be
> pretty much useless until it reached feature parity with every
> distros existing tools, and if everyone uses those instead of
> furthering the reference implementation, it fails on the starting
> blocks.

In order to work now it would need to have a relatively simple way for
distro's to hook in their own enhancements, and someone would have to
do most of the work of migrating at least the major distro's to use
the new reference "official" mkinitrd.  It's something we should have
done from the very beginning, and it's a lot harder to fix up that
mistake years later.

							- Ted

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Dave Jones wrote:
> Whilst it would be great for unified development on the tools that
> create the early boot process, I think it's a non-starter due to the
> fact that you can't really do it without throwing out everything you
> already have today.  The same reason imo, that hpa's klibc work hasn't
> gained mass-appeal from vendors.

Quite true and IMO disappointing, because klibc would make great 
infrastructure for rolling out things just like this.

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Willy Tarreau]

On Tue, Jul 15, 2008 at 08:43:07PM -0400, Jeff Garzik wrote:
> Linus Torvalds wrote:
> >
> >On Wed, 16 Jul 2008, Frans Pop wrote:
> >>Where did you get this strange notion that mkinitrd is the _only_ (your 
> >>emphasis) thing people use?
> >
> >Heh. Maybe from the fact that I don't personally use modules or initrd at 
> >all, and have no interest in doing so ;)
> >
> >Ok, so people use other things. Grr. I guess we can't just require updated 
> >tools, because we don't know what the tools are.
> 
> Yes, that's part of the problem :/
> 
> Everybody seems to love rolling their own stuff in this area, partly 
> necessitated I suppose by the wide variety of ways to boot on various 
> platforms.

Exactly. To give you an idea, I have a lot of servers running on a
"firmware" which consists in two parts :
  - a bzImage containing an initramfs with modules and a few scripts
  - an initrd which is in fact the rootfs

When the kernel boots, it mounts the initrd, does a pivot_root and mounts
its modules into /boot/$(uname -r). All my kernels run modules from /boot
and not from /lib/modules, because it makes them more convenient to add
and remove. So /lib/modules is just a symlink to /boot.

The above process is very convenient, as it is compatible with a lot of
boot methods : hard disk, CDROM, USB stick, PXE, etc...

And moreover, I can have multiple kernels with only one rootfs (SMP, etc).
Introducing firmware there would mean a major thinking and rework of the
build (and possibly boot) process. The least invasive would probably be
to stuff them next to the modules in $(uname -r)/firmware, but even then,
it will take a lot of time to switch to a new system.

And judging from what I see around, I'm far from being the only one not
using mkinitrd.

Willy

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Rene Herman]

On 15-07-08 21:14, Jeff Garzik wrote:

> Theodore Tso wrote:

>> If so, may I suggest you start implementing, instead of sending
>> e-mails?  For all the time people have spent arguing about it, 
>> maybe it could have been implemented already.
> 
> Already started, in fact, since Linus said he would not reject it out
> of hands.

Great, as I really hope this gets in. Was looking but am very scared of 
the linker goo...

I'm going to like the feature in itself once it has grown the ability 
for firmware in the module and really do not right now. One of the 
drivers I care about (old ISA SB16 CSP code) was also affected and my 
old 486 system with the hardware in it is _still_ lauging its little 
balls of at the people stating that "if you have userspace to load the 
module you have userspace to support at the hotplug/udev goo to load the 
firmware".

Given that one needs to be rather irrelevant to care about the SB16 CSP 
in the first place it wasn't enough to complain really, but yes, I'm 
really quite sure that I'm going to use the feature of building it into 
the module more than once while running around with floppies trying to 
support that old stuff.

Rene.

Re: Prosaic firmware issues

[David Woodhouse]

On Tue, 2008-07-15 at 18:45 -0700, Linus Torvalds wrote:
> 
> On Wed, 16 Jul 2008, Alexey Dobriyan wrote:
> >
> > Could somebody from firmware people fix boring stuff, please:
> > 
> > 1) after allyesconfig build, git-status show plenty of crap:
> 
> Something like the appended, perhaps?
> 
> (I'm not going to commit it, I want some firmware person to double-check 
> it).

Generally sane, but wants a few more things added.

> > 2) after "make mrproper" after allyesconfig, some files aren't removed,
> >    use "git-ls-files -o" to see them.
> 
> Ok, that needs some Makefile magic. git magic I can do.

Ah, that's caused by a stray := where there should be a +=, so the
firmware files listed before that get dropped from the list of files to
be cleaned. 

Alexey, thanks for the report. Please could you verify that this fixes
it for you?

----
Subject: Fix 'make clean' and .gitignore for firmware/ directory.

Provide a .gitignore file, and fix a typo which prevented some of the
generated binary files from being removed on 'make clean'.

Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>

diff --git a/firmware/.gitignore b/firmware/.gitignore
new file mode 100644
index 0000000..d9c6901
--- /dev/null
+++ b/firmware/.gitignore
@@ -0,0 +1,6 @@
+*.gen.S
+*.fw
+*.bin
+*.csp
+*.dsp
+ihex2fw
diff --git a/firmware/Makefile b/firmware/Makefile
index 981bd52..55ca321 100644
--- a/firmware/Makefile
+++ b/firmware/Makefile
@@ -73,7 +73,7 @@ fw-shipped-$(CONFIG_USB_SERIAL_KEYSPAN_USA28X) += keyspan/usa28x.fw
 fw-shipped-$(CONFIG_USB_SERIAL_KEYSPAN_USA49W) += keyspan/usa49w.fw
 fw-shipped-$(CONFIG_USB_SERIAL_KEYSPAN_USA49WLC) += keyspan/usa49wlc.fw
 else
-fw-shipped- := keyspan/mpr.fw keyspan/usa18x.fw keyspan/usa19.fw	\
+fw-shipped- += keyspan/mpr.fw keyspan/usa18x.fw keyspan/usa19.fw	\
 	keyspan/usa19qi.fw keyspan/usa19qw.fw keyspan/usa19w.fw		\
 	keyspan/usa28.fw keyspan/usa28xa.fw keyspan/usa28xb.fw		\
 	keyspan/usa28x.fw keyspan/usa49w.fw keyspan/usa49wlc.fw


-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Chris Adams wrote:
> Once upon a time, Jeff Garzik  <jeff@garzik.org> said:
>> * Red Hat driver disks.  The build process is unaware of the need to 
>> bundle firmware.  Any use with Fedora (or, later, RHEL) will result in a 
>> non-working driver, for the simple reason that firmware is not copied 
>> onto the image.
> 
> Then Red Hat driver disks are already broken, as there are current
> modules (e.g. qla2xxx) that require external firmware.

That doesn't change the problem any.  There are still users of this 
thing, regardless of what appellations are attached.


>> * Embedded projects like OpenWRT, that divide up kernel modules into 
>> multiple packages.  Since it's not just one big package, you have to 
>> update each package's list of files to include the newly split-out 
>> firmwares.  Failure to do so results in a successful build set of 
>> packages... of drivers that lack their firmware.
> 
> AFAIK OpenWRT and the like don't just throw in random new kernel
> releases without at least reading release notes and testing.

Sure.  The difference here is that we usually take care to make sure 
there is some way for the distributor to continue configuring their 
system in the manner known to work, while developing and testing new stuff.

But in this case, that option was not available to distributors (but 
hopefully it will be, in another day or two).

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Tue, 2008-07-15 at 13:49 -0700, David Woodhouse wrote:
> I need to hassle Sam about that.

I think I have it. Rafael, does _this_ fix it for you?

diff --git a/scripts/Makefile.fwinst b/scripts/Makefile.fwinst
index 3d2f460..994060d 100644
--- a/scripts/Makefile.fwinst
+++ b/scripts/Makefile.fwinst
@@ -43,3 +43,15 @@ __fw_install: $(installed-fw)
 __fw_modinst: $(mod-fw)
 
 FORCE:
+
+# Read all saved command lines and dependencies for the $(targets) we
+# may be building using $(if_changed{,_dep}). As an optimization, we 
+# don't need to read them if the target does not exist; we will rebuild
+# anyway in that case.
+
+targets := $(wildcard $(sort $(targets)))
+cmd_files := $(wildcard $(foreach f,$(targets),$(dir $(f)).$(notdir $(f)).cmd))
+
+ifneq ($(cmd_files),)
+  include $(cmd_files)
+endif

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Gabor Gombas]

On Wed, Jul 16, 2008 at 02:55:32AM +0200, Sven-Haegar Koch wrote:

> Something from the non-development world:
> 
> At least the initramfs-tools (which contains mkinitrd) of Debian Etch do 
> not do this (the current testing/development version Lenny does) - no 
> firmware support in the initrd at all - and not needed for booting until 
> now.
> (It has /lib/firmware support after the initrd for the not-so-important
> stuff like usb/sound/video/wireless)
> 
> Yes, it may be nearly ancient by now, but for servers it contains all we 
> need, and most certainly would not want to use something that changes and 
> needs checking/hand-fixing every month - I expect this version to be still 
> in use a year from now, then just gradually being updated to the next 
> debian version.
> 
> Currently we use 2.6.24, and I expect to use some newer kernel to support 
> newer replacement hardware before all installs are upgraded. So having the 
> firmware-inside-module support would be a help for us when modules are 
> starting to be needed for disk or network access, keeping from the need to 
> try backporting huge initrd/make-kpkg changes.

You can easyly install initramfs-tools from lenny on an etch box. It
only depends on a newer klibc, so it won't disturb any other services.

Btw., the Debian-provided 2.6.25 kernel already lacks the firmware for
some popular modules (like bnx2). So if you're using Debian, you as a
sysadmin already have to handle the "firmware is in a separate package"
issue.

Gabor

-- 
     ---------------------------------------------------------
     MTA SZTAKI Computer and Automation Research Institute
                Hungarian Academy of Sciences
     ---------------------------------------------------------

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Peter Zijlstra]

On Tue, 2008-07-15 at 23:07 -0400, Jeff Garzik wrote:
> Dave Jones wrote:
> > Whilst it would be great for unified development on the tools that
> > create the early boot process, I think it's a non-starter due to the
> > fact that you can't really do it without throwing out everything you
> > already have today.  The same reason imo, that hpa's klibc work hasn't
> > gained mass-appeal from vendors.
> 
> Quite true and IMO disappointing, because klibc would make great 
> infrastructure for rolling out things just like this.

Indeed, for one i've been told it would enable cross compiling of
initrds, without which I deem initrd useless junk - whoever still
compiles their kernel on the machine it will run on? :-)

Re: Prosaic firmware issues

[Alexey Dobriyan]

On Tue, Jul 15, 2008 at 10:54:04PM -0700, David Woodhouse wrote:
> On Tue, 2008-07-15 at 18:45 -0700, Linus Torvalds wrote:
> > 
> > On Wed, 16 Jul 2008, Alexey Dobriyan wrote:
> > >
> > > Could somebody from firmware people fix boring stuff, please:
> > > 
> > > 1) after allyesconfig build, git-status show plenty of crap:
> > 
> > Something like the appended, perhaps?
> > 
> > (I'm not going to commit it, I want some firmware person to double-check 
> > it).
> 
> Generally sane, but wants a few more things added.
> 
> > > 2) after "make mrproper" after allyesconfig, some files aren't removed,
> > >    use "git-ls-files -o" to see them.
> > 
> > Ok, that needs some Makefile magic. git magic I can do.
> 
> Ah, that's caused by a stray := where there should be a +=, so the
> firmware files listed before that get dropped from the list of files to
> be cleaned. 
> 
> Alexey, thanks for the report. Please could you verify that this fixes
> it for you?

It does, thank you.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Rafael J. Wysocki]

On Wednesday, 16 of July 2008, David Woodhouse wrote:
> On Tue, 2008-07-15 at 13:49 -0700, David Woodhouse wrote:
> > I need to hassle Sam about that.
> 
> I think I have it. Rafael, does _this_ fix it for you?

No, but the error message seems to be different:

  HOSTCC  firmware/ihex2fw
/home/rafael/src/linux-next/firmware/ihex2fw.c:268: fatal error: opening dependency file firmware/.ihex2fw.d: Read-only file system
compilation terminated.

Removing the FORCE from line 117 of scripts/Makefile.host makes it work.

Of course, I still have to run "make ... firmware_install" locally on the
server.

> diff --git a/scripts/Makefile.fwinst b/scripts/Makefile.fwinst
> index 3d2f460..994060d 100644
> --- a/scripts/Makefile.fwinst
> +++ b/scripts/Makefile.fwinst
> @@ -43,3 +43,15 @@ __fw_install: $(installed-fw)
>  __fw_modinst: $(mod-fw)
>  
>  FORCE:
> +
> +# Read all saved command lines and dependencies for the $(targets) we
> +# may be building using $(if_changed{,_dep}). As an optimization, we 
> +# don't need to read them if the target does not exist; we will rebuild
> +# anyway in that case.
> +
> +targets := $(wildcard $(sort $(targets)))
> +cmd_files := $(wildcard $(foreach f,$(targets),$(dir $(f)).$(notdir $(f)).cmd))
> +
> +ifneq ($(cmd_files),)
> +  include $(cmd_files)
> +endif
> 

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Diego Calleja]

El Tue, 15 Jul 2008 16:26:41 -0400, Jeff Garzik <jeff@garzik.org> escribió:

> And yes, it enables "weirdos" like me to continue building the firmware 
> into the driver, because it has a proven track record of high 
> reliability and simplicity.

I'm one of those "weirdos", and I'd like to explain from my user-POV (a user
who usually compiles and tests -rcs) why I prefer firmware-in-module I don't
find it "weird".

I use modules as much as I can. Apparently many people here uses non-modular
.configs. Me, I learnt lot of time ago that we live in a hotplugged world
were you can't predict what hardware you are going to need to plug, and you
don't want to tell your girlfriend that you need to compile a "kernel module"
because she brought a USB memory stick and you didn't compiled the support
when you compiled your kernel. And I don't like doing "lspci" every time I
want to know what network card I have in a box. So I compile everything as
a module (except the support for all sata/ide controllers and ext3, to
avoid the nightmares of initrds) and I let udev do all the job. I only use
external firmware when it's strictly needed for legal reasons.

Yes, just like I can do "make modules_install", I can do "make
firmware_install". It's not hard - just an extra command. But it's an
extra one. And I'm lazy to the point that I'm _not_ willing to type
that extra command. You could install the firmware automatically with
"make modules_install". Still, I'd need to cleanup frequently
/lib/firmware/<version> because of all the crap (and MB) that old
-rcs and -next's versions will leave there.

Those are _extra_ steps. Steps that force me to learn things that
I didn't need to know before. I didn't even know if my network
card had firmware or not. I never cared, and I don't want to start
caring about it now.

With firmware-in-module, I can switch a simple config option that will
get me back to my ignorance of firmware related things. And Jeff will
be my hero of the month for bringing back the simplicity of the old
times, while the rest of the people that want to force me to learn about
firmware (iow, the people who thinks that firmware-in-module is worthless)
remember me of the microkernel people, who could write books and give
speeches in universities about how insanely great is to have fulfilled
one of the most important problems in the story of CS - separating
firmware from the driver code. Despite of the fact that it does not
bring ANYTHING new for me, excepting added complexity.

I'll continue using external firmware only when for legal reasons it
can't be redistributed, of course, but fortunately for me that's an
exception on my hardware devices, not the rule.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Wed, 2008-07-16 at 23:28 +0200, Rafael J. Wysocki wrote:
> On Wednesday, 16 of July 2008, David Woodhouse wrote:
> > On Tue, 2008-07-15 at 13:49 -0700, David Woodhouse wrote:
> > > I need to hassle Sam about that.
> > 
> > I think I have it. Rafael, does _this_ fix it for you?
> 
> No, but the error message seems to be different:
> 
>   HOSTCC  firmware/ihex2fw
> /home/rafael/src/linux-next/firmware/ihex2fw.c:268: fatal error: opening dependency file firmware/.ihex2fw.d: Read-only file system
> compilation terminated.

Hm, can you do that again with V=2?

> Removing the FORCE from line 117 of scripts/Makefile.host makes it work.
> 
> Of course, I still have to run "make ... firmware_install" locally on the
> server.

Yeah, that part is easy enough to fix, once the unwanted rebuilding is
fixed.

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Rafael J. Wysocki]

On Thursday, 17 of July 2008, David Woodhouse wrote:
> On Wed, 2008-07-16 at 23:28 +0200, Rafael J. Wysocki wrote:
> > On Wednesday, 16 of July 2008, David Woodhouse wrote:
> > > On Tue, 2008-07-15 at 13:49 -0700, David Woodhouse wrote:
> > > > I need to hassle Sam about that.
> > > 
> > > I think I have it. Rafael, does _this_ fix it for you?
> > 
> > No, but the error message seems to be different:
> > 
> >   HOSTCC  firmware/ihex2fw
> > /home/rafael/src/linux-next/firmware/ihex2fw.c:268: fatal error: opening dependency file firmware/.ihex2fw.d: Read-only file system
> > compilation terminated.
> 
> Hm, can you do that again with V=2?

  HOSTCC  firmware/ihex2fw - due to: FORCE
/home/rafael/src/linux-next/firmware/ihex2fw.c:268: fatal error: opening dependency file firmware/.ihex2fw.d: Read-only file system
compilation terminated.

Thanks,
Rafael

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Thu, 2008-07-17 at 22:42 +0200, Rafael J. Wysocki wrote:
> 
>   HOSTCC  firmware/ihex2fw - due to: FORCE
> /home/rafael/src/linux-next/firmware/ihex2fw.c:268: fatal error:
> opening dependency file firmware/.ihex2fw.d: Read-only file system
> compilation terminated.

I'm very confused. I can do this...

 make INSTALL_FW_PATH=/tmp/foo firmware_install
 su otheruser
 touch firmware/.ihex2fw.cmd  (gives EPERM as expected)
 make INSTALL_FW_PATH=/tmp/foo2 firmware_install

And it _works_ for me, after I've made that change in
scripts/Makefile.fwinst. But I don't have .ihex2fw.d -- I
have .ihex2fw.cmd. What is in your .ihex2fw.d? And does it work if you
include _that_ from firmware/Makefile?

/me wonders where Sam is hiding...

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Miller]

From: Alan Cox <alan@lxorguk.ukuu.org.uk>
Date: Tue, 15 Jul 2008 13:42:46 +0100

> > * the legal subtext of these changes was not mentioned at all
> 
> The only legal subtext has been some mad fantasy proposed by Dave Miller
> that David Woodhouse current employer and previous one were both in some
> conspiracy to push firmware elsewhere. It would be very helpful if you
> stuck to the real world.

That was wrong, but there have been other legal subtexts.

request_firmware() itself, although partially done for technical reasons,
had some very real legal impetus.

Linus has even said that in this thread.

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Rafael J. Wysocki]

On Thursday, 17 of July 2008, David Woodhouse wrote:
> On Thu, 2008-07-17 at 22:42 +0200, Rafael J. Wysocki wrote:
> > 
> >   HOSTCC  firmware/ihex2fw - due to: FORCE
> > /home/rafael/src/linux-next/firmware/ihex2fw.c:268: fatal error:
> > opening dependency file firmware/.ihex2fw.d: Read-only file system
> > compilation terminated.
> 
> I'm very confused. I can do this...
> 
>  make INSTALL_FW_PATH=/tmp/foo firmware_install
>  su otheruser
>  touch firmware/.ihex2fw.cmd  (gives EPERM as expected)
>  make INSTALL_FW_PATH=/tmp/foo2 firmware_install
> 
> And it _works_ for me, after I've made that change in
> scripts/Makefile.fwinst. But I don't have .ihex2fw.d -- I
> have .ihex2fw.cmd. What is in your .ihex2fw.d?

There's no '.ihex2fw.d' under 'firmware' in my O= path.

I just do 'make modules_install' on the NFS client, after I have run
'make O=... ... firmware_install' on the server and it prints the above error
message.

Should I add anything to the 'make modules_install' command line?

Rafael

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[David Woodhouse]

On Thu, 2008-07-17 at 22:42 +0200, Rafael J. Wysocki wrote:
>   HOSTCC  firmware/ihex2fw - due to: FORCE
> /home/rafael/src/linux-next/firmware/ihex2fw.c:268: fatal error:
> opening dependency file firmware/.ihex2fw.d: Read-only file system
> compilation terminated.

I believe you can't reproduce this with Linus' current HEAD?
Please poke me if it happens again.

-- 
dwmw2

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Jeff Garzik]

Willy Tarreau wrote:
> Exactly. To give you an idea, I have a lot of servers running on a
> "firmware" which consists in two parts :
>   - a bzImage containing an initramfs with modules and a few scripts
>   - an initrd which is in fact the rootfs
> 
> When the kernel boots, it mounts the initrd, does a pivot_root and mounts
> its modules into /boot/$(uname -r). All my kernels run modules from /boot
> and not from /lib/modules, because it makes them more convenient to add
> and remove. So /lib/modules is just a symlink to /boot.
> 
> The above process is very convenient, as it is compatible with a lot of
> boot methods : hard disk, CDROM, USB stick, PXE, etc...
> 
> And moreover, I can have multiple kernels with only one rootfs (SMP, etc).
> Introducing firmware there would mean a major thinking and rework of the
> build (and possibly boot) process. The least invasive would probably be
> to stuff them next to the modules in $(uname -r)/firmware, but even then,
> it will take a lot of time to switch to a new system.
> 
> And judging from what I see around, I'm far from being the only one not
> using mkinitrd.


FWIW, due to both lack of time and lack of (apparent) interest, I didn't 
bother moving forward with the compile-firmware-into-module, even though 
I still think it would be useful to add [back].

	Jeff

Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.

[Willy Tarreau]

On Fri, Sep 12, 2008 at 04:59:36PM -0400, Jeff Garzik wrote:
> Willy Tarreau wrote:
> >Exactly. To give you an idea, I have a lot of servers running on a
> >"firmware" which consists in two parts :
> >  - a bzImage containing an initramfs with modules and a few scripts
> >  - an initrd which is in fact the rootfs
> >
> >When the kernel boots, it mounts the initrd, does a pivot_root and mounts
> >its modules into /boot/$(uname -r). All my kernels run modules from /boot
> >and not from /lib/modules, because it makes them more convenient to add
> >and remove. So /lib/modules is just a symlink to /boot.
> >
> >The above process is very convenient, as it is compatible with a lot of
> >boot methods : hard disk, CDROM, USB stick, PXE, etc...
> >
> >And moreover, I can have multiple kernels with only one rootfs (SMP, etc).
> >Introducing firmware there would mean a major thinking and rework of the
> >build (and possibly boot) process. The least invasive would probably be
> >to stuff them next to the modules in $(uname -r)/firmware, but even then,
> >it will take a lot of time to switch to a new system.
> >
> >And judging from what I see around, I'm far from being the only one not
> >using mkinitrd.
> 
> 
> FWIW, due to both lack of time and lack of (apparent) interest, I didn't 
> bother moving forward with the compile-firmware-into-module, even though 

This is what I was assuming, but thanks for clarifying this, Jeff.

> I still think it would be useful to add [back].

So do I. I think it will be easier to find time and work efficiently on
the subject when directly facing an issue.

Regards,
Willy