LKML Archive on lore.kernel.org
 help / Atom feed
* [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues
@ 2016-06-16 22:54 Luis R. Rodriguez
  2016-06-16 22:54 ` [PATCH v2 1/5] MAINTAINERS: extend firmware_class maintainer list Luis R. Rodriguez
                   ` (6 more replies)
  0 siblings, 7 replies; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-06-16 22:54 UTC (permalink / raw)
  To: ming.lei, akpm, mmarek, gregkh
  Cc: linux-kernel, markivx, stephen.boyd, zohar, broonie, tiwai,
	johannes, chunkeey, hauke, jwboyer, dmitry.torokhov, dwmw2,
	jslaby, torvalds, luto, fengguang.wu, rpurdie, j.anaszewski,
	Abhay_Salunke, Julia.Lawall, Gilles.Muller, nicolas.palix, teg,
	dhowells, Luis R. Rodriguez

The firmware API has had some issues a while ago, some of this is
not well documented, and its still hard to grasp. This documents
some of these issues, adds SmPL grammar rules to enable us to hunt
for issues, and annotations to help us with our effort to finally
compartamentalize that pesky usermode helper.

Previously this was just one patch, the grammar rule to help
find request firmware API users on init or probe, this series
extends that effort with usermode helper grammar rules, and some
annotations and documentation on the firmware_class driver to
avoid further issues. Documenting the usermode helper and making
it clear why we cannot remove it is important for analysis for
the next series which adds the new flexible sysdata firmware API.

This series depends on the coccicheck series which enables
annotations on coccinelle patches to require a specific
version of coccinelle [0], as such coordination with Michal is
in order. This series is also further extended next with the new sydata
API, the full set of changes is available on my linux-next tree [1].

Perhaps now a good time to discuss -- if 0-day should enable the rule
scripts/coccinelle/api/request_firmware-usermode.cocci to be called on
every 0-day iteration, it runs rather fast and it should help police
against avoiding futher explicit users of the usermode helper.

[0] https://lkml.kernel.org/r/1466116292-21843-1-git-send-email-mcgrof@kernel.org
[1] https://git.kernel.org/cgit/linux/kernel/git/mcgrof/linux-next.git/log/?h=20160616-sysdata-v2

Luis R. Rodriguez (5):
  MAINTAINERS: extend firmware_class maintainer list
  firmware: annotate thou shalt not request fw on init or probe
  firmware: update usermode helper docs and add SmPL report
  firmware: add usermode helper DECLARE_FW_LOADER_USER() annotation
  firmware: fix fw cache to avoid usermode helper on suspend

 Documentation/firmware_class/README                |  59 +++++++++-
 MAINTAINERS                                        |   1 +
 drivers/base/Kconfig                               |   2 +-
 drivers/base/firmware_class.c                      |   2 +-
 drivers/firmware/dell_rbu.c                        |   1 +
 drivers/leds/leds-lp55xx-common.c                  |   1 +
 include/linux/firmware.h                           |   7 ++
 .../request_firmware-avoid-init-probe-init.cocci   | 130 +++++++++++++++++++++
 .../coccinelle/api/request_firmware-usermode.cocci |  44 +++++++
 9 files changed, 240 insertions(+), 7 deletions(-)
 create mode 100644 scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
 create mode 100644 scripts/coccinelle/api/request_firmware-usermode.cocci

-- 
2.8.2

^ permalink raw reply	[flat|nested] 80+ messages in thread

* [PATCH v2 1/5] MAINTAINERS: extend firmware_class maintainer list
  2016-06-16 22:54 [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues Luis R. Rodriguez
@ 2016-06-16 22:54 ` Luis R. Rodriguez
  2016-06-16 22:54 ` [PATCH v2 2/5] firmware: annotate thou shalt not request fw on init or probe Luis R. Rodriguez
                   ` (5 subsequent siblings)
  6 siblings, 0 replies; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-06-16 22:54 UTC (permalink / raw)
  To: ming.lei, akpm, mmarek, gregkh
  Cc: linux-kernel, markivx, stephen.boyd, zohar, broonie, tiwai,
	johannes, chunkeey, hauke, jwboyer, dmitry.torokhov, dwmw2,
	jslaby, torvalds, luto, fengguang.wu, rpurdie, j.anaszewski,
	Abhay_Salunke, Julia.Lawall, Gilles.Muller, nicolas.palix, teg,
	dhowells, Luis R. Rodriguez

I've been reviewing changes proactively, and plan on doing
more of this work. I'm doing this early as I should be getting
e-mailed about proposed changes.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 MAINTAINERS | 1 +
 1 file changed, 1 insertion(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index d86978d92eeb..b030a95dcb97 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4780,6 +4780,7 @@ F:	tools/firewire/
 
 FIRMWARE LOADER (request_firmware)
 M:	Ming Lei <ming.lei@canonical.com>
+M:	Luis R. Rodriguez <mcgrof@kernel.org>
 L:	linux-kernel@vger.kernel.org
 S:	Maintained
 F:	Documentation/firmware_class/
-- 
2.8.2

^ permalink raw reply	[flat|nested] 80+ messages in thread

* [PATCH v2 2/5] firmware: annotate thou shalt not request fw on init or probe
  2016-06-16 22:54 [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues Luis R. Rodriguez
  2016-06-16 22:54 ` [PATCH v2 1/5] MAINTAINERS: extend firmware_class maintainer list Luis R. Rodriguez
@ 2016-06-16 22:54 ` Luis R. Rodriguez
  2016-08-24  6:55   ` Daniel Vetter
  2016-06-16 22:54 ` [PATCH v2 3/5] firmware: update usermode helper docs and add SmPL report Luis R. Rodriguez
                   ` (4 subsequent siblings)
  6 siblings, 1 reply; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-06-16 22:54 UTC (permalink / raw)
  To: ming.lei, akpm, mmarek, gregkh
  Cc: linux-kernel, markivx, stephen.boyd, zohar, broonie, tiwai,
	johannes, chunkeey, hauke, jwboyer, dmitry.torokhov, dwmw2,
	jslaby, torvalds, luto, fengguang.wu, rpurdie, j.anaszewski,
	Abhay_Salunke, Julia.Lawall, Gilles.Muller, nicolas.palix, teg,
	dhowells, Luis R. Rodriguez, Alessandro Rubini, Kevin Cernekee,
	Daniel Vetter, Kees Cook, Jonathan Corbet, Thierry Martinez,
	cocci, linux-serial, linux-doc, linuxppc-dev

Thou shalt not make firmware calls early on init or probe.

systemd already ripped support out for the usermode helper
a while ago, there are still users that require the usermode helper,
however systemd's use of the usermode helper exacerbated a long
lasting issue of the fact that we have many drivers that load
firmware on init or probe. Independent of the usermode helper
loading firmware on init or probe is a bad idea for a few reasons.

When the firmware is read directly from the filesystem by the kernel,
if the driver is built-in to the kernel the firmware may not yet be
available, for such uses one could use initramfs and stuff the firmware
inside, or one also use CONFIG_EXTRA_FIRMWARE; however not all distributions
use this, as such generally one cannot count on this. There is another
corner cases to consider, since we are accessing the firmware directly folks
cannot expect new found firmware on a filesystem after we switch off from
an initramfs with pivot_root().

Supporting such situations is possible today but fixing it for good is
really complex due to the large amount of variablity in the boot up
process.

Instead just document the expectations properly and add a grammar rule to
enable folks to check / validate / police if drivers are using the request
firmware API early on init or probe.

The SmPL rule used to check for the probe routine is loose and is
currently defined through a regexp, that can easily be extended to any
other known bus probe routine names. It also uses the new Python
iteration support which allows us to backtrack from a request_firmware*()
call back to a possible probe or init, iteratively. Without iteration
we would only be able to get reports for callers who directly use the
request_firmware*() API on the initial probe or init routine.

There are 4 offenders at this time:

mcgrof@ergon ~/linux-next (git::20160609)$ export COCCI=scripts/coccinelle/api/request_firmware.cocci
mcgrof@ergon ~/linux-next (git::20160609)$ make coccicheck MODE=report

drivers/fmc/fmc-fakedev.c: ERROR: driver call request firmware call on its init routine on line 321.
drivers/fmc/fmc-write-eeprom.c: ERROR: driver call request firmware call on its probe routine on line 136.
drivers/tty/serial/rp2.c: ERROR: driver call request firmware call on its probe routine on line 796.
drivers/tty/serial/ucc_uart.c: ERROR: driver call request firmware call on its probe routine on line 1246.

I checked and verified all these are valid reports. This list also matches
the same list as in 20150805, so we have fortunately not gotten worse.
Let's keep it that way and also fix these drivers.

v2: changes from v1 [0]:

o This now supports iteration, this extends our coverage on the report

o Update documentation and commit log to accept the fate of not being
  able to remove the usermode helper.

[0] https://lkml.kernel.org/r/1440811107-861-1-git-send-email-mcgrof@do-not-panic.com

Cc: Alessandro Rubini <rubini@gnudd.com>
Cc: Kevin Cernekee <cernekee@gmail.com>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: Mimi Zohar <zohar@linux.vnet.ibm.com>
Cc: David Woodhouse <dwmw2@infradead.org>
Cc: Kees Cook <keescook@chromium.org>
Cc: Dmitry Torokhov <dmitry.torokhov@gmail.com>
Cc: Ming Lei <ming.lei@canonical.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Julia Lawall <Julia.Lawall@lip6.fr>
Cc: Gilles Muller <Gilles.Muller@lip6.fr>
Cc: Nicolas Palix <nicolas.palix@imag.fr>
Cc: Thierry Martinez <martinez@nsup.org>
Cc: Michal Marek <mmarek@suse.com>
Cc: cocci@systeme.lip6.fr
Cc: Alessandro Rubini <rubini@gnudd.com>
Cc: Kevin Cernekee <cernekee@gmail.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Jiri Slaby <jslaby@suse.com>
Cc: linux-serial@vger.kernel.org
Cc: linux-doc@vger.kernel.org
Cc: linux-serial@vger.kernel.org
Cc: linuxppc-dev@lists.ozlabs.org
Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 Documentation/firmware_class/README                |  20 ++++
 drivers/base/Kconfig                               |   2 +-
 .../request_firmware-avoid-init-probe-init.cocci   | 130 +++++++++++++++++++++
 3 files changed, 151 insertions(+), 1 deletion(-)
 create mode 100644 scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci

diff --git a/Documentation/firmware_class/README b/Documentation/firmware_class/README
index cafdca8b3b15..056d1cb9d365 100644
--- a/Documentation/firmware_class/README
+++ b/Documentation/firmware_class/README
@@ -93,6 +93,26 @@
    user contexts to request firmware asynchronously, but can't be called
    in atomic contexts.
 
+Requirements:
+=============
+
+You should avoid at all costs requesting firmware on both init and probe paths
+of your device driver. Reason for this is the complexity needed to ensure a
+firmware will be available for a driver early in boot through different
+build configurations. Consider built-in drivers needing firmware early, or
+consider a driver assuming it will only get firmware after pivot_root().
+
+Drivers that really need firmware early should use stuff the firmware in
+initramfs or consider using CONFIG_EXTRA_FIRMWARE. Using initramfs is much
+more portable to more distributions as not all distributions wish to enable
+CONFIG_EXTRA_FIRMWARE. Should a driver require the firmware being built-in
+it should depend on CONFIG_EXTRA_FIRMWARE. There is no current annotation for
+requiring a firmware on initramfs.
+
+If you're a maintainer you can help police this with:
+
+$ export COCCI=scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
+$ make coccicheck MODE=report
 
  about in-kernel persistence:
  ---------------------------
diff --git a/drivers/base/Kconfig b/drivers/base/Kconfig
index 98504ec99c7d..12b4f5551501 100644
--- a/drivers/base/Kconfig
+++ b/drivers/base/Kconfig
@@ -152,7 +152,7 @@ config FW_LOADER_USER_HELPER
 	bool
 
 config FW_LOADER_USER_HELPER_FALLBACK
-	bool "Fallback user-helper invocation for firmware loading"
+	bool "Fallback user-helper invocation for firmware loading (avoid)"
 	depends on FW_LOADER
 	select FW_LOADER_USER_HELPER
 	help
diff --git a/scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci b/scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
new file mode 100644
index 000000000000..cf180c59e042
--- /dev/null
+++ b/scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
@@ -0,0 +1,130 @@
+///
+/// Thou shalt not request firmware on init or probe
+///
+// Confidence: High
+// Copyright: (C) 2016 Luis R. Rodriguez <mcgrof@kernel.org>
+// Copyright: (C) 2015 Julia Lawall, Inria/LIP6.
+//
+// GPLv2.
+// URL: http://coccinelle.lip6.fr/
+// Options: --no-includes --include-headers --no-show-diff
+// Requires: 1.0.5
+//
+// Coccinelle 1.0.5 is required given that this uses the shiny new
+// python iteration feature.
+
+virtual report
+virtual after_start
+
+// -------------------------------------------------------------------------
+
+@initialize:python@
+@@
+
+seen = set()
+
+def add_if_not_present(f, file, starter_var):
+    if (f, file, starter_var) not in seen:
+        seen.add((f, file, starter_var))
+        it = Iteration()
+        if file != None:
+            it.set_files([file])
+    it.add_virtual_rule(after_start)
+    it.add_virtual_rule(report)
+    it.add_virtual_identifier(fn, f)
+    it.add_virtual_identifier(starter, starter_var)
+    it.register()
+
+reported = set()
+
+def print_err(str, file, line):
+    if (file, line) not in reported:
+        reported.add((file, line))
+	print "%s: ERROR: driver call request firmware call on its %s routine on line %d." % (file, str, line)
+
+@ defines_module_init@
+declarer name module_init;
+identifier init;
+@@
+
+module_init(init);
+
+@ has_probe depends on defines_module_init@
+identifier drv_calls, drv_probe;
+type bus_driver;
+identifier probe_op =~ "(probe)";
+@@
+
+bus_driver drv_calls = {
+	.probe_op = drv_probe,
+};
+
+@hascall depends on !after_start && defines_module_init@
+position p;
+@@
+
+(
+request_firmware@p(...)
+|
+request_firmware_nowait@p(...)
+|
+request_firmware_direct@p(...)
+)
+
+@script:python@
+init << defines_module_init.init;
+p << hascall.p;
+@@
+
+if p[0].current_element == init:
+    print_err("init", p[0].file, int(p[0].line))
+    cocci.include_match(False)
+
+@script:python@
+drv_probe << has_probe.drv_probe;
+p << hascall.p;
+@@
+
+if p[0].current_element == drv_probe:
+    print_err("probe", p[0].file, int(p[0].line))
+    cocci.include_match(False)
+
+@script:python@
+p << hascall.p;
+@@
+
+add_if_not_present(p[0].current_element, p[0].file, p[0].line)
+
+@hasrecall depends on after_start@
+position p;
+identifier virtual.fn;
+@@
+
+fn@p(...)
+
+@script:python@
+init << defines_module_init.init;
+p << hasrecall.p;
+starter << virtual.starter;
+@@
+
+if p[0].current_element == init:
+    print_err("init", p[0].file, int(starter))
+    cocci.include_match(False)
+
+@script:python@
+drv_probe << has_probe.drv_probe;
+p << hasrecall.p;
+starter << virtual.starter;
+@@
+
+if p[0].current_element == drv_probe:
+    print_err("probe", p[0].file, int(starter))
+    cocci.include_match(False)
+
+@script:python@
+p << hasrecall.p;
+starter << virtual.starter;
+@@
+
+add_if_not_present(p[0].current_element, p[0].file, starter)
-- 
2.8.2

^ permalink raw reply	[flat|nested] 80+ messages in thread

* [PATCH v2 3/5] firmware: update usermode helper docs and add SmPL report
  2016-06-16 22:54 [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues Luis R. Rodriguez
  2016-06-16 22:54 ` [PATCH v2 1/5] MAINTAINERS: extend firmware_class maintainer list Luis R. Rodriguez
  2016-06-16 22:54 ` [PATCH v2 2/5] firmware: annotate thou shalt not request fw on init or probe Luis R. Rodriguez
@ 2016-06-16 22:54 ` Luis R. Rodriguez
  2016-06-16 22:54 ` [PATCH v2 4/5] firmware: add usermode helper DECLARE_FW_LOADER_USER() annotation Luis R. Rodriguez
                   ` (3 subsequent siblings)
  6 siblings, 0 replies; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-06-16 22:54 UTC (permalink / raw)
  To: ming.lei, akpm, mmarek, gregkh
  Cc: linux-kernel, markivx, stephen.boyd, zohar, broonie, tiwai,
	johannes, chunkeey, hauke, jwboyer, dmitry.torokhov, dwmw2,
	jslaby, torvalds, luto, fengguang.wu, rpurdie, j.anaszewski,
	Abhay_Salunke, Julia.Lawall, Gilles.Muller, nicolas.palix, teg,
	dhowells, Luis R. Rodriguez, linux-leds

We've determined that very sadly we cannot nuke the usermode helper.
The firmware code is a bit hard to follow, and so is the history,
document the reasons for why we cannot remove the usermode helper and
the only thing we can do is compartamentalize it.

While it, add a Coccinelle SmPL patch to help maintainers police
for *infidels* still using the usermode helper. Its accepted that perhaps
we can't get rid of all use cases, as otherwise we'd break userspace, so
best we can do is go on a hunt and fix incorrect users of it. Drivers
can only be transitioned out of the usermode helper once we know old
userspace cannot be not be broken by a kernel change.

The current SmPL patch reports:

$ export COCCI=scripts/coccinelle/api/request_firmware-usermode.cocci
$ make coccicheck MODE=report

drivers/leds/leds-lp55xx-common.c:227:8-31: WARNING: please check if user really needs the usermode helper
drivers/firmware/dell_rbu.c:622:17-40: WARNING: please check if user really needs the usermode helper

Cc: Richard Purdie <rpurdie@rpsys.net>
Cc: Jacek Anaszewski <j.anaszewski@samsung.com>
Cc: linux-leds@vger.kernel.org
Cc: Abhay Salunke <Abhay_Salunke@dell.com>
Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 Documentation/firmware_class/README                | 36 ++++++++++++++++++---
 .../coccinelle/api/request_firmware-usermode.cocci | 37 ++++++++++++++++++++++
 2 files changed, 68 insertions(+), 5 deletions(-)
 create mode 100644 scripts/coccinelle/api/request_firmware-usermode.cocci

diff --git a/Documentation/firmware_class/README b/Documentation/firmware_class/README
index 056d1cb9d365..00604b6d7675 100644
--- a/Documentation/firmware_class/README
+++ b/Documentation/firmware_class/README
@@ -33,7 +33,7 @@
 	than 256, user should pass 'firmware_class.path=$CUSTOMIZED_PATH'
 	if firmware_class is built in kernel(the general situation)
 
- 2), userspace:
+ 2), userspace: (AVOID)
  	- /sys/class/firmware/xxx/{loading,data} appear.
 	- hotplug gets called with a firmware identifier in $FIRMWARE
 	  and the usual hotplug environment.
@@ -41,14 +41,14 @@
 
  3), kernel: Discard any previous partial load.
 
- 4), userspace:
+ 4), userspace: (AVOID)
 		- hotplug: cat appropriate_firmware_image > \
 					/sys/class/firmware/xxx/data
 
  5), kernel: grows a buffer in PAGE_SIZE increments to hold the image as it
 	 comes in.
 
- 6), userspace:
+ 6), userspace: (AVOID)
 		- hotplug: echo 0 > /sys/class/firmware/xxx/loading
 
  7), kernel: request_firmware() returns and the driver has the firmware
@@ -66,8 +66,8 @@
 	 	copy_fw_to_device(fw_entry->data, fw_entry->size);
 	 release_firmware(fw_entry);
 
- Sample/simple hotplug script:
- ============================
+ Sample/simple hotplug script: (AVOID)
+ ==========================================
 
 	# Both $DEVPATH and $FIRMWARE are already provided in the environment.
 
@@ -114,6 +114,32 @@ If you're a maintainer you can help police this with:
 $ export COCCI=scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
 $ make coccicheck MODE=report
 
+Usermode helper
+===============
+
+The firmware API supports a usermode helper that lets userspace fetch and
+hand off firmware to a driver through sysfs. While in theory this was a
+nice feature it also presented many issues, so in practice it should be
+avoided at all costs now.
+
+Only drivers that have a really good reason for a usermode helper should
+use it. This use case must be well documented. You should try really hard
+to avoid it, at all costs.
+
+CONFIG_FW_LOADER_USER_HELPER_FALLBACK: disabled by most distributions now
+CONFIG_FW_LOADER_USER_HELPER: still enabled by most disributions
+
+When CONFIG_FW_LOADER_USER_HELPER_FALLBACK is enabled request_firmware()
+*always* calls the usermode helpers. When CONFIG_FW_LOADER_USER_HELPER is
+enabled, only if you specifically ask for it wil the usermode helper be
+called. If CONFIG_FW_LOADER_USER_HELPER_FALLBACK is disabled drivers can
+still require the usermode helper by using request_firmware_nowait().
+
+To help police for user of the usermode helper you can use:
+
+$ export COCCI=scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
+$ make coccicheck MODE=report
+
  about in-kernel persistence:
  ---------------------------
  Under some circumstances, as explained below, it would be interesting to keep
diff --git a/scripts/coccinelle/api/request_firmware-usermode.cocci b/scripts/coccinelle/api/request_firmware-usermode.cocci
new file mode 100644
index 000000000000..94ab95cb7c75
--- /dev/null
+++ b/scripts/coccinelle/api/request_firmware-usermode.cocci
@@ -0,0 +1,37 @@
+// Avoid the usermode helper at all costs
+//
+// request_firmware_nowait() API enables explicit request for the
+// usermode helper. Chances are high its use is just a copy and
+// paste bug. Before you fix the driver be sure to *verify* no
+// usermode helper tool exists that would otherwise break if
+// we replace the driver to use a non-usermode helper variant.
+//
+// Confidence: High
+//
+// Reason for low confidence:
+//
+// Copyright: (C) 2016 Luis R. Rodriguez <mcgrof@kernel.org> GPLv2.
+//
+// Options: --include-headers
+
+virtual report
+virtual context
+
+@ r1 depends on report || context @
+expression mod, name, dev, gfp, drv, cb;
+position p;
+@@
+
+(
+*request_firmware_nowait@p(mod, false, name, dev, gfp, drv, cb)
+|
+*request_firmware_nowait@p(mod, 0, name, dev, gfp, drv, cb)
+|
+*request_firmware_nowait@p(mod, FW_ACTION_NOHOTPLUG, name, dev, gfp, drv, cb)
+)
+
+@script:python depends on report@
+p << r1.p;
+@@
+
+coccilib.report.print_report(p[0], "WARNING: please check if user really needs the usermode helper")
-- 
2.8.2

^ permalink raw reply	[flat|nested] 80+ messages in thread

* [PATCH v2 4/5] firmware: add usermode helper DECLARE_FW_LOADER_USER() annotation
  2016-06-16 22:54 [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues Luis R. Rodriguez
                   ` (2 preceding siblings ...)
  2016-06-16 22:54 ` [PATCH v2 3/5] firmware: update usermode helper docs and add SmPL report Luis R. Rodriguez
@ 2016-06-16 22:54 ` Luis R. Rodriguez
  2016-06-16 22:54 ` [PATCH v2 5/5] firmware: fix fw cache to avoid usermode helper on suspend Luis R. Rodriguez
                   ` (2 subsequent siblings)
  6 siblings, 0 replies; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-06-16 22:54 UTC (permalink / raw)
  To: ming.lei, akpm, mmarek, gregkh
  Cc: linux-kernel, markivx, stephen.boyd, zohar, broonie, tiwai,
	johannes, chunkeey, hauke, jwboyer, dmitry.torokhov, dwmw2,
	jslaby, torvalds, luto, fengguang.wu, rpurdie, j.anaszewski,
	Abhay_Salunke, Julia.Lawall, Gilles.Muller, nicolas.palix, teg,
	dhowells, Luis R. Rodriguez, linux-leds

We need to ensure no one else adds *anything* that requests the
usermode helper without really meaning it, to police it we now have
an SmPL script but no formal annotation is present to help us ensure
the call has been validated.

Add a dummy DECLARE_FW_LOADER_USER() which we can use to annotate
validated calls and also clearly outline the documentation over where
the required usermode helper is documented.

All other future callers can be reported via 0-day by making use of the
Coccinelle SmPL patch.

Cc: Fengguang Wu <fengguang.wu@intel.com>
Cc: Richard Purdie <rpurdie@rpsys.net>
Cc: Jacek Anaszewski <j.anaszewski@samsung.com>
Cc: linux-leds@vger.kernel.org
Cc: Abhay Salunke <Abhay_Salunke@dell.com>
Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 drivers/firmware/dell_rbu.c                            | 1 +
 drivers/leds/leds-lp55xx-common.c                      | 1 +
 include/linux/firmware.h                               | 7 +++++++
 scripts/coccinelle/api/request_firmware-usermode.cocci | 9 ++++++++-
 4 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/drivers/firmware/dell_rbu.c b/drivers/firmware/dell_rbu.c
index 2f452f1f7c8a..53b3869c0900 100644
--- a/drivers/firmware/dell_rbu.c
+++ b/drivers/firmware/dell_rbu.c
@@ -586,6 +586,7 @@ static ssize_t read_rbu_image_type(struct file *filp, struct kobject *kobj,
 	return size;
 }
 
+DECLARE_FW_LOADER_USER("Documentation/dell_rbu.txt");
 static ssize_t write_rbu_image_type(struct file *filp, struct kobject *kobj,
 				    struct bin_attribute *bin_attr,
 				    char *buffer, loff_t pos, size_t count)
diff --git a/drivers/leds/leds-lp55xx-common.c b/drivers/leds/leds-lp55xx-common.c
index 5377f22ff994..ce1d087979a1 100644
--- a/drivers/leds/leds-lp55xx-common.c
+++ b/drivers/leds/leds-lp55xx-common.c
@@ -219,6 +219,7 @@ out:
 	release_firmware(chip->fw);
 }
 
+DECLARE_FW_LOADER_USER("Documentation/leds/leds-lp55xx.txt");
 static int lp55xx_request_firmware(struct lp55xx_chip *chip)
 {
 	const char *name = chip->cl->name;
diff --git a/include/linux/firmware.h b/include/linux/firmware.h
index bdc24ee92823..791ab8632f02 100644
--- a/include/linux/firmware.h
+++ b/include/linux/firmware.h
@@ -8,6 +8,13 @@
 #define FW_ACTION_NOHOTPLUG 0
 #define FW_ACTION_HOTPLUG 1
 
+/*
+ * Helper for scripts/coccinelle/api/request_firmware-usermode.cocci
+ * and so users can also easily search for the respectively needed
+ * usermode helper.
+ */
+#define DECLARE_FW_LOADER_USER(__usermode_helper)
+
 struct firmware {
 	size_t size;
 	const u8 *data;
diff --git a/scripts/coccinelle/api/request_firmware-usermode.cocci b/scripts/coccinelle/api/request_firmware-usermode.cocci
index 94ab95cb7c75..1775fd3067e8 100644
--- a/scripts/coccinelle/api/request_firmware-usermode.cocci
+++ b/scripts/coccinelle/api/request_firmware-usermode.cocci
@@ -17,6 +17,13 @@
 virtual report
 virtual context
 
+@ r0 depends on report || context @
+declarer name DECLARE_FW_LOADER_USER;
+expression E;
+@@
+
+DECLARE_FW_LOADER_USER(E);
+
 @ r1 depends on report || context @
 expression mod, name, dev, gfp, drv, cb;
 position p;
@@ -30,7 +37,7 @@ position p;
 *request_firmware_nowait@p(mod, FW_ACTION_NOHOTPLUG, name, dev, gfp, drv, cb)
 )
 
-@script:python depends on report@
+@script:python depends on report && !r0 @
 p << r1.p;
 @@
 
-- 
2.8.2

^ permalink raw reply	[flat|nested] 80+ messages in thread

* [PATCH v2 5/5] firmware: fix fw cache to avoid usermode helper on suspend
  2016-06-16 22:54 [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues Luis R. Rodriguez
                   ` (3 preceding siblings ...)
  2016-06-16 22:54 ` [PATCH v2 4/5] firmware: add usermode helper DECLARE_FW_LOADER_USER() annotation Luis R. Rodriguez
@ 2016-06-16 22:54 ` Luis R. Rodriguez
  2016-07-07  0:56 ` [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues Luis R. Rodriguez
  2016-08-24  0:45 ` [PATCH v3 " mcgrof
  6 siblings, 0 replies; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-06-16 22:54 UTC (permalink / raw)
  To: ming.lei, akpm, mmarek, gregkh
  Cc: linux-kernel, markivx, stephen.boyd, zohar, broonie, tiwai,
	johannes, chunkeey, hauke, jwboyer, dmitry.torokhov, dwmw2,
	jslaby, torvalds, luto, fengguang.wu, rpurdie, j.anaszewski,
	Abhay_Salunke, Julia.Lawall, Gilles.Muller, nicolas.palix, teg,
	dhowells, Luis R. Rodriguez

The firmware cache purposely kills all non-udev (usermode helper)
pending requests prior to suspend with kill_requests_without_uevent()
right before it calls out to request for firmware for the fw cache.
It is pointless to again run into the possible issue of queing up
further usermode helpers during suspend, furthermore its actually
buggy to have required the usermode helper in some cases where
clearly the driver originally had not wanted that. Fix this by
simply using the direct call.

This doesn't fix any known bug however if it should be an optimization
for suspend/resume. While at it extend documentation to ensure folks of
the usermode helper are aware that they must cache the firmware on their
own for suspend / resume.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 Documentation/firmware_class/README | 3 +++
 drivers/base/firmware_class.c       | 2 +-
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/Documentation/firmware_class/README b/Documentation/firmware_class/README
index 00604b6d7675..b7b99f4e3aaf 100644
--- a/Documentation/firmware_class/README
+++ b/Documentation/firmware_class/README
@@ -172,3 +172,6 @@ $ make coccicheck MODE=report
  resume callback, and callers needn't cache the firmware by
  themselves any more for dealing with firmware loss during system
  resume.
+
+The firmware cache is only for non-user mode helper users. Drivers that
+require the usermode helper must deal with caching on their own.
diff --git a/drivers/base/firmware_class.c b/drivers/base/firmware_class.c
index 22d1760a4278..dca4f9cbf4db 100644
--- a/drivers/base/firmware_class.c
+++ b/drivers/base/firmware_class.c
@@ -1412,7 +1412,7 @@ static int cache_firmware(const char *fw_name)
 
 	pr_debug("%s: %s\n", __func__, fw_name);
 
-	ret = request_firmware(&fw, fw_name, NULL);
+	ret = request_firmware_direct(&fw, fw_name, NULL);
 	if (!ret)
 		kfree(fw);
 
-- 
2.8.2

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues
  2016-06-16 22:54 [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues Luis R. Rodriguez
                   ` (4 preceding siblings ...)
  2016-06-16 22:54 ` [PATCH v2 5/5] firmware: fix fw cache to avoid usermode helper on suspend Luis R. Rodriguez
@ 2016-07-07  0:56 ` Luis R. Rodriguez
  2016-07-13 21:47   ` Luis R. Rodriguez
  2016-07-13 23:52   ` Fengguang Wu
  2016-08-24  0:45 ` [PATCH v3 " mcgrof
  6 siblings, 2 replies; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-07-07  0:56 UTC (permalink / raw)
  To: Luis R. Rodriguez
  Cc: ming.lei, akpm, mmarek, gregkh, linux-kernel, markivx,
	stephen.boyd, zohar, broonie, tiwai, johannes, chunkeey, hauke,
	jwboyer, dmitry.torokhov, dwmw2, jslaby, torvalds, luto,
	fengguang.wu, rpurdie, j.anaszewski, Abhay_Salunke, Julia.Lawall,
	Gilles.Muller, nicolas.palix, teg, dhowells, martin.blumenstingl,
	nbd, mark.rutland, robh+dt, arend.vanspriel, dev, kvalo

On Thu, Jun 16, 2016 at 03:54:16PM -0700, Luis R. Rodriguez wrote:
> The firmware API has had some issues a while ago, some of this is
> not well documented, and its still hard to grasp. This documents
> some of these issues, adds SmPL grammar rules to enable us to hunt
> for issues, and annotations to help us with our effort to finally
> compartamentalize that pesky usermode helper.
> 
> Previously this was just one patch, the grammar rule to help
> find request firmware API users on init or probe, this series
> extends that effort with usermode helper grammar rules, and some
> annotations and documentation on the firmware_class driver to
> avoid further issues. Documenting the usermode helper and making
> it clear why we cannot remove it is important for analysis for
> the next series which adds the new flexible sysdata firmware API.
> 
> This series depends on the coccicheck series which enables
> annotations on coccinelle patches to require a specific
> version of coccinelle [0], as such coordination with Michal is
> in order.

Michal is out until July 11, and upon further thought such coordination
is not need, the annotation is in place as comments and as such
merging this now won't have any negative effects other than the version
check. Also the patches in question for the coccicheck change are all
acked now and I expect them to be merged anyway.

Which tree should firmware changes go through ?

> This series is also further extended next with the new sydata
> API, the full set of changes is available on my linux-next tree [1].
> 
> Perhaps now a good time to discuss -- if 0-day should enable the rule
> scripts/coccinelle/api/request_firmware-usermode.cocci to be called on
> every 0-day iteration, it runs rather fast and it should help police
> against avoiding futher explicit users of the usermode helper.

And if we are going to merge this anyone oppose enabling hunting
for further explicit users of the usermode helper using grammar through
0-day ?

  Luis

> 
> [0] https://lkml.kernel.org/r/1466116292-21843-1-git-send-email-mcgrof@kernel.org
> [1] https://git.kernel.org/cgit/linux/kernel/git/mcgrof/linux-next.git/log/?h=20160616-sysdata-v2
> 
> Luis R. Rodriguez (5):
>   MAINTAINERS: extend firmware_class maintainer list
>   firmware: annotate thou shalt not request fw on init or probe
>   firmware: update usermode helper docs and add SmPL report
>   firmware: add usermode helper DECLARE_FW_LOADER_USER() annotation
>   firmware: fix fw cache to avoid usermode helper on suspend
> 
>  Documentation/firmware_class/README                |  59 +++++++++-
>  MAINTAINERS                                        |   1 +
>  drivers/base/Kconfig                               |   2 +-
>  drivers/base/firmware_class.c                      |   2 +-
>  drivers/firmware/dell_rbu.c                        |   1 +
>  drivers/leds/leds-lp55xx-common.c                  |   1 +
>  include/linux/firmware.h                           |   7 ++
>  .../request_firmware-avoid-init-probe-init.cocci   | 130 +++++++++++++++++++++
>  .../coccinelle/api/request_firmware-usermode.cocci |  44 +++++++
>  9 files changed, 240 insertions(+), 7 deletions(-)
>  create mode 100644 scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
>  create mode 100644 scripts/coccinelle/api/request_firmware-usermode.cocci
> 
> -- 
> 2.8.2
> 
> 

-- 
Luis Rodriguez, SUSE LINUX GmbH
Maxfeldstrasse 5; D-90409 Nuernberg

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues
  2016-07-07  0:56 ` [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues Luis R. Rodriguez
@ 2016-07-13 21:47   ` Luis R. Rodriguez
  2016-07-28  0:41     ` Luis R. Rodriguez
  2016-07-13 23:52   ` Fengguang Wu
  1 sibling, 1 reply; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-07-13 21:47 UTC (permalink / raw)
  To: gregkh, ming.lei, akpm, mmarek
  Cc: linux-kernel, markivx, stephen.boyd, zohar, broonie, tiwai,
	johannes, chunkeey, hauke, jwboyer, dmitry.torokhov, dwmw2,
	jslaby, torvalds, luto, fengguang.wu, rpurdie, j.anaszewski,
	Abhay_Salunke, Julia.Lawall, Gilles.Muller, nicolas.palix, teg,
	dhowells, martin.blumenstingl, nbd, mark.rutland, robh+dt,
	arend.vanspriel, dev, kvalo

On Thu, Jul 07, 2016 at 02:56:44AM +0200, Luis R. Rodriguez wrote:
> On Thu, Jun 16, 2016 at 03:54:16PM -0700, Luis R. Rodriguez wrote:
> > The firmware API has had some issues a while ago, some of this is
> > not well documented, and its still hard to grasp. This documents
> > some of these issues, adds SmPL grammar rules to enable us to hunt
> > for issues, and annotations to help us with our effort to finally
> > compartamentalize that pesky usermode helper.
> > 
> > Previously this was just one patch, the grammar rule to help
> > find request firmware API users on init or probe, this series
> > extends that effort with usermode helper grammar rules, and some
> > annotations and documentation on the firmware_class driver to
> > avoid further issues. Documenting the usermode helper and making
> > it clear why we cannot remove it is important for analysis for
> > the next series which adds the new flexible sysdata firmware API.
> > 
> > This series depends on the coccicheck series which enables
> > annotations on coccinelle patches to require a specific
> > version of coccinelle [0], as such coordination with Michal is
> > in order.
> 
> Michal is out until July 11, and upon further thought such coordination
> is not need, the annotation is in place as comments and as such
> merging this now won't have any negative effects other than the version
> check. Also the patches in question for the coccicheck change are all
> acked now and I expect them to be merged anyway.
> 
> Which tree should firmware changes go through ?
> 
> > This series is also further extended next with the new sydata
> > API, the full set of changes is available on my linux-next tree [1].
> > 
> > Perhaps now a good time to discuss -- if 0-day should enable the rule
> > scripts/coccinelle/api/request_firmware-usermode.cocci to be called on
> > every 0-day iteration, it runs rather fast and it should help police
> > against avoiding futher explicit users of the usermode helper.
> 
> And if we are going to merge this anyone oppose enabling hunting
> for further explicit users of the usermode helper using grammar through
> 0-day ?

*Poke*

  Luis

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues
  2016-07-07  0:56 ` [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues Luis R. Rodriguez
  2016-07-13 21:47   ` Luis R. Rodriguez
@ 2016-07-13 23:52   ` Fengguang Wu
  2016-07-14  2:15     ` Luis R. Rodriguez
  1 sibling, 1 reply; 80+ messages in thread
From: Fengguang Wu @ 2016-07-13 23:52 UTC (permalink / raw)
  To: Luis R. Rodriguez
  Cc: ming.lei, akpm, mmarek, gregkh, linux-kernel, markivx,
	stephen.boyd, zohar, broonie, tiwai, johannes, chunkeey, hauke,
	jwboyer, dmitry.torokhov, dwmw2, jslaby, torvalds, luto, rpurdie,
	j.anaszewski, Abhay_Salunke, Julia.Lawall, Gilles.Muller,
	nicolas.palix, teg, dhowells, martin.blumenstingl, nbd,
	mark.rutland, robh+dt, arend.vanspriel, dev, kvalo

Hi Luis,

On Thu, Jul 07, 2016 at 02:56:44AM +0200, Luis R. Rodriguez wrote:
>On Thu, Jun 16, 2016 at 03:54:16PM -0700, Luis R. Rodriguez wrote:
>> The firmware API has had some issues a while ago, some of this is
>> not well documented, and its still hard to grasp. This documents
>> some of these issues, adds SmPL grammar rules to enable us to hunt
>> for issues, and annotations to help us with our effort to finally
>> compartamentalize that pesky usermode helper.
>>
>> Previously this was just one patch, the grammar rule to help
>> find request firmware API users on init or probe, this series
>> extends that effort with usermode helper grammar rules, and some
>> annotations and documentation on the firmware_class driver to
>> avoid further issues. Documenting the usermode helper and making
>> it clear why we cannot remove it is important for analysis for
>> the next series which adds the new flexible sysdata firmware API.
>>
>> This series depends on the coccicheck series which enables
>> annotations on coccinelle patches to require a specific
>> version of coccinelle [0], as such coordination with Michal is
>> in order.
>
>Michal is out until July 11, and upon further thought such coordination
>is not need, the annotation is in place as comments and as such
>merging this now won't have any negative effects other than the version
>check. Also the patches in question for the coccicheck change are all
>acked now and I expect them to be merged anyway.
>
>Which tree should firmware changes go through ?

>> This series is also further extended next with the new sydata
>> API, the full set of changes is available on my linux-next tree [1].
>>
>> Perhaps now a good time to discuss -- if 0-day should enable the rule
>> scripts/coccinelle/api/request_firmware-usermode.cocci to be called on
>> every 0-day iteration, it runs rather fast and it should help police
>> against avoiding futher explicit users of the usermode helper.
>
>And if we are going to merge this anyone oppose enabling hunting
>for further explicit users of the usermode helper using grammar through
>0-day ?

When *.cocci scripts lands upstream they'll be auto picked up by the
0-day bot to guard new patches/commits. Are there further steps 0-day
should do for request_firmware-upstream.cocci?

Thanks,
Fengguang

>>
>> [0] https://lkml.kernel.org/r/1466116292-21843-1-git-send-email-mcgrof@kernel.org
>> [1] https://git.kernel.org/cgit/linux/kernel/git/mcgrof/linux-next.git/log/?h=20160616-sysdata-v2
>>
>> Luis R. Rodriguez (5):
>>   MAINTAINERS: extend firmware_class maintainer list
>>   firmware: annotate thou shalt not request fw on init or probe
>>   firmware: update usermode helper docs and add SmPL report
>>   firmware: add usermode helper DECLARE_FW_LOADER_USER() annotation
>>   firmware: fix fw cache to avoid usermode helper on suspend
>>
>>  Documentation/firmware_class/README                |  59 +++++++++-
>>  MAINTAINERS                                        |   1 +
>>  drivers/base/Kconfig                               |   2 +-
>>  drivers/base/firmware_class.c                      |   2 +-
>>  drivers/firmware/dell_rbu.c                        |   1 +
>>  drivers/leds/leds-lp55xx-common.c                  |   1 +
>>  include/linux/firmware.h                           |   7 ++
>>  .../request_firmware-avoid-init-probe-init.cocci   | 130 +++++++++++++++++++++
>>  .../coccinelle/api/request_firmware-usermode.cocci |  44 +++++++
>>  9 files changed, 240 insertions(+), 7 deletions(-)
>>  create mode 100644 scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
>>  create mode 100644 scripts/coccinelle/api/request_firmware-usermode.cocci
>>
>> --
>> 2.8.2
>>
>>
>
>-- 
>Luis Rodriguez, SUSE LINUX GmbH
>Maxfeldstrasse 5; D-90409 Nuernberg

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues
  2016-07-13 23:52   ` Fengguang Wu
@ 2016-07-14  2:15     ` Luis R. Rodriguez
  2016-07-14  2:23       ` Fengguang Wu
  0 siblings, 1 reply; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-07-14  2:15 UTC (permalink / raw)
  To: Fengguang Wu
  Cc: Luis R. Rodriguez, ming.lei, akpm, mmarek, gregkh, linux-kernel,
	markivx, stephen.boyd, zohar, broonie, tiwai, johannes, chunkeey,
	hauke, jwboyer, dmitry.torokhov, dwmw2, jslaby, torvalds, luto,
	rpurdie, j.anaszewski, Abhay_Salunke, Julia.Lawall,
	Gilles.Muller, nicolas.palix, teg, dhowells, martin.blumenstingl,
	nbd, mark.rutland, robh+dt, arend.vanspriel, dev, kvalo

On Thu, Jul 14, 2016 at 07:52:07AM +0800, Fengguang Wu wrote:
> Hi Luis,
> 
> On Thu, Jul 07, 2016 at 02:56:44AM +0200, Luis R. Rodriguez wrote:
> >On Thu, Jun 16, 2016 at 03:54:16PM -0700, Luis R. Rodriguez wrote:
> >>The firmware API has had some issues a while ago, some of this is
> >>not well documented, and its still hard to grasp. This documents
> >>some of these issues, adds SmPL grammar rules to enable us to hunt
> >>for issues, and annotations to help us with our effort to finally
> >>compartamentalize that pesky usermode helper.
> >>
> >>Previously this was just one patch, the grammar rule to help
> >>find request firmware API users on init or probe, this series
> >>extends that effort with usermode helper grammar rules, and some
> >>annotations and documentation on the firmware_class driver to
> >>avoid further issues. Documenting the usermode helper and making
> >>it clear why we cannot remove it is important for analysis for
> >>the next series which adds the new flexible sysdata firmware API.
> >>
> >>This series depends on the coccicheck series which enables
> >>annotations on coccinelle patches to require a specific
> >>version of coccinelle [0], as such coordination with Michal is
> >>in order.
> >
> >Michal is out until July 11, and upon further thought such coordination
> >is not need, the annotation is in place as comments and as such
> >merging this now won't have any negative effects other than the version
> >check. Also the patches in question for the coccicheck change are all
> >acked now and I expect them to be merged anyway.
> >
> >Which tree should firmware changes go through ?
> 
> >>This series is also further extended next with the new sydata
> >>API, the full set of changes is available on my linux-next tree [1].
> >>
> >>Perhaps now a good time to discuss -- if 0-day should enable the rule
> >>scripts/coccinelle/api/request_firmware-usermode.cocci to be called on
> >>every 0-day iteration, it runs rather fast and it should help police
> >>against avoiding futher explicit users of the usermode helper.
> >
> >And if we are going to merge this anyone oppose enabling hunting
> >for further explicit users of the usermode helper using grammar through
> >0-day ?
> 
> When *.cocci scripts lands upstream they'll be auto picked up by the
> 0-day bot to guard new patches/commits.

Great thanks!

> Are there further steps 0-day should do for request_firmware-upstream.cocci?

It just requires coccinelle >= 1.0.5.

  Luis

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues
  2016-07-14  2:15     ` Luis R. Rodriguez
@ 2016-07-14  2:23       ` Fengguang Wu
  2016-07-14  3:08         ` Luis R. Rodriguez
  0 siblings, 1 reply; 80+ messages in thread
From: Fengguang Wu @ 2016-07-14  2:23 UTC (permalink / raw)
  To: Luis R. Rodriguez
  Cc: ming.lei, akpm, mmarek, gregkh, linux-kernel, markivx,
	stephen.boyd, zohar, broonie, tiwai, johannes, chunkeey, hauke,
	jwboyer, dmitry.torokhov, dwmw2, jslaby, torvalds, luto, rpurdie,
	j.anaszewski, Abhay_Salunke, Julia.Lawall, Gilles.Muller,
	nicolas.palix, teg, dhowells, martin.blumenstingl, nbd,
	mark.rutland, robh+dt, arend.vanspriel, dev, kvalo, Philip Li

On Thu, Jul 14, 2016 at 04:15:01AM +0200, Luis R. Rodriguez wrote:
>On Thu, Jul 14, 2016 at 07:52:07AM +0800, Fengguang Wu wrote:
>> Hi Luis,
>>
>> On Thu, Jul 07, 2016 at 02:56:44AM +0200, Luis R. Rodriguez wrote:
>> >On Thu, Jun 16, 2016 at 03:54:16PM -0700, Luis R. Rodriguez wrote:
>> >>The firmware API has had some issues a while ago, some of this is
>> >>not well documented, and its still hard to grasp. This documents
>> >>some of these issues, adds SmPL grammar rules to enable us to hunt
>> >>for issues, and annotations to help us with our effort to finally
>> >>compartamentalize that pesky usermode helper.
>> >>
>> >>Previously this was just one patch, the grammar rule to help
>> >>find request firmware API users on init or probe, this series
>> >>extends that effort with usermode helper grammar rules, and some
>> >>annotations and documentation on the firmware_class driver to
>> >>avoid further issues. Documenting the usermode helper and making
>> >>it clear why we cannot remove it is important for analysis for
>> >>the next series which adds the new flexible sysdata firmware API.
>> >>
>> >>This series depends on the coccicheck series which enables
>> >>annotations on coccinelle patches to require a specific
>> >>version of coccinelle [0], as such coordination with Michal is
>> >>in order.
>> >
>> >Michal is out until July 11, and upon further thought such coordination
>> >is not need, the annotation is in place as comments and as such
>> >merging this now won't have any negative effects other than the version
>> >check. Also the patches in question for the coccicheck change are all
>> >acked now and I expect them to be merged anyway.
>> >
>> >Which tree should firmware changes go through ?
>>
>> >>This series is also further extended next with the new sydata
>> >>API, the full set of changes is available on my linux-next tree [1].
>> >>
>> >>Perhaps now a good time to discuss -- if 0-day should enable the rule
>> >>scripts/coccinelle/api/request_firmware-usermode.cocci to be called on
>> >>every 0-day iteration, it runs rather fast and it should help police
>> >>against avoiding futher explicit users of the usermode helper.
>> >
>> >And if we are going to merge this anyone oppose enabling hunting
>> >for further explicit users of the usermode helper using grammar through
>> >0-day ?
>>
>> When *.cocci scripts lands upstream they'll be auto picked up by the
>> 0-day bot to guard new patches/commits.
>
>Great thanks!
>
>> Are there further steps 0-day should do for request_firmware-upstream.cocci?
>
>It just requires coccinelle >= 1.0.5.

That looks easy. When do you estimate the script will land upstream?
So we can make sure upgrade coccinelle before that time.

Thanks,
Fengguang

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues
  2016-07-14  2:23       ` Fengguang Wu
@ 2016-07-14  3:08         ` Luis R. Rodriguez
  2016-07-14  3:35           ` Fengguang Wu
  0 siblings, 1 reply; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-07-14  3:08 UTC (permalink / raw)
  To: Fengguang Wu
  Cc: Luis R. Rodriguez, ming.lei, akpm, mmarek, gregkh, linux-kernel,
	markivx, stephen.boyd, zohar, broonie, tiwai, johannes, chunkeey,
	hauke, jwboyer, dmitry.torokhov, dwmw2, jslaby, torvalds, luto,
	rpurdie, j.anaszewski, Abhay_Salunke, Julia.Lawall,
	Gilles.Muller, nicolas.palix, teg, dhowells, martin.blumenstingl,
	nbd, mark.rutland, robh+dt, arend.vanspriel, dev, kvalo,
	Philip Li

On Thu, Jul 14, 2016 at 10:23:36AM +0800, Fengguang Wu wrote:
> On Thu, Jul 14, 2016 at 04:15:01AM +0200, Luis R. Rodriguez wrote:
> >On Thu, Jul 14, 2016 at 07:52:07AM +0800, Fengguang Wu wrote:
> >>Hi Luis,
> >>
> >>On Thu, Jul 07, 2016 at 02:56:44AM +0200, Luis R. Rodriguez wrote:
> >>>On Thu, Jun 16, 2016 at 03:54:16PM -0700, Luis R. Rodriguez wrote:
> >>>>The firmware API has had some issues a while ago, some of this is
> >>>>not well documented, and its still hard to grasp. This documents
> >>>>some of these issues, adds SmPL grammar rules to enable us to hunt
> >>>>for issues, and annotations to help us with our effort to finally
> >>>>compartamentalize that pesky usermode helper.
> >>>>
> >>>>Previously this was just one patch, the grammar rule to help
> >>>>find request firmware API users on init or probe, this series
> >>>>extends that effort with usermode helper grammar rules, and some
> >>>>annotations and documentation on the firmware_class driver to
> >>>>avoid further issues. Documenting the usermode helper and making
> >>>>it clear why we cannot remove it is important for analysis for
> >>>>the next series which adds the new flexible sysdata firmware API.
> >>>>
> >>>>This series depends on the coccicheck series which enables
> >>>>annotations on coccinelle patches to require a specific
> >>>>version of coccinelle [0], as such coordination with Michal is
> >>>>in order.
> >>>
> >>>Michal is out until July 11, and upon further thought such coordination
> >>>is not need, the annotation is in place as comments and as such
> >>>merging this now won't have any negative effects other than the version
> >>>check. Also the patches in question for the coccicheck change are all
> >>>acked now and I expect them to be merged anyway.
> >>>
> >>>Which tree should firmware changes go through ?
> >>
> >>>>This series is also further extended next with the new sydata
> >>>>API, the full set of changes is available on my linux-next tree [1].
> >>>>
> >>>>Perhaps now a good time to discuss -- if 0-day should enable the rule
> >>>>scripts/coccinelle/api/request_firmware-usermode.cocci to be called on
> >>>>every 0-day iteration, it runs rather fast and it should help police
> >>>>against avoiding futher explicit users of the usermode helper.
> >>>
> >>>And if we are going to merge this anyone oppose enabling hunting
> >>>for further explicit users of the usermode helper using grammar through
> >>>0-day ?
> >>
> >>When *.cocci scripts lands upstream they'll be auto picked up by the
> >>0-day bot to guard new patches/commits.
> >
> >Great thanks!
> >
> >>Are there further steps 0-day should do for request_firmware-upstream.cocci?
> >
> >It just requires coccinelle >= 1.0.5.
> 
> That looks easy. 

Nice!

> When do you estimate the script will land upstream?

Well, this series has gone by a while now without any complaints, so
I was poking to see if they can be merged now.

> So we can make sure upgrade coccinelle before that time.

There is another series which modernizes coccicheck [0] for which I just poked
at as a well [1], one change which may be of importance to you is groks the
Requires: tag on top of an SmPL patch, with that we simply just skip an SmPL
patch if the version of coccinelle is older than the one specified, with that
in place you can just upgrade when you want -- you'd just gain support for more
SmPL patches when you do. Without that coccinelle would not work (fail) on the
SmPL patch when tried. For this reason I originally had suggested perhaps
this series should be carried by Michal.

[0] https://lkml.kernel.org/r/1467238499-10889-1-git-send-email-mcgrof@kernel.org
[1] https://lkml.kernel.org/r/20160713214539.GE6239@wotan.suse.de

  Luis

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues
  2016-07-14  3:08         ` Luis R. Rodriguez
@ 2016-07-14  3:35           ` Fengguang Wu
  0 siblings, 0 replies; 80+ messages in thread
From: Fengguang Wu @ 2016-07-14  3:35 UTC (permalink / raw)
  To: Luis R. Rodriguez
  Cc: ming.lei, akpm, mmarek, gregkh, linux-kernel, markivx,
	stephen.boyd, zohar, broonie, tiwai, johannes, chunkeey, hauke,
	jwboyer, dmitry.torokhov, dwmw2, jslaby, torvalds, luto, rpurdie,
	j.anaszewski, Abhay_Salunke, Julia.Lawall, Gilles.Muller,
	nicolas.palix, teg, dhowells, martin.blumenstingl, nbd,
	mark.rutland, robh+dt, arend.vanspriel, dev, kvalo, Philip Li

On Thu, Jul 14, 2016 at 05:08:12AM +0200, Luis R. Rodriguez wrote:
>On Thu, Jul 14, 2016 at 10:23:36AM +0800, Fengguang Wu wrote:
>> On Thu, Jul 14, 2016 at 04:15:01AM +0200, Luis R. Rodriguez wrote:
>> >On Thu, Jul 14, 2016 at 07:52:07AM +0800, Fengguang Wu wrote:
>> >>Hi Luis,
>> >>
>> >>On Thu, Jul 07, 2016 at 02:56:44AM +0200, Luis R. Rodriguez wrote:
>> >>>On Thu, Jun 16, 2016 at 03:54:16PM -0700, Luis R. Rodriguez wrote:
>> >>>>The firmware API has had some issues a while ago, some of this is
>> >>>>not well documented, and its still hard to grasp. This documents
>> >>>>some of these issues, adds SmPL grammar rules to enable us to hunt
>> >>>>for issues, and annotations to help us with our effort to finally
>> >>>>compartamentalize that pesky usermode helper.
>> >>>>
>> >>>>Previously this was just one patch, the grammar rule to help
>> >>>>find request firmware API users on init or probe, this series
>> >>>>extends that effort with usermode helper grammar rules, and some
>> >>>>annotations and documentation on the firmware_class driver to
>> >>>>avoid further issues. Documenting the usermode helper and making
>> >>>>it clear why we cannot remove it is important for analysis for
>> >>>>the next series which adds the new flexible sysdata firmware API.
>> >>>>
>> >>>>This series depends on the coccicheck series which enables
>> >>>>annotations on coccinelle patches to require a specific
>> >>>>version of coccinelle [0], as such coordination with Michal is
>> >>>>in order.
>> >>>
>> >>>Michal is out until July 11, and upon further thought such coordination
>> >>>is not need, the annotation is in place as comments and as such
>> >>>merging this now won't have any negative effects other than the version
>> >>>check. Also the patches in question for the coccicheck change are all
>> >>>acked now and I expect them to be merged anyway.
>> >>>
>> >>>Which tree should firmware changes go through ?
>> >>
>> >>>>This series is also further extended next with the new sydata
>> >>>>API, the full set of changes is available on my linux-next tree [1].
>> >>>>
>> >>>>Perhaps now a good time to discuss -- if 0-day should enable the rule
>> >>>>scripts/coccinelle/api/request_firmware-usermode.cocci to be called on
>> >>>>every 0-day iteration, it runs rather fast and it should help police
>> >>>>against avoiding futher explicit users of the usermode helper.
>> >>>
>> >>>And if we are going to merge this anyone oppose enabling hunting
>> >>>for further explicit users of the usermode helper using grammar through
>> >>>0-day ?
>> >>
>> >>When *.cocci scripts lands upstream they'll be auto picked up by the
>> >>0-day bot to guard new patches/commits.
>> >
>> >Great thanks!
>> >
>> >>Are there further steps 0-day should do for request_firmware-upstream.cocci?
>> >
>> >It just requires coccinelle >= 1.0.5.
>>
>> That looks easy.
>
>Nice!
>
>> When do you estimate the script will land upstream?
>
>Well, this series has gone by a while now without any complaints, so
>I was poking to see if they can be merged now.

OK.

>> So we can make sure upgrade coccinelle before that time.
>
>There is another series which modernizes coccicheck [0] for which I just poked
>at as a well [1], one change which may be of importance to you is groks the
>Requires: tag on top of an SmPL patch, with that we simply just skip an SmPL
>patch if the version of coccinelle is older than the one specified, with that
>in place you can just upgrade when you want -- you'd just gain support for more
>SmPL patches when you do. Without that coccinelle would not work (fail) on the
>SmPL patch when tried. For this reason I originally had suggested perhaps
>this series should be carried by Michal.
>
>[0] https://lkml.kernel.org/r/1467238499-10889-1-git-send-email-mcgrof@kernel.org
>[1] https://lkml.kernel.org/r/20160713214539.GE6239@wotan.suse.de

It's glad to know these improvements. We'll watch their progress and
keep up in time. :)

Thanks,
Fengguang

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues
  2016-07-13 21:47   ` Luis R. Rodriguez
@ 2016-07-28  0:41     ` Luis R. Rodriguez
  2016-08-03 14:50       ` Luis R. Rodriguez
  0 siblings, 1 reply; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-07-28  0:41 UTC (permalink / raw)
  To: Luis R. Rodriguez
  Cc: gregkh, ming.lei, akpm, mmarek, linux-kernel, markivx,
	stephen.boyd, zohar, broonie, tiwai, johannes, chunkeey, hauke,
	jwboyer, dmitry.torokhov, dwmw2, jslaby, torvalds, luto,
	fengguang.wu, rpurdie, j.anaszewski, Abhay_Salunke, Julia.Lawall,
	Gilles.Muller, nicolas.palix, teg, dhowells, martin.blumenstingl,
	nbd, mark.rutland, robh+dt, arend.vanspriel, dev, kvalo

On Wed, Jul 13, 2016 at 11:47:52PM +0200, Luis R. Rodriguez wrote:
> On Thu, Jul 07, 2016 at 02:56:44AM +0200, Luis R. Rodriguez wrote:
> > On Thu, Jun 16, 2016 at 03:54:16PM -0700, Luis R. Rodriguez wrote:
> > > The firmware API has had some issues a while ago, some of this is
> > > not well documented, and its still hard to grasp. This documents
> > > some of these issues, adds SmPL grammar rules to enable us to hunt
> > > for issues, and annotations to help us with our effort to finally
> > > compartamentalize that pesky usermode helper.
> > > 
> > > Previously this was just one patch, the grammar rule to help
> > > find request firmware API users on init or probe, this series
> > > extends that effort with usermode helper grammar rules, and some
> > > annotations and documentation on the firmware_class driver to
> > > avoid further issues. Documenting the usermode helper and making
> > > it clear why we cannot remove it is important for analysis for
> > > the next series which adds the new flexible sysdata firmware API.
> > > 
> > > This series depends on the coccicheck series which enables
> > > annotations on coccinelle patches to require a specific
> > > version of coccinelle [0], as such coordination with Michal is
> > > in order.
> > 
> > Michal is out until July 11, and upon further thought such coordination
> > is not need, the annotation is in place as comments and as such
> > merging this now won't have any negative effects other than the version
> > check. Also the patches in question for the coccicheck change are all
> > acked now and I expect them to be merged anyway.
> > 
> > Which tree should firmware changes go through ?
> > 
> > > This series is also further extended next with the new sydata
> > > API, the full set of changes is available on my linux-next tree [1].
> > > 
> > > Perhaps now a good time to discuss -- if 0-day should enable the rule
> > > scripts/coccinelle/api/request_firmware-usermode.cocci to be called on
> > > every 0-day iteration, it runs rather fast and it should help police
> > > against avoiding futher explicit users of the usermode helper.
> > 
> > And if we are going to merge this anyone oppose enabling hunting
> > for further explicit users of the usermode helper using grammar through
> > 0-day ?
> 
> *Poke*

*Re-poke*

  Luis

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues
  2016-07-28  0:41     ` Luis R. Rodriguez
@ 2016-08-03 14:50       ` Luis R. Rodriguez
  2016-08-03 15:04         ` Greg KH
  0 siblings, 1 reply; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-08-03 14:50 UTC (permalink / raw)
  To: gregkh, ming.lei, akpm, mmarek
  Cc: linux-kernel, markivx, stephen.boyd, zohar, broonie, tiwai,
	johannes, chunkeey, hauke, jwboyer, dmitry.torokhov, dwmw2,
	jslaby, torvalds, luto, fengguang.wu, rpurdie, j.anaszewski,
	Abhay_Salunke, Julia.Lawall, Gilles.Muller, nicolas.palix, teg,
	dhowells, martin.blumenstingl, nbd, mark.rutland, robh+dt,
	arend.vanspriel, dev, kvalo, Luis R. Rodriguez

On Thu, Jul 28, 2016 at 02:41:48AM +0200, Luis R. Rodriguez wrote:
> On Wed, Jul 13, 2016 at 11:47:52PM +0200, Luis R. Rodriguez wrote:
> > On Thu, Jul 07, 2016 at 02:56:44AM +0200, Luis R. Rodriguez wrote:
> > > On Thu, Jun 16, 2016 at 03:54:16PM -0700, Luis R. Rodriguez wrote:
> > > > The firmware API has had some issues a while ago, some of this is
> > > > not well documented, and its still hard to grasp. This documents
> > > > some of these issues, adds SmPL grammar rules to enable us to hunt
> > > > for issues, and annotations to help us with our effort to finally
> > > > compartamentalize that pesky usermode helper.
> > > > 
> > > > Previously this was just one patch, the grammar rule to help
> > > > find request firmware API users on init or probe, this series
> > > > extends that effort with usermode helper grammar rules, and some
> > > > annotations and documentation on the firmware_class driver to
> > > > avoid further issues. Documenting the usermode helper and making
> > > > it clear why we cannot remove it is important for analysis for
> > > > the next series which adds the new flexible sysdata firmware API.
> > > > 
> > > > This series depends on the coccicheck series which enables
> > > > annotations on coccinelle patches to require a specific
> > > > version of coccinelle [0], as such coordination with Michal is
> > > > in order.
> > > 
> > > Michal is out until July 11, and upon further thought such coordination
> > > is not need, the annotation is in place as comments and as such
> > > merging this now won't have any negative effects other than the version
> > > check. Also the patches in question for the coccicheck change are all
> > > acked now and I expect them to be merged anyway.
> > > 
> > > Which tree should firmware changes go through ?
> > > 
> > > > This series is also further extended next with the new sydata
> > > > API, the full set of changes is available on my linux-next tree [1].
> > > > 
> > > > Perhaps now a good time to discuss -- if 0-day should enable the rule
> > > > scripts/coccinelle/api/request_firmware-usermode.cocci to be called on
> > > > every 0-day iteration, it runs rather fast and it should help police
> > > > against avoiding futher explicit users of the usermode helper.
> > > 
> > > And if we are going to merge this anyone oppose enabling hunting
> > > for further explicit users of the usermode helper using grammar through
> > > 0-day ?
> > 
> > *Poke*
> 
> *Re-poke*

Re-re-poke.

The scripts/coccicheck changes are now merged on Linus' tree, so these patches
have no other pending changes upstream.

Who's tree can this go through or is this too late now?
Andrew can these go through your tree?

  Luis

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues
  2016-08-03 14:50       ` Luis R. Rodriguez
@ 2016-08-03 15:04         ` Greg KH
  2016-08-03 17:06           ` Luis R. Rodriguez
  0 siblings, 1 reply; 80+ messages in thread
From: Greg KH @ 2016-08-03 15:04 UTC (permalink / raw)
  To: Luis R. Rodriguez
  Cc: ming.lei, akpm, mmarek, linux-kernel, markivx, stephen.boyd,
	zohar, broonie, tiwai, johannes, chunkeey, hauke, jwboyer,
	dmitry.torokhov, dwmw2, jslaby, torvalds, luto, fengguang.wu,
	rpurdie, j.anaszewski, Abhay_Salunke, Julia.Lawall,
	Gilles.Muller, nicolas.palix, teg, dhowells, martin.blumenstingl,
	nbd, mark.rutland, robh+dt, arend.vanspriel, dev, kvalo

On Wed, Aug 03, 2016 at 04:50:14PM +0200, Luis R. Rodriguez wrote:
> On Thu, Jul 28, 2016 at 02:41:48AM +0200, Luis R. Rodriguez wrote:
> > On Wed, Jul 13, 2016 at 11:47:52PM +0200, Luis R. Rodriguez wrote:
> > > On Thu, Jul 07, 2016 at 02:56:44AM +0200, Luis R. Rodriguez wrote:
> > > > On Thu, Jun 16, 2016 at 03:54:16PM -0700, Luis R. Rodriguez wrote:
> > > > > The firmware API has had some issues a while ago, some of this is
> > > > > not well documented, and its still hard to grasp. This documents
> > > > > some of these issues, adds SmPL grammar rules to enable us to hunt
> > > > > for issues, and annotations to help us with our effort to finally
> > > > > compartamentalize that pesky usermode helper.
> > > > > 
> > > > > Previously this was just one patch, the grammar rule to help
> > > > > find request firmware API users on init or probe, this series
> > > > > extends that effort with usermode helper grammar rules, and some
> > > > > annotations and documentation on the firmware_class driver to
> > > > > avoid further issues. Documenting the usermode helper and making
> > > > > it clear why we cannot remove it is important for analysis for
> > > > > the next series which adds the new flexible sysdata firmware API.
> > > > > 
> > > > > This series depends on the coccicheck series which enables
> > > > > annotations on coccinelle patches to require a specific
> > > > > version of coccinelle [0], as such coordination with Michal is
> > > > > in order.
> > > > 
> > > > Michal is out until July 11, and upon further thought such coordination
> > > > is not need, the annotation is in place as comments and as such
> > > > merging this now won't have any negative effects other than the version
> > > > check. Also the patches in question for the coccicheck change are all
> > > > acked now and I expect them to be merged anyway.
> > > > 
> > > > Which tree should firmware changes go through ?
> > > > 
> > > > > This series is also further extended next with the new sydata
> > > > > API, the full set of changes is available on my linux-next tree [1].
> > > > > 
> > > > > Perhaps now a good time to discuss -- if 0-day should enable the rule
> > > > > scripts/coccinelle/api/request_firmware-usermode.cocci to be called on
> > > > > every 0-day iteration, it runs rather fast and it should help police
> > > > > against avoiding futher explicit users of the usermode helper.
> > > > 
> > > > And if we are going to merge this anyone oppose enabling hunting
> > > > for further explicit users of the usermode helper using grammar through
> > > > 0-day ?
> > > 
> > > *Poke*
> > 
> > *Re-poke*
> 
> Re-re-poke.
> 
> The scripts/coccicheck changes are now merged on Linus' tree, so these patches
> have no other pending changes upstream.
> 
> Who's tree can this go through or is this too late now?

It's way too late for 4.8-rc1, it will have to go into a maintainer tree
after 4.8-rc1 is out.

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues
  2016-08-03 15:04         ` Greg KH
@ 2016-08-03 17:06           ` Luis R. Rodriguez
  2016-08-03 19:32             ` Greg KH
  0 siblings, 1 reply; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-08-03 17:06 UTC (permalink / raw)
  To: Greg KH
  Cc: Luis R. Rodriguez, ming.lei, akpm, mmarek, linux-kernel, markivx,
	stephen.boyd, zohar, broonie, tiwai, johannes, chunkeey, hauke,
	jwboyer, dmitry.torokhov, dwmw2, jslaby, torvalds, luto,
	fengguang.wu, rpurdie, j.anaszewski, Abhay_Salunke, Julia.Lawall,
	Gilles.Muller, nicolas.palix, teg, dhowells, martin.blumenstingl,
	nbd, mark.rutland, robh+dt, arend.vanspriel, dev, kvalo

On Wed, Aug 03, 2016 at 05:04:39PM +0200, Greg KH wrote:
> On Wed, Aug 03, 2016 at 04:50:14PM +0200, Luis R. Rodriguez wrote:
> > On Thu, Jul 28, 2016 at 02:41:48AM +0200, Luis R. Rodriguez wrote:
> > > On Wed, Jul 13, 2016 at 11:47:52PM +0200, Luis R. Rodriguez wrote:
> > > > On Thu, Jul 07, 2016 at 02:56:44AM +0200, Luis R. Rodriguez wrote:
> > > > > On Thu, Jun 16, 2016 at 03:54:16PM -0700, Luis R. Rodriguez wrote:
> > > > > > The firmware API has had some issues a while ago, some of this is
> > > > > > not well documented, and its still hard to grasp. This documents
> > > > > > some of these issues, adds SmPL grammar rules to enable us to hunt
> > > > > > for issues, and annotations to help us with our effort to finally
> > > > > > compartamentalize that pesky usermode helper.
> > > > > > 
> > > > > > Previously this was just one patch, the grammar rule to help
> > > > > > find request firmware API users on init or probe, this series
> > > > > > extends that effort with usermode helper grammar rules, and some
> > > > > > annotations and documentation on the firmware_class driver to
> > > > > > avoid further issues. Documenting the usermode helper and making
> > > > > > it clear why we cannot remove it is important for analysis for
> > > > > > the next series which adds the new flexible sysdata firmware API.
> > > > > > 
> > > > > > This series depends on the coccicheck series which enables
> > > > > > annotations on coccinelle patches to require a specific
> > > > > > version of coccinelle [0], as such coordination with Michal is
> > > > > > in order.
> > > > > 
> > > > > Michal is out until July 11, and upon further thought such coordination
> > > > > is not need, the annotation is in place as comments and as such
> > > > > merging this now won't have any negative effects other than the version
> > > > > check. Also the patches in question for the coccicheck change are all
> > > > > acked now and I expect them to be merged anyway.
> > > > > 
> > > > > Which tree should firmware changes go through ?
> > > > > 
> > > > > > This series is also further extended next with the new sydata
> > > > > > API, the full set of changes is available on my linux-next tree [1].
> > > > > > 
> > > > > > Perhaps now a good time to discuss -- if 0-day should enable the rule
> > > > > > scripts/coccinelle/api/request_firmware-usermode.cocci to be called on
> > > > > > every 0-day iteration, it runs rather fast and it should help police
> > > > > > against avoiding futher explicit users of the usermode helper.
> > > > > 
> > > > > And if we are going to merge this anyone oppose enabling hunting
> > > > > for further explicit users of the usermode helper using grammar through
> > > > > 0-day ?
> > > > 
> > > > *Poke*
> > > 
> > > *Re-poke*
> > 
> > Re-re-poke.
> > 
> > The scripts/coccicheck changes are now merged on Linus' tree, so these patches
> > have no other pending changes upstream.
> > 
> > Who's tree can this go through or is this too late now?
> 
> It's way too late for 4.8-rc1, it will have to go into a maintainer tree
> after 4.8-rc1 is out.

OK thanks, what maintainer tree should this go through ? firmware_class changes
seem to sporadically go through different maintainers, and often some changes
go in without much review. Not sure if this is helping.

For instance Stephen Boyd's commit a098ecd2fa7db8fa4f ("firmware: support
loading into a pre-allocated buffer") when in through Andrew and I saw no
formal ACK from the maintainer. Meanwhile this series lingered just as long,
had IMHO less controversial changes, but went no where. I am adding myself to
the list of maintainers for firmware_class in this series to help with this as
I've already been helping with review and I'd like to help with maintenance but
its not clear if the current tree setup helps. I don't think setting up a tree
just for firmware_class makes sense.. but not sure what to do about the tree
situation.  Would setting on going through you through a driver-core branch be
best?  How can we make things better ? Your advice is appreciated.

  Luis

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues
  2016-08-03 17:06           ` Luis R. Rodriguez
@ 2016-08-03 19:32             ` Greg KH
  2016-08-03 19:46               ` Luis R. Rodriguez
  0 siblings, 1 reply; 80+ messages in thread
From: Greg KH @ 2016-08-03 19:32 UTC (permalink / raw)
  To: Luis R. Rodriguez
  Cc: ming.lei, akpm, mmarek, linux-kernel, markivx, stephen.boyd,
	zohar, broonie, tiwai, johannes, chunkeey, hauke, jwboyer,
	dmitry.torokhov, dwmw2, jslaby, torvalds, luto, fengguang.wu,
	rpurdie, j.anaszewski, Abhay_Salunke, Julia.Lawall,
	Gilles.Muller, nicolas.palix, teg, dhowells, martin.blumenstingl,
	nbd, mark.rutland, robh+dt, arend.vanspriel, dev, kvalo

On Wed, Aug 03, 2016 at 07:06:07PM +0200, Luis R. Rodriguez wrote:
> On Wed, Aug 03, 2016 at 05:04:39PM +0200, Greg KH wrote:
> > On Wed, Aug 03, 2016 at 04:50:14PM +0200, Luis R. Rodriguez wrote:
> > > On Thu, Jul 28, 2016 at 02:41:48AM +0200, Luis R. Rodriguez wrote:
> > > > On Wed, Jul 13, 2016 at 11:47:52PM +0200, Luis R. Rodriguez wrote:
> > > > > On Thu, Jul 07, 2016 at 02:56:44AM +0200, Luis R. Rodriguez wrote:
> > > > > > On Thu, Jun 16, 2016 at 03:54:16PM -0700, Luis R. Rodriguez wrote:
> > > > > > > The firmware API has had some issues a while ago, some of this is
> > > > > > > not well documented, and its still hard to grasp. This documents
> > > > > > > some of these issues, adds SmPL grammar rules to enable us to hunt
> > > > > > > for issues, and annotations to help us with our effort to finally
> > > > > > > compartamentalize that pesky usermode helper.
> > > > > > > 
> > > > > > > Previously this was just one patch, the grammar rule to help
> > > > > > > find request firmware API users on init or probe, this series
> > > > > > > extends that effort with usermode helper grammar rules, and some
> > > > > > > annotations and documentation on the firmware_class driver to
> > > > > > > avoid further issues. Documenting the usermode helper and making
> > > > > > > it clear why we cannot remove it is important for analysis for
> > > > > > > the next series which adds the new flexible sysdata firmware API.
> > > > > > > 
> > > > > > > This series depends on the coccicheck series which enables
> > > > > > > annotations on coccinelle patches to require a specific
> > > > > > > version of coccinelle [0], as such coordination with Michal is
> > > > > > > in order.
> > > > > > 
> > > > > > Michal is out until July 11, and upon further thought such coordination
> > > > > > is not need, the annotation is in place as comments and as such
> > > > > > merging this now won't have any negative effects other than the version
> > > > > > check. Also the patches in question for the coccicheck change are all
> > > > > > acked now and I expect them to be merged anyway.
> > > > > > 
> > > > > > Which tree should firmware changes go through ?
> > > > > > 
> > > > > > > This series is also further extended next with the new sydata
> > > > > > > API, the full set of changes is available on my linux-next tree [1].
> > > > > > > 
> > > > > > > Perhaps now a good time to discuss -- if 0-day should enable the rule
> > > > > > > scripts/coccinelle/api/request_firmware-usermode.cocci to be called on
> > > > > > > every 0-day iteration, it runs rather fast and it should help police
> > > > > > > against avoiding futher explicit users of the usermode helper.
> > > > > > 
> > > > > > And if we are going to merge this anyone oppose enabling hunting
> > > > > > for further explicit users of the usermode helper using grammar through
> > > > > > 0-day ?
> > > > > 
> > > > > *Poke*
> > > > 
> > > > *Re-poke*
> > > 
> > > Re-re-poke.
> > > 
> > > The scripts/coccicheck changes are now merged on Linus' tree, so these patches
> > > have no other pending changes upstream.
> > > 
> > > Who's tree can this go through or is this too late now?
> > 
> > It's way too late for 4.8-rc1, it will have to go into a maintainer tree
> > after 4.8-rc1 is out.
> 
> OK thanks, what maintainer tree should this go through ? firmware_class changes
> seem to sporadically go through different maintainers, and often some changes
> go in without much review. Not sure if this is helping.

I usually take them, after they are acked by the firmware maintainer.  I
haven't looked to see if this series is, if it is, I can easily take
them.

But again, I can't do anything until after 4.8-rc1 is out, and then I
get to start working through my huge backlog due to a vacation and
conference travel I did last month.

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues
  2016-08-03 19:32             ` Greg KH
@ 2016-08-03 19:46               ` Luis R. Rodriguez
  0 siblings, 0 replies; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-08-03 19:46 UTC (permalink / raw)
  To: Greg KH
  Cc: Luis R. Rodriguez, ming.lei, akpm, mmarek, linux-kernel, markivx,
	stephen.boyd, zohar, broonie, tiwai, johannes, chunkeey, hauke,
	jwboyer, dmitry.torokhov, dwmw2, jslaby, torvalds, luto,
	fengguang.wu, rpurdie, j.anaszewski, Abhay_Salunke, Julia.Lawall,
	Gilles.Muller, nicolas.palix, teg, dhowells, martin.blumenstingl,
	nbd, mark.rutland, robh+dt, arend.vanspriel, dev, kvalo,
	Daniel Wagner

On Wed, Aug 03, 2016 at 09:32:41PM +0200, Greg KH wrote:
> On Wed, Aug 03, 2016 at 07:06:07PM +0200, Luis R. Rodriguez wrote:
> > On Wed, Aug 03, 2016 at 05:04:39PM +0200, Greg KH wrote:
> > > On Wed, Aug 03, 2016 at 04:50:14PM +0200, Luis R. Rodriguez wrote:
> > > > On Thu, Jul 28, 2016 at 02:41:48AM +0200, Luis R. Rodriguez wrote:
> > > > > On Wed, Jul 13, 2016 at 11:47:52PM +0200, Luis R. Rodriguez wrote:
> > > > > > On Thu, Jul 07, 2016 at 02:56:44AM +0200, Luis R. Rodriguez wrote:
> > > > > > > On Thu, Jun 16, 2016 at 03:54:16PM -0700, Luis R. Rodriguez wrote:
> > > > > > > > The firmware API has had some issues a while ago, some of this is
> > > > > > > > not well documented, and its still hard to grasp. This documents
> > > > > > > > some of these issues, adds SmPL grammar rules to enable us to hunt
> > > > > > > > for issues, and annotations to help us with our effort to finally
> > > > > > > > compartamentalize that pesky usermode helper.
> > > > > > > > 
> > > > > > > > Previously this was just one patch, the grammar rule to help
> > > > > > > > find request firmware API users on init or probe, this series
> > > > > > > > extends that effort with usermode helper grammar rules, and some
> > > > > > > > annotations and documentation on the firmware_class driver to
> > > > > > > > avoid further issues. Documenting the usermode helper and making
> > > > > > > > it clear why we cannot remove it is important for analysis for
> > > > > > > > the next series which adds the new flexible sysdata firmware API.
> > > > > > > > 
> > > > > > > > This series depends on the coccicheck series which enables
> > > > > > > > annotations on coccinelle patches to require a specific
> > > > > > > > version of coccinelle [0], as such coordination with Michal is
> > > > > > > > in order.
> > > > > > > 
> > > > > > > Michal is out until July 11, and upon further thought such coordination
> > > > > > > is not need, the annotation is in place as comments and as such
> > > > > > > merging this now won't have any negative effects other than the version
> > > > > > > check. Also the patches in question for the coccicheck change are all
> > > > > > > acked now and I expect them to be merged anyway.
> > > > > > > 
> > > > > > > Which tree should firmware changes go through ?
> > > > > > > 
> > > > > > > > This series is also further extended next with the new sydata
> > > > > > > > API, the full set of changes is available on my linux-next tree [1].
> > > > > > > > 
> > > > > > > > Perhaps now a good time to discuss -- if 0-day should enable the rule
> > > > > > > > scripts/coccinelle/api/request_firmware-usermode.cocci to be called on
> > > > > > > > every 0-day iteration, it runs rather fast and it should help police
> > > > > > > > against avoiding futher explicit users of the usermode helper.
> > > > > > > 
> > > > > > > And if we are going to merge this anyone oppose enabling hunting
> > > > > > > for further explicit users of the usermode helper using grammar through
> > > > > > > 0-day ?
> > > > > > 
> > > > > > *Poke*
> > > > > 
> > > > > *Re-poke*
> > > > 
> > > > Re-re-poke.
> > > > 
> > > > The scripts/coccicheck changes are now merged on Linus' tree, so these patches
> > > > have no other pending changes upstream.
> > > > 
> > > > Who's tree can this go through or is this too late now?
> > > 
> > > It's way too late for 4.8-rc1, it will have to go into a maintainer tree
> > > after 4.8-rc1 is out.
> > 
> > OK thanks, what maintainer tree should this go through ? firmware_class changes
> > seem to sporadically go through different maintainers, and often some changes
> > go in without much review. Not sure if this is helping.
> 
> I usually take them, after they are acked by the firmware maintainer.  I
> haven't looked to see if this series is, if it is, I can easily take
> them.

That's the thing Ming is busy it seems so this series nor the changes that
actually went through Andrew were ACKed (and I actually NACK'd the changes that
eventually did go through Andrew after I saw them appear on linux-next), so I'm
adding myself to the MAINTAINERS list to help with this.

> But again, I can't do anything until after 4.8-rc1 is out, and then I
> get to start working through my huge backlog due to a vacation and
> conference travel I did last month.

Understood. Will post once 4.8-rc1 is out.

  Luis

^ permalink raw reply	[flat|nested] 80+ messages in thread

* [PATCH v3 0/5] firmware: add SmPL grammar to avoid issues
  2016-06-16 22:54 [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues Luis R. Rodriguez
                   ` (5 preceding siblings ...)
  2016-07-07  0:56 ` [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues Luis R. Rodriguez
@ 2016-08-24  0:45 ` " mcgrof
  2016-08-24  0:45   ` [PATCH v3 1/5] MAINTAINERS: extend firmware_class maintainer list mcgrof
                     ` (5 more replies)
  6 siblings, 6 replies; 80+ messages in thread
From: mcgrof @ 2016-08-24  0:45 UTC (permalink / raw)
  To: ming.lei, akpm, gregkh
  Cc: daniel.wagner, mmarek, linux-kernel, markivx, stephen.boyd,
	zohar, broonie, tiwai, johannes, chunkeey, hauke, jwboyer,
	dmitry.torokhov, dwmw2, jslaby, torvalds, luto, fengguang.wu,
	rpurdie, j.anaszewski, Abhay_Salunke, Julia.Lawall,
	Gilles.Muller, nicolas.palix, teg, dhowells, bjorn.andersson,
	arend.vanspriel, kvalo, Luis R. Rodriguez

From: "Luis R. Rodriguez" <mcgrof@kernel.org>

Greg,

This v3 is a simple respin without any modifications other than a rebase
from the last v2 series [0] to apply to today's linux-next tre. The v2 series
did not make it in in time.

[0] https://lkml.kernel.org/r/1466117661-22075-1-git-send-email-mcgrof@kernel.org

Luis R. Rodriguez (5):
  MAINTAINERS: extend firmware_class maintainer list
  firmware: annotate thou shalt not request fw on init or probe
  firmware: update usermode helper docs and add SmPL report
  firmware: add usermode helper DECLARE_FW_LOADER_USER() annotation
  firmware: fix fw cache to avoid usermode helper on suspend

 Documentation/firmware_class/README                |  59 +++++++++-
 MAINTAINERS                                        |   1 +
 drivers/base/Kconfig                               |   2 +-
 drivers/base/firmware_class.c                      |   2 +-
 drivers/firmware/dell_rbu.c                        |   1 +
 drivers/leds/leds-lp55xx-common.c                  |   1 +
 include/linux/firmware.h                           |   7 ++
 .../request_firmware-avoid-init-probe-init.cocci   | 130 +++++++++++++++++++++
 .../coccinelle/api/request_firmware-usermode.cocci |  44 +++++++
 9 files changed, 240 insertions(+), 7 deletions(-)
 create mode 100644 scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
 create mode 100644 scripts/coccinelle/api/request_firmware-usermode.cocci

-- 
2.9.2

^ permalink raw reply	[flat|nested] 80+ messages in thread

* [PATCH v3 1/5] MAINTAINERS: extend firmware_class maintainer list
  2016-08-24  0:45 ` [PATCH v3 " mcgrof
@ 2016-08-24  0:45   ` mcgrof
  2016-08-24  0:45   ` [PATCH v3 2/5] firmware: annotate thou shalt not request fw on init or probe mcgrof
                     ` (4 subsequent siblings)
  5 siblings, 0 replies; 80+ messages in thread
From: mcgrof @ 2016-08-24  0:45 UTC (permalink / raw)
  To: ming.lei, akpm, gregkh
  Cc: daniel.wagner, mmarek, linux-kernel, markivx, stephen.boyd,
	zohar, broonie, tiwai, johannes, chunkeey, hauke, jwboyer,
	dmitry.torokhov, dwmw2, jslaby, torvalds, luto, fengguang.wu,
	rpurdie, j.anaszewski, Abhay_Salunke, Julia.Lawall,
	Gilles.Muller, nicolas.palix, teg, dhowells, bjorn.andersson,
	arend.vanspriel, kvalo, Luis R. Rodriguez

From: "Luis R. Rodriguez" <mcgrof@kernel.org>

I've been reviewing changes proactively, and plan on doing
more of this work. I'm doing this early as I should be getting
e-mailed about proposed changes.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 MAINTAINERS | 1 +
 1 file changed, 1 insertion(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 320cce87d6f8..38895e8b7b42 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4865,6 +4865,7 @@ F:	tools/firewire/
 
 FIRMWARE LOADER (request_firmware)
 M:	Ming Lei <ming.lei@canonical.com>
+M:	Luis R. Rodriguez <mcgrof@kernel.org>
 L:	linux-kernel@vger.kernel.org
 S:	Maintained
 F:	Documentation/firmware_class/
-- 
2.9.2

^ permalink raw reply	[flat|nested] 80+ messages in thread

* [PATCH v3 2/5] firmware: annotate thou shalt not request fw on init or probe
  2016-08-24  0:45 ` [PATCH v3 " mcgrof
  2016-08-24  0:45   ` [PATCH v3 1/5] MAINTAINERS: extend firmware_class maintainer list mcgrof
@ 2016-08-24  0:45   ` mcgrof
  2016-08-24  8:17     ` Gabriel Paubert
  2016-08-24  0:45   ` [PATCH v3 3/5] firmware: update usermode helper docs and add SmPL report mcgrof
                     ` (3 subsequent siblings)
  5 siblings, 1 reply; 80+ messages in thread
From: mcgrof @ 2016-08-24  0:45 UTC (permalink / raw)
  To: ming.lei, akpm, gregkh
  Cc: daniel.wagner, mmarek, linux-kernel, markivx, stephen.boyd,
	zohar, broonie, tiwai, johannes, chunkeey, hauke, jwboyer,
	dmitry.torokhov, dwmw2, jslaby, torvalds, luto, fengguang.wu,
	rpurdie, j.anaszewski, Abhay_Salunke, Julia.Lawall,
	Gilles.Muller, nicolas.palix, teg, dhowells, bjorn.andersson,
	arend.vanspriel, kvalo, Luis R. Rodriguez, Alessandro Rubini,
	Kevin Cernekee, Daniel Vetter, Kees Cook, Jonathan Corbet,
	Thierry Martinez, cocci, linux-serial, linux-doc, linuxppc-dev

From: "Luis R. Rodriguez" <mcgrof@kernel.org>

Thou shalt not make firmware calls early on init or probe.

systemd already ripped support out for the usermode helper
a while ago, there are still users that require the usermode helper,
however systemd's use of the usermode helper exacerbated a long
lasting issue of the fact that we have many drivers that load
firmware on init or probe. Independent of the usermode helper
loading firmware on init or probe is a bad idea for a few reasons.

When the firmware is read directly from the filesystem by the kernel,
if the driver is built-in to the kernel the firmware may not yet be
available, for such uses one could use initramfs and stuff the firmware
inside, or one also use CONFIG_EXTRA_FIRMWARE; however not all distributions
use this, as such generally one cannot count on this. There is another
corner cases to consider, since we are accessing the firmware directly folks
cannot expect new found firmware on a filesystem after we switch off from
an initramfs with pivot_root().

Supporting such situations is possible today but fixing it for good is
really complex due to the large amount of variablity in the boot up
process.

Instead just document the expectations properly and add a grammar rule to
enable folks to check / validate / police if drivers are using the request
firmware API early on init or probe.

The SmPL rule used to check for the probe routine is loose and is
currently defined through a regexp, that can easily be extended to any
other known bus probe routine names. It also uses the new Python
iteration support which allows us to backtrack from a request_firmware*()
call back to a possible probe or init, iteratively. Without iteration
we would only be able to get reports for callers who directly use the
request_firmware*() API on the initial probe or init routine.

There are 4 offenders at this time:

mcgrof@ergon ~/linux-next (git::20160609)$ export COCCI=scripts/coccinelle/api/request_firmware.cocci
mcgrof@ergon ~/linux-next (git::20160609)$ make coccicheck MODE=report

drivers/fmc/fmc-fakedev.c: ERROR: driver call request firmware call on its init routine on line 321.
drivers/fmc/fmc-write-eeprom.c: ERROR: driver call request firmware call on its probe routine on line 136.
drivers/tty/serial/rp2.c: ERROR: driver call request firmware call on its probe routine on line 796.
drivers/tty/serial/ucc_uart.c: ERROR: driver call request firmware call on its probe routine on line 1246.

I checked and verified all these are valid reports. This list also matches
the same list as in 20150805, so we have fortunately not gotten worse.
Let's keep it that way and also fix these drivers.

v2: changes from v1 [0]:

o This now supports iteration, this extends our coverage on the report

o Update documentation and commit log to accept the fate of not being
  able to remove the usermode helper.

[0] https://lkml.kernel.org/r/1440811107-861-1-git-send-email-mcgrof@do-not-panic.com

Cc: Alessandro Rubini <rubini@gnudd.com>
Cc: Kevin Cernekee <cernekee@gmail.com>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: Mimi Zohar <zohar@linux.vnet.ibm.com>
Cc: David Woodhouse <dwmw2@infradead.org>
Cc: Kees Cook <keescook@chromium.org>
Cc: Dmitry Torokhov <dmitry.torokhov@gmail.com>
Cc: Ming Lei <ming.lei@canonical.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Julia Lawall <Julia.Lawall@lip6.fr>
Cc: Gilles Muller <Gilles.Muller@lip6.fr>
Cc: Nicolas Palix <nicolas.palix@imag.fr>
Cc: Thierry Martinez <martinez@nsup.org>
Cc: Michal Marek <mmarek@suse.com>
Cc: cocci@systeme.lip6.fr
Cc: Alessandro Rubini <rubini@gnudd.com>
Cc: Kevin Cernekee <cernekee@gmail.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Jiri Slaby <jslaby@suse.com>
Cc: linux-serial@vger.kernel.org
Cc: linux-doc@vger.kernel.org
Cc: linux-serial@vger.kernel.org
Cc: linuxppc-dev@lists.ozlabs.org
Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 Documentation/firmware_class/README                |  20 ++++
 drivers/base/Kconfig                               |   2 +-
 .../request_firmware-avoid-init-probe-init.cocci   | 130 +++++++++++++++++++++
 3 files changed, 151 insertions(+), 1 deletion(-)
 create mode 100644 scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci

diff --git a/Documentation/firmware_class/README b/Documentation/firmware_class/README
index cafdca8b3b15..056d1cb9d365 100644
--- a/Documentation/firmware_class/README
+++ b/Documentation/firmware_class/README
@@ -93,6 +93,26 @@
    user contexts to request firmware asynchronously, but can't be called
    in atomic contexts.
 
+Requirements:
+=============
+
+You should avoid at all costs requesting firmware on both init and probe paths
+of your device driver. Reason for this is the complexity needed to ensure a
+firmware will be available for a driver early in boot through different
+build configurations. Consider built-in drivers needing firmware early, or
+consider a driver assuming it will only get firmware after pivot_root().
+
+Drivers that really need firmware early should use stuff the firmware in
+initramfs or consider using CONFIG_EXTRA_FIRMWARE. Using initramfs is much
+more portable to more distributions as not all distributions wish to enable
+CONFIG_EXTRA_FIRMWARE. Should a driver require the firmware being built-in
+it should depend on CONFIG_EXTRA_FIRMWARE. There is no current annotation for
+requiring a firmware on initramfs.
+
+If you're a maintainer you can help police this with:
+
+$ export COCCI=scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
+$ make coccicheck MODE=report
 
  about in-kernel persistence:
  ---------------------------
diff --git a/drivers/base/Kconfig b/drivers/base/Kconfig
index 98504ec99c7d..12b4f5551501 100644
--- a/drivers/base/Kconfig
+++ b/drivers/base/Kconfig
@@ -152,7 +152,7 @@ config FW_LOADER_USER_HELPER
 	bool
 
 config FW_LOADER_USER_HELPER_FALLBACK
-	bool "Fallback user-helper invocation for firmware loading"
+	bool "Fallback user-helper invocation for firmware loading (avoid)"
 	depends on FW_LOADER
 	select FW_LOADER_USER_HELPER
 	help
diff --git a/scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci b/scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
new file mode 100644
index 000000000000..cf180c59e042
--- /dev/null
+++ b/scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
@@ -0,0 +1,130 @@
+///
+/// Thou shalt not request firmware on init or probe
+///
+// Confidence: High
+// Copyright: (C) 2016 Luis R. Rodriguez <mcgrof@kernel.org>
+// Copyright: (C) 2015 Julia Lawall, Inria/LIP6.
+//
+// GPLv2.
+// URL: http://coccinelle.lip6.fr/
+// Options: --no-includes --include-headers --no-show-diff
+// Requires: 1.0.5
+//
+// Coccinelle 1.0.5 is required given that this uses the shiny new
+// python iteration feature.
+
+virtual report
+virtual after_start
+
+// -------------------------------------------------------------------------
+
+@initialize:python@
+@@
+
+seen = set()
+
+def add_if_not_present(f, file, starter_var):
+    if (f, file, starter_var) not in seen:
+        seen.add((f, file, starter_var))
+        it = Iteration()
+        if file != None:
+            it.set_files([file])
+    it.add_virtual_rule(after_start)
+    it.add_virtual_rule(report)
+    it.add_virtual_identifier(fn, f)
+    it.add_virtual_identifier(starter, starter_var)
+    it.register()
+
+reported = set()
+
+def print_err(str, file, line):
+    if (file, line) not in reported:
+        reported.add((file, line))
+	print "%s: ERROR: driver call request firmware call on its %s routine on line %d." % (file, str, line)
+
+@ defines_module_init@
+declarer name module_init;
+identifier init;
+@@
+
+module_init(init);
+
+@ has_probe depends on defines_module_init@
+identifier drv_calls, drv_probe;
+type bus_driver;
+identifier probe_op =~ "(probe)";
+@@
+
+bus_driver drv_calls = {
+	.probe_op = drv_probe,
+};
+
+@hascall depends on !after_start && defines_module_init@
+position p;
+@@
+
+(
+request_firmware@p(...)
+|
+request_firmware_nowait@p(...)
+|
+request_firmware_direct@p(...)
+)
+
+@script:python@
+init << defines_module_init.init;
+p << hascall.p;
+@@
+
+if p[0].current_element == init:
+    print_err("init", p[0].file, int(p[0].line))
+    cocci.include_match(False)
+
+@script:python@
+drv_probe << has_probe.drv_probe;
+p << hascall.p;
+@@
+
+if p[0].current_element == drv_probe:
+    print_err("probe", p[0].file, int(p[0].line))
+    cocci.include_match(False)
+
+@script:python@
+p << hascall.p;
+@@
+
+add_if_not_present(p[0].current_element, p[0].file, p[0].line)
+
+@hasrecall depends on after_start@
+position p;
+identifier virtual.fn;
+@@
+
+fn@p(...)
+
+@script:python@
+init << defines_module_init.init;
+p << hasrecall.p;
+starter << virtual.starter;
+@@
+
+if p[0].current_element == init:
+    print_err("init", p[0].file, int(starter))
+    cocci.include_match(False)
+
+@script:python@
+drv_probe << has_probe.drv_probe;
+p << hasrecall.p;
+starter << virtual.starter;
+@@
+
+if p[0].current_element == drv_probe:
+    print_err("probe", p[0].file, int(starter))
+    cocci.include_match(False)
+
+@script:python@
+p << hasrecall.p;
+starter << virtual.starter;
+@@
+
+add_if_not_present(p[0].current_element, p[0].file, starter)
-- 
2.9.2

^ permalink raw reply	[flat|nested] 80+ messages in thread

* [PATCH v3 3/5] firmware: update usermode helper docs and add SmPL report
  2016-08-24  0:45 ` [PATCH v3 " mcgrof
  2016-08-24  0:45   ` [PATCH v3 1/5] MAINTAINERS: extend firmware_class maintainer list mcgrof
  2016-08-24  0:45   ` [PATCH v3 2/5] firmware: annotate thou shalt not request fw on init or probe mcgrof
@ 2016-08-24  0:45   ` mcgrof
  2016-08-24  0:45   ` [PATCH v3 4/5] firmware: add usermode helper DECLARE_FW_LOADER_USER() annotation mcgrof
                     ` (2 subsequent siblings)
  5 siblings, 0 replies; 80+ messages in thread
From: mcgrof @ 2016-08-24  0:45 UTC (permalink / raw)
  To: ming.lei, akpm, gregkh
  Cc: daniel.wagner, mmarek, linux-kernel, markivx, stephen.boyd,
	zohar, broonie, tiwai, johannes, chunkeey, hauke, jwboyer,
	dmitry.torokhov, dwmw2, jslaby, torvalds, luto, fengguang.wu,
	rpurdie, j.anaszewski, Abhay_Salunke, Julia.Lawall,
	Gilles.Muller, nicolas.palix, teg, dhowells, bjorn.andersson,
	arend.vanspriel, kvalo, Luis R. Rodriguez, linux-leds

From: "Luis R. Rodriguez" <mcgrof@kernel.org>

We've determined that very sadly we cannot nuke the usermode helper.
The firmware code is a bit hard to follow, and so is the history,
document the reasons for why we cannot remove the usermode helper and
the only thing we can do is compartamentalize it.

While it, add a Coccinelle SmPL patch to help maintainers police
for *infidels* still using the usermode helper. Its accepted that perhaps
we can't get rid of all use cases, as otherwise we'd break userspace, so
best we can do is go on a hunt and fix incorrect users of it. Drivers
can only be transitioned out of the usermode helper once we know old
userspace cannot be not be broken by a kernel change.

The current SmPL patch reports:

$ export COCCI=scripts/coccinelle/api/request_firmware-usermode.cocci
$ make coccicheck MODE=report

drivers/leds/leds-lp55xx-common.c:227:8-31: WARNING: please check if user really needs the usermode helper
drivers/firmware/dell_rbu.c:622:17-40: WARNING: please check if user really needs the usermode helper

Cc: Richard Purdie <rpurdie@rpsys.net>
Cc: Jacek Anaszewski <j.anaszewski@samsung.com>
Cc: linux-leds@vger.kernel.org
Cc: Abhay Salunke <Abhay_Salunke@dell.com>
Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 Documentation/firmware_class/README                | 36 ++++++++++++++++++---
 .../coccinelle/api/request_firmware-usermode.cocci | 37 ++++++++++++++++++++++
 2 files changed, 68 insertions(+), 5 deletions(-)
 create mode 100644 scripts/coccinelle/api/request_firmware-usermode.cocci

diff --git a/Documentation/firmware_class/README b/Documentation/firmware_class/README
index 056d1cb9d365..00604b6d7675 100644
--- a/Documentation/firmware_class/README
+++ b/Documentation/firmware_class/README
@@ -33,7 +33,7 @@
 	than 256, user should pass 'firmware_class.path=$CUSTOMIZED_PATH'
 	if firmware_class is built in kernel(the general situation)
 
- 2), userspace:
+ 2), userspace: (AVOID)
  	- /sys/class/firmware/xxx/{loading,data} appear.
 	- hotplug gets called with a firmware identifier in $FIRMWARE
 	  and the usual hotplug environment.
@@ -41,14 +41,14 @@
 
  3), kernel: Discard any previous partial load.
 
- 4), userspace:
+ 4), userspace: (AVOID)
 		- hotplug: cat appropriate_firmware_image > \
 					/sys/class/firmware/xxx/data
 
  5), kernel: grows a buffer in PAGE_SIZE increments to hold the image as it
 	 comes in.
 
- 6), userspace:
+ 6), userspace: (AVOID)
 		- hotplug: echo 0 > /sys/class/firmware/xxx/loading
 
  7), kernel: request_firmware() returns and the driver has the firmware
@@ -66,8 +66,8 @@
 	 	copy_fw_to_device(fw_entry->data, fw_entry->size);
 	 release_firmware(fw_entry);
 
- Sample/simple hotplug script:
- ============================
+ Sample/simple hotplug script: (AVOID)
+ ==========================================
 
 	# Both $DEVPATH and $FIRMWARE are already provided in the environment.
 
@@ -114,6 +114,32 @@ If you're a maintainer you can help police this with:
 $ export COCCI=scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
 $ make coccicheck MODE=report
 
+Usermode helper
+===============
+
+The firmware API supports a usermode helper that lets userspace fetch and
+hand off firmware to a driver through sysfs. While in theory this was a
+nice feature it also presented many issues, so in practice it should be
+avoided at all costs now.
+
+Only drivers that have a really good reason for a usermode helper should
+use it. This use case must be well documented. You should try really hard
+to avoid it, at all costs.
+
+CONFIG_FW_LOADER_USER_HELPER_FALLBACK: disabled by most distributions now
+CONFIG_FW_LOADER_USER_HELPER: still enabled by most disributions
+
+When CONFIG_FW_LOADER_USER_HELPER_FALLBACK is enabled request_firmware()
+*always* calls the usermode helpers. When CONFIG_FW_LOADER_USER_HELPER is
+enabled, only if you specifically ask for it wil the usermode helper be
+called. If CONFIG_FW_LOADER_USER_HELPER_FALLBACK is disabled drivers can
+still require the usermode helper by using request_firmware_nowait().
+
+To help police for user of the usermode helper you can use:
+
+$ export COCCI=scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
+$ make coccicheck MODE=report
+
  about in-kernel persistence:
  ---------------------------
  Under some circumstances, as explained below, it would be interesting to keep
diff --git a/scripts/coccinelle/api/request_firmware-usermode.cocci b/scripts/coccinelle/api/request_firmware-usermode.cocci
new file mode 100644
index 000000000000..94ab95cb7c75
--- /dev/null
+++ b/scripts/coccinelle/api/request_firmware-usermode.cocci
@@ -0,0 +1,37 @@
+// Avoid the usermode helper at all costs
+//
+// request_firmware_nowait() API enables explicit request for the
+// usermode helper. Chances are high its use is just a copy and
+// paste bug. Before you fix the driver be sure to *verify* no
+// usermode helper tool exists that would otherwise break if
+// we replace the driver to use a non-usermode helper variant.
+//
+// Confidence: High
+//
+// Reason for low confidence:
+//
+// Copyright: (C) 2016 Luis R. Rodriguez <mcgrof@kernel.org> GPLv2.
+//
+// Options: --include-headers
+
+virtual report
+virtual context
+
+@ r1 depends on report || context @
+expression mod, name, dev, gfp, drv, cb;
+position p;
+@@
+
+(
+*request_firmware_nowait@p(mod, false, name, dev, gfp, drv, cb)
+|
+*request_firmware_nowait@p(mod, 0, name, dev, gfp, drv, cb)
+|
+*request_firmware_nowait@p(mod, FW_ACTION_NOHOTPLUG, name, dev, gfp, drv, cb)
+)
+
+@script:python depends on report@
+p << r1.p;
+@@
+
+coccilib.report.print_report(p[0], "WARNING: please check if user really needs the usermode helper")
-- 
2.9.2

^ permalink raw reply	[flat|nested] 80+ messages in thread

* [PATCH v3 4/5] firmware: add usermode helper DECLARE_FW_LOADER_USER() annotation
  2016-08-24  0:45 ` [PATCH v3 " mcgrof
                     ` (2 preceding siblings ...)
  2016-08-24  0:45   ` [PATCH v3 3/5] firmware: update usermode helper docs and add SmPL report mcgrof
@ 2016-08-24  0:45   ` mcgrof
  2016-08-24  0:45   ` [PATCH v3 5/5] firmware: fix fw cache to avoid usermode helper on suspend mcgrof
  2016-09-07  0:42   ` [PATCH v4 0/5] firmware: add SmPL grammar to avoid issues Luis R. Rodriguez
  5 siblings, 0 replies; 80+ messages in thread
From: mcgrof @ 2016-08-24  0:45 UTC (permalink / raw)
  To: ming.lei, akpm, gregkh
  Cc: daniel.wagner, mmarek, linux-kernel, markivx, stephen.boyd,
	zohar, broonie, tiwai, johannes, chunkeey, hauke, jwboyer,
	dmitry.torokhov, dwmw2, jslaby, torvalds, luto, fengguang.wu,
	rpurdie, j.anaszewski, Abhay_Salunke, Julia.Lawall,
	Gilles.Muller, nicolas.palix, teg, dhowells, bjorn.andersson,
	arend.vanspriel, kvalo, Luis R. Rodriguez, linux-leds

From: "Luis R. Rodriguez" <mcgrof@kernel.org>

We need to ensure no one else adds *anything* that requests the
usermode helper without really meaning it, to police it we now have
an SmPL script but no formal annotation is present to help us ensure
the call has been validated.

Add a dummy DECLARE_FW_LOADER_USER() which we can use to annotate
validated calls and also clearly outline the documentation over where
the required usermode helper is documented.

All other future callers can be reported via 0-day by making use of the
Coccinelle SmPL patch.

Cc: Fengguang Wu <fengguang.wu@intel.com>
Cc: Richard Purdie <rpurdie@rpsys.net>
Cc: Jacek Anaszewski <j.anaszewski@samsung.com>
Cc: linux-leds@vger.kernel.org
Cc: Abhay Salunke <Abhay_Salunke@dell.com>
Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 drivers/firmware/dell_rbu.c                            | 1 +
 drivers/leds/leds-lp55xx-common.c                      | 1 +
 include/linux/firmware.h                               | 7 +++++++
 scripts/coccinelle/api/request_firmware-usermode.cocci | 9 ++++++++-
 4 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/drivers/firmware/dell_rbu.c b/drivers/firmware/dell_rbu.c
index 2f452f1f7c8a..53b3869c0900 100644
--- a/drivers/firmware/dell_rbu.c
+++ b/drivers/firmware/dell_rbu.c
@@ -586,6 +586,7 @@ static ssize_t read_rbu_image_type(struct file *filp, struct kobject *kobj,
 	return size;
 }
 
+DECLARE_FW_LOADER_USER("Documentation/dell_rbu.txt");
 static ssize_t write_rbu_image_type(struct file *filp, struct kobject *kobj,
 				    struct bin_attribute *bin_attr,
 				    char *buffer, loff_t pos, size_t count)
diff --git a/drivers/leds/leds-lp55xx-common.c b/drivers/leds/leds-lp55xx-common.c
index 5377f22ff994..ce1d087979a1 100644
--- a/drivers/leds/leds-lp55xx-common.c
+++ b/drivers/leds/leds-lp55xx-common.c
@@ -219,6 +219,7 @@ out:
 	release_firmware(chip->fw);
 }
 
+DECLARE_FW_LOADER_USER("Documentation/leds/leds-lp55xx.txt");
 static int lp55xx_request_firmware(struct lp55xx_chip *chip)
 {
 	const char *name = chip->cl->name;
diff --git a/include/linux/firmware.h b/include/linux/firmware.h
index b1f9f0ccb8ac..147b3062039b 100644
--- a/include/linux/firmware.h
+++ b/include/linux/firmware.h
@@ -8,6 +8,13 @@
 #define FW_ACTION_NOHOTPLUG 0
 #define FW_ACTION_HOTPLUG 1
 
+/*
+ * Helper for scripts/coccinelle/api/request_firmware-usermode.cocci
+ * and so users can also easily search for the respectively needed
+ * usermode helper.
+ */
+#define DECLARE_FW_LOADER_USER(__usermode_helper)
+
 struct firmware {
 	size_t size;
 	const u8 *data;
diff --git a/scripts/coccinelle/api/request_firmware-usermode.cocci b/scripts/coccinelle/api/request_firmware-usermode.cocci
index 94ab95cb7c75..1775fd3067e8 100644
--- a/scripts/coccinelle/api/request_firmware-usermode.cocci
+++ b/scripts/coccinelle/api/request_firmware-usermode.cocci
@@ -17,6 +17,13 @@
 virtual report
 virtual context
 
+@ r0 depends on report || context @
+declarer name DECLARE_FW_LOADER_USER;
+expression E;
+@@
+
+DECLARE_FW_LOADER_USER(E);
+
 @ r1 depends on report || context @
 expression mod, name, dev, gfp, drv, cb;
 position p;
@@ -30,7 +37,7 @@ position p;
 *request_firmware_nowait@p(mod, FW_ACTION_NOHOTPLUG, name, dev, gfp, drv, cb)
 )
 
-@script:python depends on report@
+@script:python depends on report && !r0 @
 p << r1.p;
 @@
 
-- 
2.9.2

^ permalink raw reply	[flat|nested] 80+ messages in thread

* [PATCH v3 5/5] firmware: fix fw cache to avoid usermode helper on suspend
  2016-08-24  0:45 ` [PATCH v3 " mcgrof
                     ` (3 preceding siblings ...)
  2016-08-24  0:45   ` [PATCH v3 4/5] firmware: add usermode helper DECLARE_FW_LOADER_USER() annotation mcgrof
@ 2016-08-24  0:45   ` mcgrof
  2016-08-31  7:03     ` Daniel Wagner
  2016-09-07  0:42   ` [PATCH v4 0/5] firmware: add SmPL grammar to avoid issues Luis R. Rodriguez
  5 siblings, 1 reply; 80+ messages in thread
From: mcgrof @ 2016-08-24  0:45 UTC (permalink / raw)
  To: ming.lei, akpm, gregkh
  Cc: daniel.wagner, mmarek, linux-kernel, markivx, stephen.boyd,
	zohar, broonie, tiwai, johannes, chunkeey, hauke, jwboyer,
	dmitry.torokhov, dwmw2, jslaby, torvalds, luto, fengguang.wu,
	rpurdie, j.anaszewski, Abhay_Salunke, Julia.Lawall,
	Gilles.Muller, nicolas.palix, teg, dhowells, bjorn.andersson,
	arend.vanspriel, kvalo, Luis R. Rodriguez

From: "Luis R. Rodriguez" <mcgrof@kernel.org>

The firmware cache purposely kills all non-udev (usermode helper)
pending requests prior to suspend with kill_requests_without_uevent()
right before it calls out to request for firmware for the fw cache.
It is pointless to again run into the possible issue of queing up
further usermode helpers during suspend, furthermore its actually
buggy to have required the usermode helper in some cases where
clearly the driver originally had not wanted that. Fix this by
simply using the direct call.

This doesn't fix any known bug however if it should be an optimization
for suspend/resume. While at it extend documentation to ensure folks of
the usermode helper are aware that they must cache the firmware on their
own for suspend / resume.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 Documentation/firmware_class/README | 3 +++
 drivers/base/firmware_class.c       | 2 +-
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/Documentation/firmware_class/README b/Documentation/firmware_class/README
index 00604b6d7675..b7b99f4e3aaf 100644
--- a/Documentation/firmware_class/README
+++ b/Documentation/firmware_class/README
@@ -172,3 +172,6 @@ $ make coccicheck MODE=report
  resume callback, and callers needn't cache the firmware by
  themselves any more for dealing with firmware loss during system
  resume.
+
+The firmware cache is only for non-user mode helper users. Drivers that
+require the usermode helper must deal with caching on their own.
diff --git a/drivers/base/firmware_class.c b/drivers/base/firmware_class.c
index 22d1760a4278..dca4f9cbf4db 100644
--- a/drivers/base/firmware_class.c
+++ b/drivers/base/firmware_class.c
@@ -1412,7 +1412,7 @@ static int cache_firmware(const char *fw_name)
 
 	pr_debug("%s: %s\n", __func__, fw_name);
 
-	ret = request_firmware(&fw, fw_name, NULL);
+	ret = request_firmware_direct(&fw, fw_name, NULL);
 	if (!ret)
 		kfree(fw);
 
-- 
2.9.2

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v2 2/5] firmware: annotate thou shalt not request fw on init or probe
  2016-06-16 22:54 ` [PATCH v2 2/5] firmware: annotate thou shalt not request fw on init or probe Luis R. Rodriguez
@ 2016-08-24  6:55   ` Daniel Vetter
  2016-08-24 20:39     ` Luis R. Rodriguez
  0 siblings, 1 reply; 80+ messages in thread
From: Daniel Vetter @ 2016-08-24  6:55 UTC (permalink / raw)
  To: Luis R. Rodriguez
  Cc: ming.lei, Andrew Morton, Michal Marek, Greg KH,
	Linux Kernel Mailing List, markivx, stephen.boyd, zohar,
	Mark Brown, Takashi Iwai, Johannes Berg, chunkeey, hauke,
	Josh Boyer, Dmitry Torokhov, David Woodhouse, jslaby,
	Linus Torvalds, Andy Lutomirski, Wu Fengguang, rpurdie,
	j.anaszewski, Abhay_Salunke, Julia Lawall, Gilles.Muller,
	nicolas.palix, teg, David Howells, Alessandro Rubini,
	Kevin Cernekee, Kees Cook, Jonathan Corbet, Thierry Martinez,
	cocci, linux-serial, linux-doc, linuxppc-dev

On Fri, Jun 17, 2016 at 12:54 AM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> Thou shalt not make firmware calls early on init or probe.
>
> systemd already ripped support out for the usermode helper
> a while ago, there are still users that require the usermode helper,
> however systemd's use of the usermode helper exacerbated a long
> lasting issue of the fact that we have many drivers that load
> firmware on init or probe. Independent of the usermode helper
> loading firmware on init or probe is a bad idea for a few reasons.
>
> When the firmware is read directly from the filesystem by the kernel,
> if the driver is built-in to the kernel the firmware may not yet be
> available, for such uses one could use initramfs and stuff the firmware
> inside, or one also use CONFIG_EXTRA_FIRMWARE; however not all distributions
> use this, as such generally one cannot count on this. There is another
> corner cases to consider, since we are accessing the firmware directly folks
> cannot expect new found firmware on a filesystem after we switch off from
> an initramfs with pivot_root().
>
> Supporting such situations is possible today but fixing it for good is
> really complex due to the large amount of variablity in the boot up
> process.
>
> Instead just document the expectations properly and add a grammar rule to
> enable folks to check / validate / police if drivers are using the request
> firmware API early on init or probe.
>
> The SmPL rule used to check for the probe routine is loose and is
> currently defined through a regexp, that can easily be extended to any
> other known bus probe routine names. It also uses the new Python
> iteration support which allows us to backtrack from a request_firmware*()
> call back to a possible probe or init, iteratively. Without iteration
> we would only be able to get reports for callers who directly use the
> request_firmware*() API on the initial probe or init routine.
>
> There are 4 offenders at this time:
>
> mcgrof@ergon ~/linux-next (git::20160609)$ export COCCI=scripts/coccinelle/api/request_firmware.cocci
> mcgrof@ergon ~/linux-next (git::20160609)$ make coccicheck MODE=report
>
> drivers/fmc/fmc-fakedev.c: ERROR: driver call request firmware call on its init routine on line 321.
> drivers/fmc/fmc-write-eeprom.c: ERROR: driver call request firmware call on its probe routine on line 136.
> drivers/tty/serial/rp2.c: ERROR: driver call request firmware call on its probe routine on line 796.
> drivers/tty/serial/ucc_uart.c: ERROR: driver call request firmware call on its probe routine on line 1246.

Plus all gpu drivers which need firmware. And yes we must load them at
probe because people are generally pissed when they boot their machine
and the screen goes black. On top of that a lot of people want their
gpu drivers to be built-in, but can't ship the firmware blobs in the
kernel image because gpl. Yep, there's a bit a contradiction there ...

I think what would work is loading the different subsystems of the
driver in parallel (we already do that largely), and then if one
firmware blob isn't there yet simply stall that async worker until it
shows up. But the answers I've gotten thus far from request_firmware()
folks (well at least Greg) is don't do that ...

Is that problem still somewhere on the radar? Atm there's various
wait_for_rootfs hacks out there floating in vendor/product trees.
"Avoid at all costs" sounds like upstream prefers to not care about
android/cros in those case (yes I know most arm socs don't need
firmware, which would make it a problem fo just be a subset of all
devices).
-Daniel

> I checked and verified all these are valid reports. This list also matches
> the same list as in 20150805, so we have fortunately not gotten worse.
> Let's keep it that way and also fix these drivers.
>
> v2: changes from v1 [0]:
>
> o This now supports iteration, this extends our coverage on the report
>
> o Update documentation and commit log to accept the fate of not being
>   able to remove the usermode helper.
>
> [0] https://lkml.kernel.org/r/1440811107-861-1-git-send-email-mcgrof@do-not-panic.com
>
> Cc: Alessandro Rubini <rubini@gnudd.com>
> Cc: Kevin Cernekee <cernekee@gmail.com>
> Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
> Cc: Mimi Zohar <zohar@linux.vnet.ibm.com>
> Cc: David Woodhouse <dwmw2@infradead.org>
> Cc: Kees Cook <keescook@chromium.org>
> Cc: Dmitry Torokhov <dmitry.torokhov@gmail.com>
> Cc: Ming Lei <ming.lei@canonical.com>
> Cc: Jonathan Corbet <corbet@lwn.net>
> Cc: Julia Lawall <Julia.Lawall@lip6.fr>
> Cc: Gilles Muller <Gilles.Muller@lip6.fr>
> Cc: Nicolas Palix <nicolas.palix@imag.fr>
> Cc: Thierry Martinez <martinez@nsup.org>
> Cc: Michal Marek <mmarek@suse.com>
> Cc: cocci@systeme.lip6.fr
> Cc: Alessandro Rubini <rubini@gnudd.com>
> Cc: Kevin Cernekee <cernekee@gmail.com>
> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> Cc: Jiri Slaby <jslaby@suse.com>
> Cc: linux-serial@vger.kernel.org
> Cc: linux-doc@vger.kernel.org
> Cc: linux-serial@vger.kernel.org
> Cc: linuxppc-dev@lists.ozlabs.org
> Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
> ---
>  Documentation/firmware_class/README                |  20 ++++
>  drivers/base/Kconfig                               |   2 +-
>  .../request_firmware-avoid-init-probe-init.cocci   | 130 +++++++++++++++++++++
>  3 files changed, 151 insertions(+), 1 deletion(-)
>  create mode 100644 scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
>
> diff --git a/Documentation/firmware_class/README b/Documentation/firmware_class/README
> index cafdca8b3b15..056d1cb9d365 100644
> --- a/Documentation/firmware_class/README
> +++ b/Documentation/firmware_class/README
> @@ -93,6 +93,26 @@
>     user contexts to request firmware asynchronously, but can't be called
>     in atomic contexts.
>
> +Requirements:
> +=============
> +
> +You should avoid at all costs requesting firmware on both init and probe paths
> +of your device driver. Reason for this is the complexity needed to ensure a
> +firmware will be available for a driver early in boot through different
> +build configurations. Consider built-in drivers needing firmware early, or
> +consider a driver assuming it will only get firmware after pivot_root().
> +
> +Drivers that really need firmware early should use stuff the firmware in
> +initramfs or consider using CONFIG_EXTRA_FIRMWARE. Using initramfs is much
> +more portable to more distributions as not all distributions wish to enable
> +CONFIG_EXTRA_FIRMWARE. Should a driver require the firmware being built-in
> +it should depend on CONFIG_EXTRA_FIRMWARE. There is no current annotation for
> +requiring a firmware on initramfs.
> +
> +If you're a maintainer you can help police this with:
> +
> +$ export COCCI=scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
> +$ make coccicheck MODE=report
>
>   about in-kernel persistence:
>   ---------------------------
> diff --git a/drivers/base/Kconfig b/drivers/base/Kconfig
> index 98504ec99c7d..12b4f5551501 100644
> --- a/drivers/base/Kconfig
> +++ b/drivers/base/Kconfig
> @@ -152,7 +152,7 @@ config FW_LOADER_USER_HELPER
>         bool
>
>  config FW_LOADER_USER_HELPER_FALLBACK
> -       bool "Fallback user-helper invocation for firmware loading"
> +       bool "Fallback user-helper invocation for firmware loading (avoid)"
>         depends on FW_LOADER
>         select FW_LOADER_USER_HELPER
>         help
> diff --git a/scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci b/scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
> new file mode 100644
> index 000000000000..cf180c59e042
> --- /dev/null
> +++ b/scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
> @@ -0,0 +1,130 @@
> +///
> +/// Thou shalt not request firmware on init or probe
> +///
> +// Confidence: High
> +// Copyright: (C) 2016 Luis R. Rodriguez <mcgrof@kernel.org>
> +// Copyright: (C) 2015 Julia Lawall, Inria/LIP6.
> +//
> +// GPLv2.
> +// URL: http://coccinelle.lip6.fr/
> +// Options: --no-includes --include-headers --no-show-diff
> +// Requires: 1.0.5
> +//
> +// Coccinelle 1.0.5 is required given that this uses the shiny new
> +// python iteration feature.
> +
> +virtual report
> +virtual after_start
> +
> +// -------------------------------------------------------------------------
> +
> +@initialize:python@
> +@@
> +
> +seen = set()
> +
> +def add_if_not_present(f, file, starter_var):
> +    if (f, file, starter_var) not in seen:
> +        seen.add((f, file, starter_var))
> +        it = Iteration()
> +        if file != None:
> +            it.set_files([file])
> +    it.add_virtual_rule(after_start)
> +    it.add_virtual_rule(report)
> +    it.add_virtual_identifier(fn, f)
> +    it.add_virtual_identifier(starter, starter_var)
> +    it.register()
> +
> +reported = set()
> +
> +def print_err(str, file, line):
> +    if (file, line) not in reported:
> +        reported.add((file, line))
> +       print "%s: ERROR: driver call request firmware call on its %s routine on line %d." % (file, str, line)
> +
> +@ defines_module_init@
> +declarer name module_init;
> +identifier init;
> +@@
> +
> +module_init(init);
> +
> +@ has_probe depends on defines_module_init@
> +identifier drv_calls, drv_probe;
> +type bus_driver;
> +identifier probe_op =~ "(probe)";
> +@@
> +
> +bus_driver drv_calls = {
> +       .probe_op = drv_probe,
> +};
> +
> +@hascall depends on !after_start && defines_module_init@
> +position p;
> +@@
> +
> +(
> +request_firmware@p(...)
> +|
> +request_firmware_nowait@p(...)
> +|
> +request_firmware_direct@p(...)
> +)
> +
> +@script:python@
> +init << defines_module_init.init;
> +p << hascall.p;
> +@@
> +
> +if p[0].current_element == init:
> +    print_err("init", p[0].file, int(p[0].line))
> +    cocci.include_match(False)
> +
> +@script:python@
> +drv_probe << has_probe.drv_probe;
> +p << hascall.p;
> +@@
> +
> +if p[0].current_element == drv_probe:
> +    print_err("probe", p[0].file, int(p[0].line))
> +    cocci.include_match(False)
> +
> +@script:python@
> +p << hascall.p;
> +@@
> +
> +add_if_not_present(p[0].current_element, p[0].file, p[0].line)
> +
> +@hasrecall depends on after_start@
> +position p;
> +identifier virtual.fn;
> +@@
> +
> +fn@p(...)
> +
> +@script:python@
> +init << defines_module_init.init;
> +p << hasrecall.p;
> +starter << virtual.starter;
> +@@
> +
> +if p[0].current_element == init:
> +    print_err("init", p[0].file, int(starter))
> +    cocci.include_match(False)
> +
> +@script:python@
> +drv_probe << has_probe.drv_probe;
> +p << hasrecall.p;
> +starter << virtual.starter;
> +@@
> +
> +if p[0].current_element == drv_probe:
> +    print_err("probe", p[0].file, int(starter))
> +    cocci.include_match(False)
> +
> +@script:python@
> +p << hasrecall.p;
> +starter << virtual.starter;
> +@@
> +
> +add_if_not_present(p[0].current_element, p[0].file, starter)
> --
> 2.8.2
>



-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v3 2/5] firmware: annotate thou shalt not request fw on init or probe
  2016-08-24  0:45   ` [PATCH v3 2/5] firmware: annotate thou shalt not request fw on init or probe mcgrof
@ 2016-08-24  8:17     ` Gabriel Paubert
  2016-09-02 18:26       ` Luis R. Rodriguez
  0 siblings, 1 reply; 80+ messages in thread
From: Gabriel Paubert @ 2016-08-24  8:17 UTC (permalink / raw)
  To: mcgrof
  Cc: ming.lei, akpm, gregkh, dmitry.torokhov, linux-doc,
	Gilles.Muller, tiwai, Daniel Vetter, Alessandro Rubini,
	stephen.boyd, bjorn.andersson, teg, chunkeey, cocci, jwboyer,
	Jonathan Corbet, Kevin Cernekee, Thierry Martinez, linux-serial,
	jslaby, zohar, arend.vanspriel, linuxppc-dev, Kees Cook, hauke,
	nicolas.palix, Abhay_Salunke, Julia.Lawall, broonie,
	j.anaszewski, kvalo, dhowells, dwmw2, markivx, daniel.wagner,
	linux-kernel, luto, mmarek, rpurdie, johannes, fengguang.wu,
	torvalds

On Tue, Aug 23, 2016 at 05:45:04PM -0700, mcgrof@kernel.org wrote:

[snip]
> ---
>  Documentation/firmware_class/README                |  20 ++++
>  drivers/base/Kconfig                               |   2 +-
>  .../request_firmware-avoid-init-probe-init.cocci   | 130 +++++++++++++++++++++
>  3 files changed, 151 insertions(+), 1 deletion(-)
>  create mode 100644 scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
> 
> diff --git a/Documentation/firmware_class/README b/Documentation/firmware_class/README
> index cafdca8b3b15..056d1cb9d365 100644
> --- a/Documentation/firmware_class/README
> +++ b/Documentation/firmware_class/README
> @@ -93,6 +93,26 @@
>     user contexts to request firmware asynchronously, but can't be called
>     in atomic contexts.
>  
> +Requirements:
> +=============
> +
> +You should avoid at all costs requesting firmware on both init and probe paths
> +of your device driver. Reason for this is the complexity needed to ensure a
> +firmware will be available for a driver early in boot through different
> +build configurations. Consider built-in drivers needing firmware early, or
> +consider a driver assuming it will only get firmware after pivot_root().
> +
> +Drivers that really need firmware early should use stuff the firmware in

Minor grammatical nit: s/use//

> +initramfs or consider using CONFIG_EXTRA_FIRMWARE. Using initramfs is much
> +more portable to more distributions as not all distributions wish to enable
> +CONFIG_EXTRA_FIRMWARE. Should a driver require the firmware being built-in
> +it should depend on CONFIG_EXTRA_FIRMWARE. There is no current annotation for
> +requiring a firmware on initramfs.
> +
> +If you're a maintainer you can help police this with:
> +
> +$ export COCCI=scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
> +$ make coccicheck MODE=report
>  
>   about in-kernel persistence:
>   ---------------------------

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v2 2/5] firmware: annotate thou shalt not request fw on init or probe
  2016-08-24  6:55   ` Daniel Vetter
@ 2016-08-24 20:39     ` Luis R. Rodriguez
  2016-08-25 11:05       ` Daniel Vetter
  0 siblings, 1 reply; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-08-24 20:39 UTC (permalink / raw)
  To: Daniel Vetter
  Cc: Luis R. Rodriguez, ming.lei, Andrew Morton, Michal Marek,
	Greg KH, Linux Kernel Mailing List, markivx, stephen.boyd, zohar,
	Mark Brown, Takashi Iwai, Johannes Berg, chunkeey, hauke,
	Josh Boyer, Dmitry Torokhov, David Woodhouse, jslaby,
	Linus Torvalds, Andy Lutomirski, Wu Fengguang, rpurdie,
	j.anaszewski, Abhay_Salunke, Julia Lawall, Gilles.Muller,
	nicolas.palix, teg, David Howells, Alessandro Rubini,
	Kevin Cernekee, Kees Cook, Jonathan Corbet, Thierry Martinez,
	cocci, linux-serial, linux-doc, linuxppc-dev

On Wed, Aug 24, 2016 at 08:55:55AM +0200, Daniel Vetter wrote:
> On Fri, Jun 17, 2016 at 12:54 AM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> > Thou shalt not make firmware calls early on init or probe.

<-- snip -->

> > There are 4 offenders at this time:
> >
> > mcgrof@ergon ~/linux-next (git::20160609)$ export COCCI=scripts/coccinelle/api/request_firmware.cocci
> > mcgrof@ergon ~/linux-next (git::20160609)$ make coccicheck MODE=report
> >
> > drivers/fmc/fmc-fakedev.c: ERROR: driver call request firmware call on its init routine on line 321.
> > drivers/fmc/fmc-write-eeprom.c: ERROR: driver call request firmware call on its probe routine on line 136.
> > drivers/tty/serial/rp2.c: ERROR: driver call request firmware call on its probe routine on line 796.
> > drivers/tty/serial/ucc_uart.c: ERROR: driver call request firmware call on its probe routine on line 1246.
> 
> Plus all gpu drivers which need firmware. And yes we must load them at
> probe

Do you have an upstream driver in mind that does this ? Is it on device
drier module probe or a DRM subsystem specific probe call of some sort ?

> because people are generally pissed when they boot their machine
> and the screen goes black. On top of that a lot of people want their
> gpu drivers to be built-in, but can't ship the firmware blobs in the
> kernel image because gpl. Yep, there's a bit a contradiction there ...

Can they use initramfs for this ?

Also just curious -- as with other subsystems, is it possible to load
a generic driver first, say vesa, and then a more enhanced one later ?
I am not saying this is ideal or am I suggesting this, I'd just like
to know the feasibility of this.

> I think what would work is loading the different subsystems of the
> driver in parallel (we already do that largely)

Init level stuff is actually pretty synchronous, and in fact both
init and probe are called serially. There are a few subsystems which
have been doing things a bit differently, but these are exceptions.

When you say we already do this largely, can you describe a bit more
precisely what you mean ?

>, and then if one
> firmware blob isn't there yet simply stall that async worker until it
> shows up.

Is this an existing framework or do you mean if we add something
generic to do this async loading of subsystems ?

> But the answers I've gotten thus far from request_firmware()
> folks (well at least Greg) is don't do that ...

Well in this patch set I'm adding myself as a MAINTAINER and I've
been extending the firmware API recently to help with a few new
features I want, I've been wanting to hear more feedback from
other's needs and I had actually not gotten much -- except
only recently with the usermode helper and reasons why some
folks thought they could not use direct firmware loading from
the fs. I'm keen to hear or more use cases and needs specially if
they have to do with improving boot time and asynchronous boot.

> Is that problem still somewhere on the radar? 

Not mine.

> Atm there's various
> wait_for_rootfs hacks out there floating in vendor/product trees.

This one I've heard about recently, and I suggested two possible
solutions, with a preference to a simply notification of when
rootfs is available from userspace.

> "Avoid at all costs" sounds like upstream prefers to not care about
> android/cros in those case (yes I know most arm socs don't need
> firmware, which would make it a problem fo just be a subset of all
> devices).

In my days of dealing with Android I learned most folks did not frankly
care too much about upstream-first model. That means things were reactive.
That's a business mind set and that's fine. However for upstream we want
what is best and to discuss. I haven't seen proposals so, so long as
we just hear of "hacks" that some folks do in vendor/product trees,
what can we do ?

In so far as async probe is concerned -- that is now upstream.

http://www.do-not-panic.com/2015/12/linux-asynchronous-probe.html

In so far as modules are concerned -- this should work without issue now, and
if there is an issue its very likely a bug in the subsystem.  As I noted in the
post, built-in support requires more love. A simple option for you to test this
is to test the two debug patches at the end there and boot. Alternatively inits
can just peg the async request for all modules. Should be an easy change, just
hadn't had a change to do it yet. Maybe its time.

I'm also trying to make more async functionality possible early in boot with
dependencies annotated somehow, and have a bit of work to help with this (refer
to recent linker tables patches) already which may even be possible to used to
facelift our old kernel init levels -- but as I've studied this I've also
observed others working on very similar problems, nothing is quite taking a
large picture of this and trying to generalize this. Its why I proposed it as a
topic for KS.

So .. I agree, let's avoid the hacks. Patches welcomed.

  Luis

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v2 2/5] firmware: annotate thou shalt not request fw on init or probe
  2016-08-24 20:39     ` Luis R. Rodriguez
@ 2016-08-25 11:05       ` Daniel Vetter
  2016-08-25 19:41         ` Luis R. Rodriguez
  0 siblings, 1 reply; 80+ messages in thread
From: Daniel Vetter @ 2016-08-25 11:05 UTC (permalink / raw)
  To: Luis R. Rodriguez
  Cc: Ming Lei, Andrew Morton, Michal Marek, Greg KH,
	Linux Kernel Mailing List, Vikram Mulukutla, Stephen Boyd, zohar,
	Mark Brown, Takashi Iwai, Johannes Berg, Christian Lamparter,
	hauke, Josh Boyer, Dmitry Torokhov, David Woodhouse, jslaby,
	Linus Torvalds, Andy Lutomirski, Wu Fengguang, rpurdie,
	j.anaszewski, Abhay_Salunke, Julia Lawall, Gilles.Muller,
	nicolas.palix, Tom Gundersen, David Howells, Alessandro Rubini,
	Kevin Cernekee, Kees Cook, Jonathan Corbet, Thierry Martinez,
	cocci, linux-serial, linux-doc, linuxppc-dev

On Wed, Aug 24, 2016 at 10:39 PM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> On Wed, Aug 24, 2016 at 08:55:55AM +0200, Daniel Vetter wrote:
>> On Fri, Jun 17, 2016 at 12:54 AM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
>> > Thou shalt not make firmware calls early on init or probe.
>
> <-- snip -->
>
>> > There are 4 offenders at this time:
>> >
>> > mcgrof@ergon ~/linux-next (git::20160609)$ export COCCI=scripts/coccinelle/api/request_firmware.cocci
>> > mcgrof@ergon ~/linux-next (git::20160609)$ make coccicheck MODE=report
>> >
>> > drivers/fmc/fmc-fakedev.c: ERROR: driver call request firmware call on its init routine on line 321.
>> > drivers/fmc/fmc-write-eeprom.c: ERROR: driver call request firmware call on its probe routine on line 136.
>> > drivers/tty/serial/rp2.c: ERROR: driver call request firmware call on its probe routine on line 796.
>> > drivers/tty/serial/ucc_uart.c: ERROR: driver call request firmware call on its probe routine on line 1246.
>>
>> Plus all gpu drivers which need firmware. And yes we must load them at
>> probe
>
> Do you have an upstream driver in mind that does this ? Is it on device
> drier module probe or a DRM subsystem specific probe call of some sort ?

i915 is the one I care about for obvious reasons ;-) It's all from the
pci device probe function, but nested really deeply.

>> because people are generally pissed when they boot their machine
>> and the screen goes black. On top of that a lot of people want their
>> gpu drivers to be built-in, but can't ship the firmware blobs in the
>> kernel image because gpl. Yep, there's a bit a contradiction there ...
>
> Can they use initramfs for this ?

Apparently that's also uncool with the embedded folks. Tbh I don't
know exactly why. Also I thought initramfs is available only after
built-in drivers have finished loading, but maybe that changed.

> Also just curious -- as with other subsystems, is it possible to load
> a generic driver first, say vesa, and then a more enhanced one later ?
> I am not saying this is ideal or am I suggesting this, I'd just like
> to know the feasibility of this.

Some users want a fully running gfx stack 2s after power-on. There's
not even enough time to load an uefi or vga driver first. i915
directly initializes the gpu from power-on state on those.

>> I think what would work is loading the different subsystems of the
>> driver in parallel (we already do that largely)
>
> Init level stuff is actually pretty synchronous, and in fact both
> init and probe are called serially. There are a few subsystems which
> have been doing things a bit differently, but these are exceptions.
>
> When you say we already do this largely, can you describe a bit more
> precisely what you mean ?

Oh, this isn't subsystems as in linux device/driver model, but
different parts within the driver. We fire up a bunch of struct work
to get various bits done asynchronously.

>>, and then if one
>> firmware blob isn't there yet simply stall that async worker until it
>> shows up.
>
> Is this an existing framework or do you mean if we add something
> generic to do this async loading of subsystems ?

normal workers, and flush_work and friends to sync up. We have some
really fancy ideas for essentially async init tasks that can declare
their depencies systemd-unit file style, but that's only in the
prototype stage. We need this (eventually) since handling the ordering
correctly is getting unwieldy. But again just struct work launched
from the main pci probe function.

>> But the answers I've gotten thus far from request_firmware()
>> folks (well at least Greg) is don't do that ...
>
> Well in this patch set I'm adding myself as a MAINTAINER and I've
> been extending the firmware API recently to help with a few new
> features I want, I've been wanting to hear more feedback from
> other's needs and I had actually not gotten much -- except
> only recently with the usermode helper and reasons why some
> folks thought they could not use direct firmware loading from
> the fs. I'm keen to hear or more use cases and needs specially if
> they have to do with improving boot time and asynchronous boot.
>
>> Is that problem still somewhere on the radar?
>
> Not mine.
>
>> Atm there's various
>> wait_for_rootfs hacks out there floating in vendor/product trees.
>
> This one I've heard about recently, and I suggested two possible
> solutions, with a preference to a simply notification of when
> rootfs is available from userspace.
>
>> "Avoid at all costs" sounds like upstream prefers to not care about
>> android/cros in those case (yes I know most arm socs don't need
>> firmware, which would make it a problem fo just be a subset of all
>> devices).
>
> In my days of dealing with Android I learned most folks did not frankly
> care too much about upstream-first model. That means things were reactive.
> That's a business mind set and that's fine. However for upstream we want
> what is best and to discuss. I haven't seen proposals so, so long as
> we just hear of "hacks" that some folks do in vendor/product trees,
> what can we do ?

One of the proposals which would have worked for us died a while back:

https://lkml.org/lkml/2014/6/15/47

That's essentially what I'd like to have.

[cut discussion about async probe]

> So .. I agree, let's avoid the hacks. Patches welcomed.

Hm, this is a definite change of tack - back when I discussed this
with Greg about 2 ks ago it sounded like "don't do this". The only
thing we need is some way to wait for rootfs before we do the
request_firmware. Everything else we handle already in the kernel.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v2 2/5] firmware: annotate thou shalt not request fw on init or probe
  2016-08-25 11:05       ` Daniel Vetter
@ 2016-08-25 19:41         ` Luis R. Rodriguez
  2016-08-25 20:10           ` Daniel Vetter
                             ` (2 more replies)
  0 siblings, 3 replies; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-08-25 19:41 UTC (permalink / raw)
  To: Daniel Vetter, Felix Fietkau, David Woodhouse,
	Jörg Rödel, Joerg Roedel, Roman Pen, Bjorn Andersson
  Cc: Luis R. Rodriguez, Ming Lei, Andrew Morton, Michal Marek,
	Greg KH, Linux Kernel Mailing List, Vikram Mulukutla,
	Stephen Boyd, zohar, Mark Brown, Takashi Iwai, Johannes Berg,
	Christian Lamparter, hauke, Josh Boyer, Dmitry Torokhov, jslaby,
	Linus Torvalds, Andy Lutomirski, Wu Fengguang, rpurdie,
	j.anaszewski, Abhay_Salunke, Julia Lawall, Gilles.Muller,
	nicolas.palix, Tom Gundersen, David Howells, Alessandro Rubini,
	Kevin Cernekee, Kees Cook, Jonathan Corbet, Thierry Martinez,
	cocci, linux-serial, linux-doc, linuxppc-dev

Summoning Felix for the embedded aspect on initramfs below.
Jörg might be interested in the async facilities you speak of as well.

On Thu, Aug 25, 2016 at 01:05:44PM +0200, Daniel Vetter wrote:
> On Wed, Aug 24, 2016 at 10:39 PM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> > On Wed, Aug 24, 2016 at 08:55:55AM +0200, Daniel Vetter wrote:
> >> On Fri, Jun 17, 2016 at 12:54 AM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> >> > Thou shalt not make firmware calls early on init or probe.
> >
> > <-- snip -->
> >
> >> > There are 4 offenders at this time:
> >> >
> >> > mcgrof@ergon ~/linux-next (git::20160609)$ export COCCI=scripts/coccinelle/api/request_firmware.cocci
> >> > mcgrof@ergon ~/linux-next (git::20160609)$ make coccicheck MODE=report
> >> >
> >> > drivers/fmc/fmc-fakedev.c: ERROR: driver call request firmware call on its init routine on line 321.
> >> > drivers/fmc/fmc-write-eeprom.c: ERROR: driver call request firmware call on its probe routine on line 136.
> >> > drivers/tty/serial/rp2.c: ERROR: driver call request firmware call on its probe routine on line 796.
> >> > drivers/tty/serial/ucc_uart.c: ERROR: driver call request firmware call on its probe routine on line 1246.
> >>
> >> Plus all gpu drivers which need firmware. And yes we must load them at
> >> probe
> >
> > Do you have an upstream driver in mind that does this ? Is it on device
> > drier module probe or a DRM subsystem specific probe call of some sort ?
> 
> i915 is the one I care about for obvious reasons ;-) It's all from the
> pci device probe function, but nested really deeply.

The above SmPL grammar should capture well nested calls of calls within probe,
so curious why it didn't pick up i915. Let's see.

i915_pci_probe() --> i915_driver_load() -->
	i915_load_modeset_init() --> (drivers/gpu/drm/i915/i915_drv.c)
		a) intel_csr_ucode_init() (drivers/gpu/drm/i915/intel_csr.c)
		...
		b) intel_guc_init() (drivers/gpu/drm/i915/intel_guc_loader.c)

The two firmwares come from:

a) intel_csr_ucode_init() --> schedule_work(&dev_priv->csr.work);
	csr_load_work_fn() --> request_firmware()

b) intel_guc_init() --> guc_fw_fetch() request_firmware()

---

a) is not dealt with given the grammar does not include scheduled work,
however using work to offload loading firmware seems reasonable here.

b) This should have been picked up, but its not. Upon closer inspection
   the grammar currently expects the module init and probe to be on the
   same file. Loosening this:

diff --git a/scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci b/scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
index cf180c59e042..e19e6d3dfc0f 100644
--- a/scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
+++ b/scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
@@ -49,7 +49,7 @@ identifier init;
 
 module_init(init);
 
-@ has_probe depends on defines_module_init@
+@ has_probe @
 identifier drv_calls, drv_probe;
 type bus_driver;
 identifier probe_op =~ "(probe)";
@@ -59,7 +59,7 @@ bus_driver drv_calls = {
 	.probe_op = drv_probe,
 };
 
-@hascall depends on !after_start && defines_module_init@
+@hascall depends on !after_start @
 position p;
 @@
 

I get a lot more complaints but still -- i915 b) case is not yet covered:

./drivers/bluetooth/ath3k.c: ERROR: driver call request firmware call on its probe routine on line 546.
./drivers/bluetooth/bcm203x.c: ERROR: driver call request firmware call on its probe routine on line 193.
./drivers/bluetooth/bcm203x.c: ERROR: driver call request firmware call on its probe routine on line 218.
./drivers/bluetooth/bfusb.c: ERROR: driver call request firmware call on its probe routine on line 655.
./drivers/fmc/fmc-fakedev.c: ERROR: driver call request firmware call on its init routine on line 321.
./drivers/fmc/fmc-write-eeprom.c: ERROR: driver call request firmware call on its probe routine on line 136.
./drivers/tty/serial/ucc_uart.c: ERROR: driver call request firmware call on its probe routine on line 1246.
./sound/soc/codecs/wm2000.c: ERROR: driver call request firmware call on its probe routine on line 893.
./sound/soc/sh/siu_dai.c: ERROR: driver call request firmware call on its probe routine on line 747.
./drivers/net/wireless/intersil/orinoco/orinoco_usb.c: ERROR: driver call request firmware call on its probe routine on line 1661.
./sound/soc/intel/common/sst-acpi.c: ERROR: driver call request firmware call on its probe routine on line 161.
./drivers/input/touchscreen/goodix.c: ERROR: driver call request firmware call on its probe routine on line 744.
./drivers/media/usb/go7007/go7007-loader.c: ERROR: driver call request firmware call on its probe routine on line 78.
./drivers/media/usb/go7007/go7007-loader.c: ERROR: driver call request firmware call on its probe routine on line 93.
./drivers/tty/serial/rp2.c: ERROR: driver call request firmware call on its probe routine on line 796.
./drivers/usb/misc/isight_firmware.c: ERROR: driver call request firmware call on its probe routine on line 50.
./drivers/usb/serial/mxuport.c: ERROR: driver call request firmware call on its probe routine on line 1067.
./sound/pci/hda/hda_intel.c: ERROR: driver call request firmware call on its probe routine on line 2015.
./drivers/media/usb/s2255/s2255drv.c: ERROR: driver call request firmware call on its probe routine on line 2304.
./drivers/misc/lattice-ecp3-config.c: ERROR: driver call request firmware call on its probe routine on line 206.
./drivers/net/wireless/ath/carl9170/usb.c: ERROR: driver call request firmware call on its probe routine on line 1103.
./drivers/video/fbdev/metronomefb.c: ERROR: driver call request firmware call on its probe routine on line 681.

At a quick glance these check out though, and it was because I loosened
the requirement to not necessarily have module_init -- usb for instance has
module_usb_driver()...

I believe i915 is not picked up given the request for firmware comes in
on separate files than both the module_init *and* probe calls, that is,
its implemented in drivers/gpu/drm/i915/intel_csr.c. Tuning this a bit more
can take a bit of time, but the intent stands -- as such I think the
current grammar is justified and we should evolve it to catch more and
as it does we touch base again to see what the implications are -- clearly
this means way many more drivers are requesting firmware on probe than
expected.

> >> because people are generally pissed when they boot their machine
> >> and the screen goes black. On top of that a lot of people want their
> >> gpu drivers to be built-in, but can't ship the firmware blobs in the
> >> kernel image because gpl. Yep, there's a bit a contradiction there ...
> >
> > Can they use initramfs for this ?
> 
> Apparently that's also uncool with the embedded folks.

What's uncool with embedded folks? To use initramfs for firmware ?
If so can you explain why ?

> Tbh I don't
> know exactly why. Also I thought initramfs is available only after
> built-in drivers have finished loading, but maybe that changed.

Its still the same, we run through all init calls first (do_initcalls()
and finally the initramfs is processed via prepare_namespace()

do_basic_setup() {
	...
	driver_init();
	...
	do_initcalls();
	...
}

kernel_init_freeable() {
	...
	do_basic_setup();
	...
	if (sys_access((const char __user *) ramdisk_execute_command, 0) != 0) {
		ramdisk_execute_command = NULL;
		prepare_namespace();
        }
}

Felix, are initramfs not kosher for embedded ? If so why not ? The question
here is where to stuff firmware if its needed early in boot and since
graphics folks want video turned on ASAP they often load firmware on probe
and due to licensing may not be able to use the built-in firmware facility.
Using initrafms is what would be expected for these cases then, if that's
not kosher what do you recommend ? Granted video may not be a concern for
embedded but since we're on topic figured we should square this away.

> > Also just curious -- as with other subsystems, is it possible to load
> > a generic driver first, say vesa, and then a more enhanced one later ?
> > I am not saying this is ideal or am I suggesting this, I'd just like
> > to know the feasibility of this.
> 
> Some users want a fully running gfx stack 2s after power-on. There's
> not even enough time to load an uefi or vga driver first. i915
> directly initializes the gpu from power-on state on those.

I see.. thanks.

> >> I think what would work is loading the different subsystems of the
> >> driver in parallel (we already do that largely)
> >
> > Init level stuff is actually pretty synchronous, and in fact both
> > init and probe are called serially. There are a few subsystems which
> > have been doing things a bit differently, but these are exceptions.
> >
> > When you say we already do this largely, can you describe a bit more
> > precisely what you mean ?
> 
> Oh, this isn't subsystems as in linux device/driver model, but
> different parts within the driver. We fire up a bunch of struct work
> to get various bits done asynchronously.

Thanks for the clarification.

> >>, and then if one
> >> firmware blob isn't there yet simply stall that async worker until it
> >> shows up.
> >
> > Is this an existing framework or do you mean if we add something
> > generic to do this async loading of subsystems ?
> 
> normal workers, and flush_work and friends to sync up. We have some
> really fancy ideas for essentially async init tasks that can declare
> their depencies systemd-unit file style, but that's only in the
> prototype stage.

Great, another solution. More reason to talk at KS/Plumbers to see
exactly what each different pipe dream is coming up with.

> We need this (eventually) since handling the ordering
> correctly is getting unwieldy. But again just struct work launched
> from the main pci probe function.

You mean its a struct work based solution ?

> >> But the answers I've gotten thus far from request_firmware()
> >> folks (well at least Greg) is don't do that ...
> >
> > Well in this patch set I'm adding myself as a MAINTAINER and I've
> > been extending the firmware API recently to help with a few new
> > features I want, I've been wanting to hear more feedback from
> > other's needs and I had actually not gotten much -- except
> > only recently with the usermode helper and reasons why some
> > folks thought they could not use direct firmware loading from
> > the fs. I'm keen to hear or more use cases and needs specially if
> > they have to do with improving boot time and asynchronous boot.
> >
> >> Is that problem still somewhere on the radar?
> >
> > Not mine.
> >
> >> Atm there's various
> >> wait_for_rootfs hacks out there floating in vendor/product trees.
> >
> > This one I've heard about recently, and I suggested two possible
> > solutions, with a preference to a simply notification of when
> > rootfs is available from userspace.
> >
> >> "Avoid at all costs" sounds like upstream prefers to not care about
> >> android/cros in those case (yes I know most arm socs don't need
> >> firmware, which would make it a problem fo just be a subset of all
> >> devices).
> >
> > In my days of dealing with Android I learned most folks did not frankly
> > care too much about upstream-first model. That means things were reactive.
> > That's a business mind set and that's fine. However for upstream we want
> > what is best and to discuss. I haven't seen proposals so, so long as
> > we just hear of "hacks" that some folks do in vendor/product trees,
> > what can we do ?
> 
> One of the proposals which would have worked for us died a while back:
> 
> https://lkml.org/lkml/2014/6/15/47

Interesting, the problem stated here, which comes from the fact (as clarified
above) the rootfs comes up only *after* we run do_initcalls() as such there are
possible theoretical races even with request_firmware_nowait() -- as such that
justifies even more the SmPL grammar rule to include even request_firmware_nowait()
on probe / init as this patch has.

Anyway yeah that patch has good intentions but its wrong for a slew of reasons.
The problem is the same as discussed with Bjorn recently. The solution
discussed is that since only userspace knows when the *real* rootfs is ready
(because of slew of reasons) the best we can do is have an event issued by
userspace to inform us of that.

> That's essentially what I'd like to have.

You're not alone.

> [cut discussion about async probe]
> 
> > So .. I agree, let's avoid the hacks. Patches welcomed.
> 
> Hm, this is a definite change of tack - back when I discussed this
> with Greg about 2 ks ago it sounded like "don't do this". The only
> thing we need is some way to wait for rootfs before we do the
> request_firmware. Everything else we handle already in the kernel.

OK so lets just get this userspace event done, and we're set.

  Luis

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v2 2/5] firmware: annotate thou shalt not request fw on init or probe
  2016-08-25 19:41         ` Luis R. Rodriguez
@ 2016-08-25 20:10           ` Daniel Vetter
  2016-08-25 20:25             ` Luis R. Rodriguez
  2016-08-25 20:30           ` Dmitry Torokhov
  2016-09-02 23:59           ` Luis R. Rodriguez
  2 siblings, 1 reply; 80+ messages in thread
From: Daniel Vetter @ 2016-08-25 20:10 UTC (permalink / raw)
  To: Luis R. Rodriguez, Chris Wilson
  Cc: Felix Fietkau, David Woodhouse, Jörg Rödel,
	Joerg Roedel, Roman Pen, Bjorn Andersson, Ming Lei,
	Andrew Morton, Michal Marek, Greg KH, Linux Kernel Mailing List,
	Vikram Mulukutla, Stephen Boyd, zohar, Mark Brown, Takashi Iwai,
	Johannes Berg, Christian Lamparter, hauke, Josh Boyer,
	Dmitry Torokhov, jslaby, Linus Torvalds, Andy Lutomirski,
	Wu Fengguang, rpurdie, j.anaszewski, Abhay_Salunke, Julia Lawall,
	Gilles.Muller, nicolas.palix, Tom Gundersen, David Howells,
	Alessandro Rubini, Kevin Cernekee, Kees Cook, Jonathan Corbet,
	Thierry Martinez, cocci, linux-serial, linux-doc, linuxppc-dev

Cutting down since a lot of this is probably better discussed at
ks/lpc. Aside, if you want to check out Chris Wilson's work on our new
depency handling, it's called kfence.

https://lkml.org/lkml/2016/7/17/37

On Thu, Aug 25, 2016 at 9:41 PM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
>> > So .. I agree, let's avoid the hacks. Patches welcomed.
>>
>> Hm, this is a definite change of tack - back when I discussed this
>> with Greg about 2 ks ago it sounded like "don't do this". The only
>> thing we need is some way to wait for rootfs before we do the
>> request_firmware. Everything else we handle already in the kernel.
>
> OK so lets just get this userspace event done, and we're set.

Ah great. As long as that special wait-for-rootfs-pls firmware request
is still allowed, i915&gfx folks will be happy. We will call it from
probe though ;-) but all from our own workers.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v2 2/5] firmware: annotate thou shalt not request fw on init or probe
  2016-08-25 20:10           ` Daniel Vetter
@ 2016-08-25 20:25             ` Luis R. Rodriguez
  0 siblings, 0 replies; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-08-25 20:25 UTC (permalink / raw)
  To: Daniel Vetter
  Cc: Luis R. Rodriguez, Chris Wilson, Felix Fietkau, David Woodhouse,
	Jörg Rödel, Joerg Roedel, Roman Pen, Bjorn Andersson,
	Ming Lei, Andrew Morton, Michal Marek, Greg KH,
	Linux Kernel Mailing List, Vikram Mulukutla, Stephen Boyd, zohar,
	Mark Brown, Takashi Iwai, Johannes Berg, Christian Lamparter,
	hauke, Josh Boyer, Dmitry Torokhov, jslaby, Linus Torvalds,
	Andy Lutomirski, Wu Fengguang, rpurdie, j.anaszewski,
	Abhay_Salunke, Julia Lawall, Gilles.Muller, nicolas.palix,
	Tom Gundersen, David Howells, Alessandro Rubini, Kevin Cernekee,
	Kees Cook, Jonathan Corbet, Thierry Martinez, cocci,
	linux-serial, linux-doc, linuxppc-dev

On Thu, Aug 25, 2016 at 10:10:52PM +0200, Daniel Vetter wrote:
> Cutting down since a lot of this is probably better discussed at
> ks/lpc. Aside, if you want to check out Chris Wilson's work on our new
> depency handling, it's called kfence.
> 
> https://lkml.org/lkml/2016/7/17/37

Thanks more reading.. :)

> On Thu, Aug 25, 2016 at 9:41 PM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> >> > So .. I agree, let's avoid the hacks. Patches welcomed.
> >>
> >> Hm, this is a definite change of tack - back when I discussed this
> >> with Greg about 2 ks ago it sounded like "don't do this". The only
> >> thing we need is some way to wait for rootfs before we do the
> >> request_firmware. Everything else we handle already in the kernel.
> >
> > OK so lets just get this userspace event done, and we're set.
> 
> Ah great. As long as that special wait-for-rootfs-pls firmware request
> is still allowed, i915&gfx folks will be happy. We will call it from
> probe though ;-) but all from our own workers.

We should strive for this to be transparent to drivers, ie, this safeguard of
looking for firmware should be something handled by the kernel as otherwise
we're just forcing a race condition given the review in the last thread.

  Luis

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v2 2/5] firmware: annotate thou shalt not request fw on init or probe
  2016-08-25 19:41         ` Luis R. Rodriguez
  2016-08-25 20:10           ` Daniel Vetter
@ 2016-08-25 20:30           ` Dmitry Torokhov
  2016-09-02 23:59           ` Luis R. Rodriguez
  2 siblings, 0 replies; 80+ messages in thread
From: Dmitry Torokhov @ 2016-08-25 20:30 UTC (permalink / raw)
  To: Luis R. Rodriguez
  Cc: Daniel Vetter, Felix Fietkau, David Woodhouse,
	Jörg Rödel, Joerg Roedel, Roman Pen, Bjorn Andersson,
	Ming Lei, Andrew Morton, Michal Marek, Greg KH,
	Linux Kernel Mailing List, Vikram Mulukutla, Stephen Boyd,
	Mimi Zohar, Mark Brown, Takashi Iwai, Johannes Berg,
	Christian Lamparter, Hauke Mehrtens, Josh Boyer, Jiri Slaby,
	Linus Torvalds, Andy Lutomirski, Wu Fengguang, Richard Purdie,
	Jacek Anaszewski, Abhay_Salunke, Julia Lawall, Gilles.Muller,
	nicolas.palix, Tom Gundersen, David Howells, Alessandro Rubini,
	Kevin Cernekee, Kees Cook, Jonathan Corbet, Thierry Martinez,
	cocci, linux-serial, linux-doc, linuxppc-dev

On Thu, Aug 25, 2016 at 12:41 PM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> On Thu, Aug 25, 2016 at 01:05:44PM +0200, Daniel Vetter wrote:
>> On Wed, Aug 24, 2016 at 10:39 PM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:

>> > Can they use initramfs for this ?
>>
>> Apparently that's also uncool with the embedded folks.
>
> What's uncool with embedded folks? To use initramfs for firmware ?
> If so can you explain why ?

Because it is not needed? If you are embedded you have an option of
only compiling drivers and services that you need directly into the
kernel, then initramfs is just an extra piece that complicates your
boot process.

Thanks.

-- 
Dmitry

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v3 5/5] firmware: fix fw cache to avoid usermode helper on suspend
  2016-08-24  0:45   ` [PATCH v3 5/5] firmware: fix fw cache to avoid usermode helper on suspend mcgrof
@ 2016-08-31  7:03     ` Daniel Wagner
  2016-09-02 18:13       ` Luis R. Rodriguez
  0 siblings, 1 reply; 80+ messages in thread
From: Daniel Wagner @ 2016-08-31  7:03 UTC (permalink / raw)
  To: mcgrof, ming.lei, akpm, gregkh
  Cc: daniel.wagner, mmarek, linux-kernel, markivx, stephen.boyd,
	zohar, broonie, tiwai, johannes, chunkeey, hauke, jwboyer,
	dmitry.torokhov, dwmw2, jslaby, torvalds, luto, fengguang.wu,
	rpurdie, j.anaszewski, Abhay_Salunke, Julia.Lawall,
	Gilles.Muller, nicolas.palix, teg, dhowells, bjorn.andersson,
	arend.vanspriel, kvalo

Hi Luis,

On 08/24/2016 02:45 AM, mcgrof@kernel.org wrote:
> diff --git a/drivers/base/firmware_class.c b/drivers/base/firmware_class.c
> index 22d1760a4278..dca4f9cbf4db 100644
> --- a/drivers/base/firmware_class.c
> +++ b/drivers/base/firmware_class.c
> @@ -1412,7 +1412,7 @@ static int cache_firmware(const char *fw_name)
>
>  	pr_debug("%s: %s\n", __func__, fw_name);
>
> -	ret = request_firmware(&fw, fw_name, NULL);
> +	ret = request_firmware_direct(&fw, fw_name, NULL);
>  	if (!ret)
>  		kfree(fw);
>
>

While you are at it you should kill this here as well:

@@ -1622,7 +1637,6 @@ static void __device_uncache_fw_images(void)
  static void device_cache_fw_images(void)
  {
  	struct firmware_cache *fwc = &fw_cache;
-	int old_timeout;
  	DEFINE_WAIT(wait);

  	pr_debug("%s\n", __func__);
@@ -1630,17 +1644,6 @@ static void device_cache_fw_images(void)
  	/* cancel uncache work */
  	cancel_delayed_work_sync(&fwc->work);

-	/*
-	 * use small loading timeout for caching devices' firmware
-	 * because all these firmware images have been loaded
-	 * successfully at lease once, also system is ready for
-	 * completing firmware loading now. The maximum size of
-	 * firmware in current distributions is about 2M bytes,
-	 * so 10 secs should be enough.
-	 */
-	old_timeout = loading_timeout;
-	loading_timeout = 10;
-
  	mutex_lock(&fw_lock);
  	fwc->state = FW_LOADER_START_CACHE;
  	dpm_for_each_dev(NULL, dev_cache_fw_image);
@@ -1648,8 +1651,6 @@ static void device_cache_fw_images(void)

  	/* wait for completion of caching firmware for all devices */
  	async_synchronize_full_domain(&fw_cache_domain);
-
-	loading_timeout = old_timeout;
  }

cheers,
daniel

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v3 5/5] firmware: fix fw cache to avoid usermode helper on suspend
  2016-08-31  7:03     ` Daniel Wagner
@ 2016-09-02 18:13       ` Luis R. Rodriguez
  0 siblings, 0 replies; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-09-02 18:13 UTC (permalink / raw)
  To: Daniel Wagner
  Cc: mcgrof, ming.lei, akpm, gregkh, daniel.wagner, mmarek,
	linux-kernel, markivx, stephen.boyd, zohar, broonie, tiwai,
	johannes, chunkeey, hauke, jwboyer, dmitry.torokhov, dwmw2,
	jslaby, torvalds, luto, fengguang.wu, rpurdie, j.anaszewski,
	Abhay_Salunke, Julia.Lawall, Gilles.Muller, nicolas.palix, teg,
	dhowells, bjorn.andersson, arend.vanspriel, kvalo

On Wed, Aug 31, 2016 at 09:03:53AM +0200, Daniel Wagner wrote:
> Hi Luis,
> 
> On 08/24/2016 02:45 AM, mcgrof@kernel.org wrote:
> >diff --git a/drivers/base/firmware_class.c b/drivers/base/firmware_class.c
> >index 22d1760a4278..dca4f9cbf4db 100644
> >--- a/drivers/base/firmware_class.c
> >+++ b/drivers/base/firmware_class.c
> >@@ -1412,7 +1412,7 @@ static int cache_firmware(const char *fw_name)
> >
> > 	pr_debug("%s: %s\n", __func__, fw_name);
> >
> >-	ret = request_firmware(&fw, fw_name, NULL);
> >+	ret = request_firmware_direct(&fw, fw_name, NULL);
> > 	if (!ret)
> > 		kfree(fw);
> >
> >
> 
> While you are at it you should kill this here as well:
> 
> @@ -1622,7 +1637,6 @@ static void __device_uncache_fw_images(void)
>  static void device_cache_fw_images(void)
>  {
>  	struct firmware_cache *fwc = &fw_cache;
> -	int old_timeout;
>  	DEFINE_WAIT(wait);
> 
>  	pr_debug("%s\n", __func__);
> @@ -1630,17 +1644,6 @@ static void device_cache_fw_images(void)
>  	/* cancel uncache work */
>  	cancel_delayed_work_sync(&fwc->work);
> 
> -	/*
> -	 * use small loading timeout for caching devices' firmware
> -	 * because all these firmware images have been loaded
> -	 * successfully at lease once, also system is ready for
> -	 * completing firmware loading now. The maximum size of
> -	 * firmware in current distributions is about 2M bytes,
> -	 * so 10 secs should be enough.
> -	 */
> -	old_timeout = loading_timeout;
> -	loading_timeout = 10;
> -
>  	mutex_lock(&fw_lock);
>  	fwc->state = FW_LOADER_START_CACHE;
>  	dpm_for_each_dev(NULL, dev_cache_fw_image);
> @@ -1648,8 +1651,6 @@ static void device_cache_fw_images(void)
> 
>  	/* wait for completion of caching firmware for all devices */
>  	async_synchronize_full_domain(&fw_cache_domain);
> -
> -	loading_timeout = old_timeout;

You're right, thanks!

  Luis

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v3 2/5] firmware: annotate thou shalt not request fw on init or probe
  2016-08-24  8:17     ` Gabriel Paubert
@ 2016-09-02 18:26       ` Luis R. Rodriguez
  0 siblings, 0 replies; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-09-02 18:26 UTC (permalink / raw)
  To: Gabriel Paubert
  Cc: mcgrof, ming.lei, akpm, gregkh, dmitry.torokhov, linux-doc,
	Gilles.Muller, tiwai, Daniel Vetter, Alessandro Rubini,
	stephen.boyd, bjorn.andersson, teg, chunkeey, cocci, jwboyer,
	Jonathan Corbet, Kevin Cernekee, Thierry Martinez, linux-serial,
	jslaby, zohar, arend.vanspriel, linuxppc-dev, Kees Cook, hauke,
	nicolas.palix, Abhay_Salunke, Julia.Lawall, broonie,
	j.anaszewski, kvalo, dhowells, dwmw2, markivx, daniel.wagner,
	linux-kernel, luto, mmarek, rpurdie, johannes, fengguang.wu,
	torvalds

On Wed, Aug 24, 2016 at 10:17:38AM +0200, Gabriel Paubert wrote:
> On Tue, Aug 23, 2016 at 05:45:04PM -0700, mcgrof@kernel.org wrote:
> 
> [snip]
> > ---
> >  Documentation/firmware_class/README                |  20 ++++
> >  drivers/base/Kconfig                               |   2 +-
> >  .../request_firmware-avoid-init-probe-init.cocci   | 130 +++++++++++++++++++++
> >  3 files changed, 151 insertions(+), 1 deletion(-)
> >  create mode 100644 scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
> > 
> > diff --git a/Documentation/firmware_class/README b/Documentation/firmware_class/README
> > index cafdca8b3b15..056d1cb9d365 100644
> > --- a/Documentation/firmware_class/README
> > +++ b/Documentation/firmware_class/README
> > @@ -93,6 +93,26 @@
> >     user contexts to request firmware asynchronously, but can't be called
> >     in atomic contexts.
> >  
> > +Requirements:
> > +=============
> > +
> > +You should avoid at all costs requesting firmware on both init and probe paths
> > +of your device driver. Reason for this is the complexity needed to ensure a
> > +firmware will be available for a driver early in boot through different
> > +build configurations. Consider built-in drivers needing firmware early, or
> > +consider a driver assuming it will only get firmware after pivot_root().
> > +
> > +Drivers that really need firmware early should use stuff the firmware in
> 
> Minor grammatical nit: s/use//

Fixed, thanks.

  Luis

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v2 2/5] firmware: annotate thou shalt not request fw on init or probe
  2016-08-25 19:41         ` Luis R. Rodriguez
  2016-08-25 20:10           ` Daniel Vetter
  2016-08-25 20:30           ` Dmitry Torokhov
@ 2016-09-02 23:59           ` Luis R. Rodriguez
  2016-09-03  0:20             ` [RFC] fs: add userspace critical mounts event support Luis R. Rodriguez
  2 siblings, 1 reply; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-09-02 23:59 UTC (permalink / raw)
  To: Luis R. Rodriguez
  Cc: Daniel Vetter, Felix Fietkau, David Woodhouse,
	Jörg Rödel, Joerg Roedel, Roman Pen, Bjorn Andersson,
	Ming Lei, Andrew Morton, Michal Marek, Greg KH,
	Linux Kernel Mailing List, Vikram Mulukutla, Stephen Boyd, zohar,
	Mark Brown, Takashi Iwai, Johannes Berg, Christian Lamparter,
	hauke, Josh Boyer, Dmitry Torokhov, jslaby, Linus Torvalds,
	Andy Lutomirski, Wu Fengguang, rpurdie, j.anaszewski,
	Abhay_Salunke, Julia Lawall, Gilles.Muller, nicolas.palix,
	Tom Gundersen, David Howells, Alessandro Rubini, Kevin Cernekee,
	Kees Cook, Jonathan Corbet, Thierry Martinez, cocci,
	linux-serial, linux-doc, linuxppc-dev

On Thu, Aug 25, 2016 at 09:41:33PM +0200, Luis R. Rodriguez wrote:
> On Thu, Aug 25, 2016 at 01:05:44PM +0200, Daniel Vetter wrote:
> > On Wed, Aug 24, 2016 at 10:39 PM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> > Some users want a fully running gfx stack 2s after power-on. There's
> > not even enough time to load an uefi or vga driver first. i915
> > directly initializes the gpu from power-on state on those.
> 
> I see.. thanks.
> 
> > >> I think what would work is loading the different subsystems of the
> > >> driver in parallel (we already do that largely)
> > >
> > > Init level stuff is actually pretty synchronous, and in fact both
> > > init and probe are called serially. There are a few subsystems which
> > > have been doing things a bit differently, but these are exceptions.
> > >
> > > When you say we already do this largely, can you describe a bit more
> > > precisely what you mean ?
> > 
> > Oh, this isn't subsystems as in linux device/driver model, but
> > different parts within the driver. We fire up a bunch of struct work
> > to get various bits done asynchronously.
> 
> Thanks for the clarification.

<-- snip -->

> > One of the proposals which would have worked for us died a while back:
> > 
> > https://lkml.org/lkml/2014/6/15/47
> 
> Interesting, the problem stated here, which comes from the fact (as clarified
> above) the rootfs comes up only *after* we run do_initcalls() as such there are
> possible theoretical races even with request_firmware_nowait() -- as such that
> justifies even more the SmPL grammar rule to include even request_firmware_nowait()
> on probe / init as this patch has.
> 
> Anyway yeah that patch has good intentions but its wrong for a slew of reasons.
> The problem is the same as discussed with Bjorn recently. The solution
> discussed is that since only userspace knows when the *real* rootfs is ready
> (because of slew of reasons) the best we can do is have an event issued by
> userspace to inform us of that.

OK I have something I think we can build upon for this. Will send RFC.

  Luis

^ permalink raw reply	[flat|nested] 80+ messages in thread

* [RFC] fs: add userspace critical mounts event support
  2016-09-02 23:59           ` Luis R. Rodriguez
@ 2016-09-03  0:20             ` Luis R. Rodriguez
  2016-09-03  4:11               ` Linus Torvalds
  2016-09-14  2:38               ` Rob Landley
  0 siblings, 2 replies; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-09-03  0:20 UTC (permalink / raw)
  To: Daniel Vetter, Mimi Zohar, Felix Fietkau, David Woodhouse,
	Roman Pen, Bjorn Andersson, Ming Lei, Andrew Morton,
	Michal Marek, Greg KH
  Cc: Linux Kernel Mailing List, Vikram Mulukutla, Stephen Boyd,
	Mark Brown, Takashi Iwai, Johannes Berg, Christian Lamparter,
	hauke, Josh Boyer, Dmitry Torokhov, jslaby, Linus Torvalds,
	Andy Lutomirski, Wu Fengguang, rpurdie, Jeff Mahoney,
	j.anaszewski, Abhay_Salunke, Julia Lawall, Gilles.Muller,
	nicolas.palix, Tom Gundersen, Kay Sievers, David Howells,
	Alessandro Rubini, Kevin Cernekee, Kees Cook, Jonathan Corbet,
	Thierry Martinez, cocci, linux-serial, linux-doc, linuxppc-dev

kernel_read_file_from_path() can try to read a file from
the system's filesystem. This is typically done for firmware
for instance, which lives in /lib/firmware. One issue with
this is that the kernel cannot know for sure when the real
final /lib/firmare/ is ready, and even if you use initramfs
drivers are currently initialized *first* prior to the initramfs
kicking off. During init we run through all init calls first
(do_initcalls()) and finally the initramfs is processed via
prepare_namespace():

do_basic_setup() {
   ...
   driver_init();
   ...
   do_initcalls();
   ...
}

kernel_init_freeable() {
   ...
   do_basic_setup();
   ...
   if (sys_access((const char __user *) ramdisk_execute_command, 0) != 0) {
      ramdisk_execute_command = NULL;
      prepare_namespace();
   }
}

This leaves a possible race between loading drivers and any uses
of kernel_read_file_from_path(). Because pivot_root() can be used,
this allows userspace further possibilities in terms of defining
when a kernel critical filesystem should be ready by.

We define kernel critical filesystems as filesystems which the
kernel needs for kernel_read_file_from_path(). Since only userspace
can know when kernel critical filesystems are mounted and ready,
let userspace notify the kernel of this, and enable a new kernel
configuration which lets the kernel wait for this event before
enabling reads from kernel_read_file_from_path().

A default timeout of 10s is used for now. You can override this
through the kernel-parameters using critical_mounts_timeout_ms=T
where T is in ms. cat /sys/kernel/critical_mounts_timeout_ms the
current system value.

When userspace is ready it can simply:

  echo 1 > /sys/kernel/critical_mounts_ready

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---

Note, this still leaves the puzzle of the fact that initramfs may carry
some firmware, and some drivers may be OK in using firmware from there,
the wait stuff would just get in the way. To address this I think can
perhaps instead check *one* for the file, and if its present immediately
give it back, we'd only resort to the wait in cases of failure.

Another thing -- other than firmware we have:

security/integrity/ima/ima_fs.c:        rc = kernel_read_file_from_path(path, &data, &size, 0, READING_POLICY);
sound/oss/sound_firmware.h:     err = kernel_read_file_from_path((char *)fn, (void **)fp, &size,

What paths are these? So we can document the current uses in the Kconfig
at least.

Thoughts ?

 Documentation/kernel-parameters.txt |  6 +++
 drivers/base/Kconfig                | 48 +++++++++++++++++++++++
 fs/exec.c                           |  3 ++
 include/linux/fs.h                  |  8 ++++
 kernel/ksysfs.c                     | 77 +++++++++++++++++++++++++++++++++++++
 5 files changed, 142 insertions(+)

diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index 8ccacc44622a..1af89faa9fc9 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -849,6 +849,12 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 			It will be ignored when crashkernel=X,high is not used
 			or memory reserved is below 4G.
 
+	critical_mounts_timeout_ms=T	[KNL] timeout in ms
+			Format: <integer>
+			Use this to override the kernel's default timeout for
+			waiting for critical system mount points to become
+			available.
+
 	cryptomgr.notests
                         [KNL] Disable crypto self-tests
 
diff --git a/drivers/base/Kconfig b/drivers/base/Kconfig
index 12b4f5551501..21576c0a4898 100644
--- a/drivers/base/Kconfig
+++ b/drivers/base/Kconfig
@@ -25,6 +25,54 @@ config UEVENT_HELPER_PATH
 	  via /proc/sys/kernel/hotplug or via /sys/kernel/uevent_helper
 	  later at runtime.
 
+config CRITICAL_MOUNTS_WAIT
+	bool "Enable waiting for critical-filesystems-ready notification"
+	default n
+	help
+	  Kernel subsystems and device drivers often need to read files
+	  from the filesystem, however in doing this races are possible at
+	  bootup -- the subsystem requesting the file might look for it in /
+	  early in boot, but if we haven't yet mounted the real root
+	  filesystem we'll just tell the subsystem the file is not present and
+	  it will fail. Furthermore what path to the filesystem is used varies
+	  depending on the subsystem. To help the kernel we provide the option
+	  to let the kernel wait for all critical filesystems to mounted and
+	  ready before letting the kernel start trying to read files from the
+	  systems' filesystem. Since pivot_root() can be used and therefore a
+	  system might be configured to change its / filesystem at bootup as
+	  many times as it wishes, only userspace can realy know exactly when
+	  all critical filesystems are ready. Enabling this lets userspace
+	  communicate to the kernel when all critical filesystems are ready.
+
+	  What are the critical filesystems are obviously system specific, but
+	  examples of some are:
+
+	    o /lib/firmware/
+	    o /etc/XXX/
+
+	  If you enable this you must have a userspace init script or tool
+	  which will vet to ensure all critical filesystems are ready, once
+	  they are all ready it will inform the kenrel by setting the file
+	  /sys/kernel/critical_mounts_ready to 1.
+
+	  The kernel will wait by default 10 seconds for the event, if the
+	  the timeout is reached, it will proceed to just try to enable
+	  reading of the files from the kernel but warn.
+
+	  If not sure say "no" for now. You need proper userpace implementation
+	  for this.
+
+config CRITICAL_MOUNTS_WAIT_TIMEOUT
+	int "Timeout for critical-fs-reayd notification in miliseconds"
+	depends on CRITICAL_MOUNTS_WAIT
+	default 10000
+	help
+	  Defines the timeout for the kernel to wait for critical filesystems
+	  to be loaded. This if system specific as only the system will know
+	  exaclty when how long this typically takes. By default this is
+	  10 seconds. You can override at boot time by using the kernel
+	  parameter critical_mounts_timeout_ms.
+
 config DEVTMPFS
 	bool "Maintain a devtmpfs filesystem to mount at /dev"
 	help
diff --git a/fs/exec.c b/fs/exec.c
index 6fcfb3f7b137..0d46ad4aad11 100644
--- a/fs/exec.c
+++ b/fs/exec.c
@@ -57,6 +57,7 @@
 #include <linux/oom.h>
 #include <linux/compat.h>
 #include <linux/vmalloc.h>
+#include <linux/swait.h>
 
 #include <asm/uaccess.h>
 #include <asm/mmu_context.h>
@@ -949,6 +950,8 @@ int kernel_read_file_from_path(char *path, void **buf, loff_t *size,
 	struct file *file;
 	int ret;
 
+	wait_for_critical_mounts(id);
+
 	if (!path || !*path)
 		return -EINVAL;
 
diff --git a/include/linux/fs.h b/include/linux/fs.h
index bd57feb7cf37..f59213ac8a8b 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -3202,4 +3202,12 @@ static inline bool dir_relax_shared(struct inode *inode)
 extern bool path_noexec(const struct path *path);
 extern void inode_nohighmem(struct inode *inode);
 
+#ifdef CONFIG_CRITICAL_MOUNTS_WAIT
+void wait_for_critical_mounts(enum kernel_read_file_id id);
+#else
+static inline void wait_for_critical_mounts(enum kernel_read_file_id id)
+{
+}
+#endif /* CONFIG_CRITICAL_MOUNTS_WAIT */
+
 #endif /* _LINUX_FS_H */
diff --git a/kernel/ksysfs.c b/kernel/ksysfs.c
index ee1bc1bb8feb..232af58d8760 100644
--- a/kernel/ksysfs.c
+++ b/kernel/ksysfs.c
@@ -21,6 +21,7 @@
 #include <linux/compiler.h>
 
 #include <linux/rcupdate.h>	/* rcu_expedited and rcu_normal */
+#include <linux/swait.h>
 
 #define KERNEL_ATTR_RO(_name) \
 static struct kobj_attribute _name##_attr = __ATTR_RO(_name)
@@ -180,6 +181,78 @@ static ssize_t rcu_normal_store(struct kobject *kobj,
 KERNEL_ATTR_RW(rcu_normal);
 #endif /* #ifndef CONFIG_TINY_RCU */
 
+#ifdef CONFIG_CRITICAL_MOUNTS_WAIT
+static int are_critical_mounts_ready;
+
+static DECLARE_SWAIT_QUEUE_HEAD(critical_wq);
+static int critical_mounts_timeout_ms = CONFIG_CRITICAL_MOUNTS_WAIT_TIMEOUT;
+
+core_param(critical_mounts_timeout_ms, critical_mounts_timeout_ms, int, 0644);
+
+static bool critical_mounts_ready(void)
+{
+	return !!are_critical_mounts_ready;
+}
+
+
+static void __wait_for_critical_mounts(void)
+{
+	int ret;
+	struct swait_queue_head *wq = &critical_wq;
+
+	pr_debug("Waiting for critical filesystems...\n");
+	ret = swait_event_interruptible_timeout(*wq, critical_mounts_ready(),
+						msecs_to_jiffies(critical_mounts_timeout_ms));
+	if (ret > 0)
+		return;
+
+	WARN_ON(ret < 0);
+}
+static ssize_t critical_mounts_ready_show(struct kobject *kobj,
+					  struct kobj_attribute *attr,
+					  char *buf)
+{
+	return sprintf(buf, "%d\n", critical_mounts_ready());
+}
+static ssize_t critical_mounts_ready_store(struct kobject *kobj,
+					   struct kobj_attribute *attr,
+					   const char *buf, size_t count)
+{
+	if (kstrtoint(buf, 0, &are_critical_mounts_ready))
+		return -EINVAL;
+
+	return count;
+}
+KERNEL_ATTR_RW(critical_mounts_ready);
+
+static ssize_t critical_mounts_timeout_ms_show(struct kobject *kobj,
+					       struct kobj_attribute *attr,
+					       char *buf)
+{
+	return sprintf(buf, "%d\n", critical_mounts_timeout_ms);
+}
+KERNEL_ATTR_RO(critical_mounts_timeout_ms);
+
+void wait_for_critical_mounts(enum kernel_read_file_id id)
+{
+	switch (id) {
+	case READING_FIRMWARE:
+	case READING_FIRMWARE_PREALLOC_BUFFER:
+	case READING_POLICY:
+		if (!critical_mounts_ready()) {
+			pr_info("Waiting for critical filesystems...\n");
+			__wait_for_critical_mounts();
+		}
+		else
+			pr_info("All critical filesystems are ready!\n");
+		break;
+	default:
+		break;
+	}
+}
+EXPORT_SYMBOL_GPL(wait_for_critical_mounts);
+#endif /* CONFIG_CRITICAL_MOUNTS_WAIT */
+
 /*
  * Make /sys/kernel/notes give the raw contents of our kernel .notes section.
  */
@@ -225,6 +298,10 @@ static struct attribute * kernel_attrs[] = {
 	&rcu_expedited_attr.attr,
 	&rcu_normal_attr.attr,
 #endif
+#ifdef CONFIG_CRITICAL_MOUNTS_WAIT
+	&critical_mounts_ready_attr.attr,
+	&critical_mounts_timeout_ms_attr.attr,
+#endif
 	NULL
 };
 
-- 
2.9.2

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-09-03  0:20             ` [RFC] fs: add userspace critical mounts event support Luis R. Rodriguez
@ 2016-09-03  4:11               ` Linus Torvalds
  2016-09-03  4:20                 ` Dmitry Torokhov
  2016-09-06 17:46                 ` Bjorn Andersson
  2016-09-14  2:38               ` Rob Landley
  1 sibling, 2 replies; 80+ messages in thread
From: Linus Torvalds @ 2016-09-03  4:11 UTC (permalink / raw)
  To: Luis R. Rodriguez
  Cc: Daniel Vetter, Mimi Zohar, Felix Fietkau, David Woodhouse,
	Roman Pen, Bjorn Andersson, Ming Lei, Andrew Morton,
	Michal Marek, Greg KH, Linux Kernel Mailing List,
	Vikram Mulukutla, Stephen Boyd, Mark Brown, Takashi Iwai,
	Johannes Berg, Christian Lamparter, Hauke Mehrtens, Josh Boyer,
	Dmitry Torokhov, Jiri Slaby, Andy Lutomirski, Wu Fengguang,
	Richard Purdie, Jeff Mahoney, Jacek Anaszewski, Abhay_Salunke,
	Julia Lawall, Gilles.Muller, nicolas.palix, Tom Gundersen,
	Kay Sievers, David Howells, Alessandro Rubini, Kevin Cernekee,
	Kees Cook, Jonathan Corbet, Thierry Martinez, cocci,
	linux-serial, open list:DOCUMENTATION, linuxppc-dev

On Fri, Sep 2, 2016 at 5:20 PM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
>
> Thoughts ?

I really think this is a horrible hack.

It's basically the kernel giving up, and relying on user space to give
a single flag, and it's broken nasty crap.  Worse, it's broken nasty
crap with a user interface, so we'll be stuck with it forever. Please
no.

What are the drivers that need this, and why can't those drivers just
be fixed to not do braindead things?

                    Linus

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-09-03  4:11               ` Linus Torvalds
@ 2016-09-03  4:20                 ` Dmitry Torokhov
       [not found]                   ` <CA+55aFz4q5peXAeY9h8o3he7R=wXrBSYkOjMM9TehOw=pPoS+Q@mail.gmail.com>
  2016-09-06 17:46                 ` Bjorn Andersson
  1 sibling, 1 reply; 80+ messages in thread
From: Dmitry Torokhov @ 2016-09-03  4:20 UTC (permalink / raw)
  To: Linus Torvalds
  Cc: Luis R. Rodriguez, Daniel Vetter, Mimi Zohar, Felix Fietkau,
	David Woodhouse, Roman Pen, Bjorn Andersson, Ming Lei,
	Andrew Morton, Michal Marek, Greg KH, Linux Kernel Mailing List,
	Vikram Mulukutla, Stephen Boyd, Mark Brown, Takashi Iwai,
	Johannes Berg, Christian Lamparter, Hauke Mehrtens, Josh Boyer,
	Jiri Slaby, Andy Lutomirski, Wu Fengguang, Richard Purdie,
	Jeff Mahoney, Jacek Anaszewski, Abhay_Salunke, Julia Lawall,
	Gilles.Muller, nicolas.palix, Tom Gundersen, Kay Sievers,
	David Howells, Alessandro Rubini, Kevin Cernekee, Kees Cook,
	Jonathan Corbet, Thierry Martinez, cocci, linux-serial,
	open list:DOCUMENTATION, linuxppc-dev

On Fri, Sep 2, 2016 at 9:11 PM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
> On Fri, Sep 2, 2016 at 5:20 PM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
>>
>> Thoughts ?
>
> I really think this is a horrible hack.
>
> It's basically the kernel giving up, and relying on user space to give
> a single flag, and it's broken nasty crap.  Worse, it's broken nasty
> crap with a user interface, so we'll be stuck with it forever. Please
> no.

I agree that interface is bad, but I do believe we need something like this...

>
> What are the drivers that need this, and why can't those drivers just
> be fixed to not do braindead things?

Like what? Some devices do need to have firmware loaded so we know
their capabilities, so we really can't push the firmware loading into
"open". If your touch controller for some reason decided to crap over
it's nvram and comes in bootloader mode it is nice for it to slurp in
config data or firmware so use does not have to trigger update
manually. And while the controller is in bootloader mode we can't
create input device because we do not know what capabilities to
declare.

These devices we want to probe asynchronously and simply tell firmware
loader to wait for firmware to become available. The problem we do not
know when to give up, since we do not know where the firmware might
be. But userspace knows and can tel us.

Thanks.

-- 
Dmitry

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
       [not found]                   ` <CA+55aFz4q5peXAeY9h8o3he7R=wXrBSYkOjMM9TehOw=pPoS+Q@mail.gmail.com>
@ 2016-09-03 17:49                     ` Dmitry Torokhov
  2016-09-03 18:01                       ` Linus Torvalds
  0 siblings, 1 reply; 80+ messages in thread
From: Dmitry Torokhov @ 2016-09-03 17:49 UTC (permalink / raw)
  To: Linus Torvalds
  Cc: open list:DOCUMENTATION, cocci, Jacek Anaszewski,
	David Woodhouse, Christian Lamparter, Julia Lawall,
	Andrew Morton, linuxppc-dev, Mimi Zohar, Andy Lutomirski,
	Richard Purdie, Wu Fengguang, Johannes Berg, Luis R. Rodriguez,
	Michal Marek, Hauke Mehrtens, Mark Brown, Jiri Slaby, Ming Lei,
	Daniel Vetter, Kevin Cernekee, Jeff Mahoney, Greg KH,
	Bjorn Andersson, Jonathan Corbet, Felix Fietkau, David Howells,
	Vikram Mulukutla, Alessandro Rubini, Tom Gundersen, Kees Cook,
	Takashi Iwai, Gilles.Muller, linux-serial, Roman Pen,
	Kay Sievers, Stephen Boyd, nicolas.palix, Abhay_Salunke,
	Linux Kernel Mailing List, Thierry Martinez, Josh Boyer

On Fri, Sep 02, 2016 at 09:41:18PM -0700, Linus Torvalds wrote:
> On Sep 2, 2016 9:20 PM, "Dmitry Torokhov" <dmitry.torokhov@gmail.com> wrote:
> >
> > Like what? Some devices do need to have firmware loaded so we know
> > their capabilities, so we really can't push the firmware loading into
> > "open".
> 
> So you
> (a) document that

Document that device may come up half-broken? Not sure how that would
help end user.

> (b) make the driver only build as a module

Unfortunately module loading and availability of firmware is very
loosely coupled. Of course, if you only load modules from the same
partition that your firmware is on you can get away with it, but if some
of the modules are in initramfs and firmware is on final root fs then
it still does not work. And populating also initramfs with firmware that
might be used once in a 1000 boots is somewhat wasteful. That is not
talking about systems that do not wish to use modules for one reason or
another, or even more esoteric setups where non-essential for boot
firmware can be mounted later over nfs, etc, etc.

> (c) make sure the module and the firmware go together

I do not think it is always possible. Quite often it is though, at the
expense of increasing kernel/initramfs size.

> 
> End of problem.
> 
> Why make up random interfaces for crazy stuff?

Because we want a solution that works well for all cases, simple and
complex. This includes allowing drivers to be built into the kernel but
allow them waiting for additional data (config/firmware) that may become
available later in the game. We just need to be able to tell them when
it does not make sense to wait anymore as the data they want is not
coming, and do it more reliably then simply declaring 10 or 30 or 300
seconds time out.

Thanks.

-- 
Dmitry

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-09-03 17:49                     ` Dmitry Torokhov
@ 2016-09-03 18:01                       ` Linus Torvalds
  2016-09-03 18:10                         ` Dmitry Torokhov
  0 siblings, 1 reply; 80+ messages in thread
From: Linus Torvalds @ 2016-09-03 18:01 UTC (permalink / raw)
  To: Dmitry Torokhov
  Cc: open list:DOCUMENTATION, cocci, Jacek Anaszewski,
	David Woodhouse, Christian Lamparter, Julia Lawall,
	Andrew Morton, linuxppc-dev, Mimi Zohar, Andy Lutomirski,
	Richard Purdie, Wu Fengguang, Johannes Berg, Luis R. Rodriguez,
	Michal Marek, Hauke Mehrtens, Mark Brown, Jiri Slaby, Ming Lei,
	Daniel Vetter, Kevin Cernekee, Jeff Mahoney, Greg KH,
	Bjorn Andersson, Jonathan Corbet, Felix Fietkau, David Howells,
	Vikram Mulukutla, Alessandro Rubini, Tom Gundersen, Kees Cook,
	Takashi Iwai, Gilles.Muller, linux-serial, Roman Pen,
	Kay Sievers, Stephen Boyd, nicolas.palix, Abhay_Salunke,
	Linux Kernel Mailing List, Thierry Martinez, Josh Boyer

On Sat, Sep 3, 2016 at 10:49 AM, Dmitry Torokhov
<dmitry.torokhov@gmail.com> wrote:
>
> Unfortunately module loading and availability of firmware is very
> loosely coupled.

The whole "let's add a new magical proc entry to say that all
filesystems are mounted" is all about the user space coupling them.

I'm just saying that if user space must know about when firmware is
ready to be loaded anyway, just do it rigth. Not with some hacky "you
can now do random things" flag. But by having user space actually say
"put this module together with this firmware".

If you just put the two pieces together, then the module "will just work".

And quite frankly, that sounds like a much better maintenance practice
anyway. If some module doesn't work without the firmware, then dammit,
the two *should* go together. Putting them in two different places
would be *INSANE*.

                    Linus

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-09-03 18:01                       ` Linus Torvalds
@ 2016-09-03 18:10                         ` Dmitry Torokhov
  2016-09-06 21:52                           ` Luis R. Rodriguez
  2016-09-24  1:37                           ` Herbert, Marc
  0 siblings, 2 replies; 80+ messages in thread
From: Dmitry Torokhov @ 2016-09-03 18:10 UTC (permalink / raw)
  To: Linus Torvalds
  Cc: open list:DOCUMENTATION, cocci, Jacek Anaszewski,
	David Woodhouse, Christian Lamparter, Julia Lawall,
	Andrew Morton, linuxppc-dev, Mimi Zohar, Andy Lutomirski,
	Richard Purdie, Wu Fengguang, Johannes Berg, Luis R. Rodriguez,
	Michal Marek, Hauke Mehrtens, Mark Brown, Jiri Slaby, Ming Lei,
	Daniel Vetter, Kevin Cernekee, Jeff Mahoney, Greg KH,
	Bjorn Andersson, Jonathan Corbet, Felix Fietkau, David Howells,
	Vikram Mulukutla, Alessandro Rubini, Tom Gundersen, Kees Cook,
	Takashi Iwai, Gilles.Muller, linux-serial, Roman Pen,
	Kay Sievers, Stephen Boyd, nicolas.palix, Abhay_Salunke,
	Linux Kernel Mailing List, Thierry Martinez, Josh Boyer

On Sat, Sep 3, 2016 at 11:01 AM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
> On Sat, Sep 3, 2016 at 10:49 AM, Dmitry Torokhov
> <dmitry.torokhov@gmail.com> wrote:
>>
>> Unfortunately module loading and availability of firmware is very
>> loosely coupled.
>
> The whole "let's add a new magical proc entry to say that all
> filesystems are mounted" is all about the user space coupling them.

I will be first to say that the proposed *implementation* is nowhere
near what should be accepted. I was thinking if we kernel could post
"conditions" (maybe simple stings) that it waits for, and userspace
could unlock these "conditions". One of them might be "firmware
available".

>
> I'm just saying that if user space must know about when firmware is
> ready to be loaded anyway, just do it rigth. Not with some hacky "you
> can now do random things" flag. But by having user space actually say
> "put this module together with this firmware".
>
> If you just put the two pieces together, then the module "will just work".
>
> And quite frankly, that sounds like a much better maintenance practice
> anyway. If some module doesn't work without the firmware, then dammit,
> the two *should* go together. Putting them in two different places
> would be *INSANE*.

Quite often it does until it does not. Most of the touch controllers
work just fine until some event (abrupt cutting of power for example)
where nvram gets corrupted and they come up in bootloader mode. It is
just an example.

Thanks.

-- 
Dmitry

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-09-03  4:11               ` Linus Torvalds
  2016-09-03  4:20                 ` Dmitry Torokhov
@ 2016-09-06 17:46                 ` Bjorn Andersson
  2016-09-06 18:32                   ` Linus Torvalds
  1 sibling, 1 reply; 80+ messages in thread
From: Bjorn Andersson @ 2016-09-06 17:46 UTC (permalink / raw)
  To: Linus Torvalds
  Cc: Luis R. Rodriguez, Daniel Vetter, Mimi Zohar, Felix Fietkau,
	David Woodhouse, Roman Pen, Ming Lei, Andrew Morton,
	Michal Marek, Greg KH, Linux Kernel Mailing List,
	Vikram Mulukutla, Stephen Boyd, Mark Brown, Takashi Iwai,
	Johannes Berg, Christian Lamparter, Hauke Mehrtens, Josh Boyer,
	Dmitry Torokhov, Jiri Slaby, Andy Lutomirski, Wu Fengguang,
	Richard Purdie, Jeff Mahoney, Jacek Anaszewski, Abhay_Salunke,
	Julia Lawall, Gilles.Muller, nicolas.palix, Tom Gundersen,
	Kay Sievers, David Howells, Alessandro Rubini, Kevin Cernekee,
	Kees Cook, Jonathan Corbet, Thierry Martinez, cocci,
	linux-serial, open list:DOCUMENTATION, linuxppc-dev

On Fri 02 Sep 21:11 PDT 2016, Linus Torvalds wrote:

Linus, I reversed the order of your questions/answers to fit my answer
better.

> On Fri, Sep 2, 2016 at 5:20 PM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> >
> > Thoughts ?
> What are the drivers that need this, and why can't those drivers just
> be fixed to not do braindead things?
> 

I have several cases where remoteproc drivers are used boot DSPs upon
boot of the device, but the firmware files are way too big for being
stored in initramfs and all consumers of the provided services are
(semi-) probable as the remote processor is booted.

I.e. we need some way to figure out when these files become available so
we can bring these remote processors up.

> It's basically the kernel giving up, and relying on user space to give
> a single flag, and it's broken nasty crap.  Worse, it's broken nasty
> crap with a user interface, so we'll be stuck with it forever. Please
> no.
> 

There are several cases where there granularity of a single flag is not
enough and we do already have a working mechanism for relying on user
space to inform the kernel that firmware is available:
CONFIG_FW_LOADER_USER_HELPER_FALLBACK

Another available solution is, as you say, using kernel modules.  But I
really do not like the deployment issues that comes with kernel modules
during development. (The firmware and remoteproc driver normally does
not have the same flow through a development process)

> 
> I really think this is a horrible hack.
> 

I agree, but that said, I would appreciate a automagical mechanism that
would relieve user space from having to signal to the kernel that the
firmware partition has been mounted.

Regards,
Bjorn

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-09-06 17:46                 ` Bjorn Andersson
@ 2016-09-06 18:32                   ` Linus Torvalds
  2016-09-06 21:11                     ` Bjorn Andersson
  2016-09-06 22:32                     ` Luis R. Rodriguez
  0 siblings, 2 replies; 80+ messages in thread
From: Linus Torvalds @ 2016-09-06 18:32 UTC (permalink / raw)
  To: Bjorn Andersson
  Cc: Luis R. Rodriguez, Daniel Vetter, Mimi Zohar, Felix Fietkau,
	David Woodhouse, Roman Pen, Ming Lei, Andrew Morton,
	Michal Marek, Greg KH, Linux Kernel Mailing List,
	Vikram Mulukutla, Stephen Boyd, Mark Brown, Takashi Iwai,
	Johannes Berg, Christian Lamparter, Hauke Mehrtens, Josh Boyer,
	Dmitry Torokhov, Jiri Slaby, Andy Lutomirski, Wu Fengguang,
	Richard Purdie, Jeff Mahoney, Jacek Anaszewski, Abhay_Salunke,
	Julia Lawall, Gilles.Muller, nicolas.palix, Tom Gundersen,
	Kay Sievers, David Howells, Alessandro Rubini, Kevin Cernekee,
	Kees Cook, Jonathan Corbet, Thierry Martinez, cocci,
	linux-serial, open list:DOCUMENTATION, linuxppc-dev

On Tue, Sep 6, 2016 at 10:46 AM, Bjorn Andersson
<bjorn.andersson@linaro.org> wrote:
>
> Linus, I reversed the order of your questions/answers to fit my answer
> better.

Nobody has actually answered the "why don't we just tie the firmware
and module together" question.

Really. If the driver doesn't work without the firmware, then why the
hell is it separated from it in the first place?

The hack is a hack, and it just sounds *stupid*.

               Linus

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-09-06 18:32                   ` Linus Torvalds
@ 2016-09-06 21:11                     ` Bjorn Andersson
  2016-09-06 21:50                       ` Linus Torvalds
  2016-09-06 22:32                     ` Luis R. Rodriguez
  1 sibling, 1 reply; 80+ messages in thread
From: Bjorn Andersson @ 2016-09-06 21:11 UTC (permalink / raw)
  To: Linus Torvalds
  Cc: Luis R. Rodriguez, Daniel Vetter, Mimi Zohar, Felix Fietkau,
	David Woodhouse, Roman Pen, Ming Lei, Andrew Morton,
	Michal Marek, Greg KH, Linux Kernel Mailing List,
	Vikram Mulukutla, Stephen Boyd, Mark Brown, Takashi Iwai,
	Johannes Berg, Christian Lamparter, Hauke Mehrtens, Josh Boyer,
	Dmitry Torokhov, Jiri Slaby, Andy Lutomirski, Wu Fengguang,
	Richard Purdie, Jeff Mahoney, Jacek Anaszewski, Abhay_Salunke,
	Julia Lawall, Gilles.Muller, nicolas.palix, Tom Gundersen,
	Kay Sievers, David Howells, Alessandro Rubini, Kevin Cernekee,
	Kees Cook, Jonathan Corbet, Thierry Martinez, cocci,
	linux-serial, open list:DOCUMENTATION, linuxppc-dev

On Tue 06 Sep 11:32 PDT 2016, Linus Torvalds wrote:

> On Tue, Sep 6, 2016 at 10:46 AM, Bjorn Andersson
> <bjorn.andersson@linaro.org> wrote:
> >
> > Linus, I reversed the order of your questions/answers to fit my answer
> > better.
> 
> Nobody has actually answered the "why don't we just tie the firmware
> and module together" question.
> 

The answer to this depends on the details of the suggestion; but
generally there's a much stronger bond between the kernel and the driver
than between the driver and the firmware in my cases.

E.g. we have a single remoteproc driver loading and controlling the
Hexagon DSP found in several Qualcomm platforms, so a single kernel
binary could (practically) load hundreds of variants of the firmware.

Both the kernel binary and the firmware in this example are side-loaded
onto the device during development - independently of each other, as
they are developed by different teams (or maybe even different
companies).

I assume that you're not suggesting to actually tie the module together,
as that would be practically difficult and a waste of resources.

Which leaves us with the suggestion that we should store the kernel
module with the firmware file, which is just infeasible from a few
practical reasons - again mostly related to the development flow and how
the files are contained on the devices.

> Really. If the driver doesn't work without the firmware, then why the
> hell is it separated from it in the first place?
> 

In several cases we have a single remoteproc driver controlling several
different co-processors. Further more with the aspiration of being able
to run the same kernel binary (including modules) on more than one
product this is simply not feasible.

As I said above, beyond development there are hundreds of variants of
these firmware files in products - each weighting in at 10-50MB.

The firmware loading part (remoteproc) doesn't care about these
differences and the functional drivers attaching to the services
provided by the firmware can handle the differences between them.

> The hack is a hack, and it just sounds *stupid*.
> 

This I totally agree with.

Regards,
Bjorn

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-09-06 21:11                     ` Bjorn Andersson
@ 2016-09-06 21:50                       ` Linus Torvalds
  2016-09-06 23:04                         ` Luis R. Rodriguez
  0 siblings, 1 reply; 80+ messages in thread
From: Linus Torvalds @ 2016-09-06 21:50 UTC (permalink / raw)
  To: Bjorn Andersson
  Cc: Luis R. Rodriguez, Daniel Vetter, Mimi Zohar, Felix Fietkau,
	David Woodhouse, Roman Pen, Ming Lei, Andrew Morton,
	Michal Marek, Greg KH, Linux Kernel Mailing List,
	Vikram Mulukutla, Stephen Boyd, Mark Brown, Takashi Iwai,
	Johannes Berg, Christian Lamparter, Hauke Mehrtens, Josh Boyer,
	Dmitry Torokhov, Jiri Slaby, Andy Lutomirski, Wu Fengguang,
	Richard Purdie, Jeff Mahoney, Jacek Anaszewski, Abhay_Salunke,
	Julia Lawall, Gilles.Muller, nicolas.palix, Tom Gundersen,
	Kay Sievers, David Howells, Alessandro Rubini, Kevin Cernekee,
	Kees Cook, Jonathan Corbet, Thierry Martinez, cocci,
	linux-serial, open list:DOCUMENTATION, linuxppc-dev

On Tue, Sep 6, 2016 at 2:11 PM, Bjorn Andersson
<bjorn.andersson@linaro.org> wrote:
> On Tue 06 Sep 11:32 PDT 2016, Linus Torvalds wrote:
>
>> On Tue, Sep 6, 2016 at 10:46 AM, Bjorn Andersson
>> Nobody has actually answered the "why don't we just tie the firmware
>> and module together" question.
>
> The answer to this depends on the details of the suggestion; but
> generally there's a much stronger bond between the kernel and the driver
> than between the driver and the firmware in my cases.

I call BS.

Let me be very clear. I'm not applying that shit-for-brains stupid
patch, and will not be pulling it unless somebody tricks me into it.

Because all these arguments make no sense at all.

If the driver doesn't work without the firmware, then anybody who
distributes the driver binary without a firmware is just
*fundamentally* doing something insane. You may do it for
*development* purposes, but doing so for actual *use* would be
entirely pointless.

See my point? If a distribution is distributing the driver without the
firmware, then what the hell is the point of such a thing?

But even if you decide to do that for some odd reason, the patch is
still just stupid. Instead of adding some crazy infrastructure for
"now I've mounted everything", you could equally well just

 (a) make the driver fail the module load if it cannot find a firmware binary

 (b) after user space has mounted everything, just do "insmod -a"
again (or insmod just that driver).

See? The point is, this "generic" hacky interface is just stupid. It's
not adding any value. If you add user space "I'm ready now" points
anyway, you might as well make those points do the right thing and
just load the module that is now loadable.

We could mark such "late loading" modules explicitly if people want
to, so that you can automate the whole thing about delaying the
loading in user space.

At no point does it make sense to say "I have now mounted all the
important filesystems". Maybe the firmware is extracted later by user
space downloading it from the internet, and the module will then work
only after that point"./

This whole "I have mounted important filesystems" is just pure and
utter garbage. Stop pushing this shit.

                 Linus

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-09-03 18:10                         ` Dmitry Torokhov
@ 2016-09-06 21:52                           ` Luis R. Rodriguez
  2016-09-06 22:28                             ` Bjorn Andersson
  2016-09-24  1:37                           ` Herbert, Marc
  1 sibling, 1 reply; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-09-06 21:52 UTC (permalink / raw)
  To: Dmitry Torokhov
  Cc: Linus Torvalds, open list:DOCUMENTATION, Jacek Anaszewski,
	David Woodhouse, Christian Lamparter, Julia Lawall,
	Andrew Morton, linuxppc-dev, Mimi Zohar, Andy Lutomirski,
	Richard Purdie, Wu Fengguang, Johannes Berg, Luis R. Rodriguez,
	Michal Marek, Hauke Mehrtens, Mark Brown, Jiri Slaby, Ming Lei,
	Daniel Vetter, Kevin Cernekee, Jeff Mahoney, Greg KH,
	Bjorn Andersson, Jonathan Corbet, Felix Fietkau, David Howells,
	Vikram Mulukutla, Alessandro Rubini, Tom Gundersen, Kees Cook,
	Takashi Iwai, Gilles.Muller, linux-serial, Roman Pen,
	Kay Sievers, Stephen Boyd, nicolas.palix, Abhay_Salunke,
	Linux Kernel Mailing List, Thierry Martinez, Josh Boyer

On Sat, Sep 03, 2016 at 11:10:02AM -0700, Dmitry Torokhov wrote:
> On Sat, Sep 3, 2016 at 11:01 AM, Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
> > On Sat, Sep 3, 2016 at 10:49 AM, Dmitry Torokhov
> > <dmitry.torokhov@gmail.com> wrote:
> >>
> >> Unfortunately module loading and availability of firmware is very
> >> loosely coupled.
> >
> > The whole "let's add a new magical proc entry 

To be fair it was using a generic sysfs entry.

> > to say that all
> > filesystems are mounted" is all about the user space coupling them.
> 
> I was thinking if we kernel could post "conditions" (maybe simple strings)
> that it waits for, and userspace could unlock these "conditions". One of them
> might be "firmware available".

Dmitry, you seem to be suggesting a generic kernel-userspace "registry" for
intertwined dependencies that the kernel needs and only userspace can provide,
is that right?

If so it sounds overly complicated to resolve this. Do we have other uses
outside of kernel_read_file_from_path() you had in mind for this?

> > I'm just saying that if user space must know about when firmware is
> > ready to be loaded anyway, just do it right. Not with some hacky "you
> > can now do random things" flag.

Note, this isn't just about firmware since we now have a generic API to read
files from the kernel, so kernel_read_file_from_path(). Firmware is now just
*one* user case. Also a complexity here is this is not just for modules but
also for built-in drivers as well. And you want the option to update the
files without the driver.

The proposed solution provides a generic broad syfs entry for letting userspace
inform the kernel when files from the filesystems where it would typically read
from (I'm calling them critical filesystems for lack of a better term) can be
accessible. Something more specific requires a bit more thought given this is
not anymore about just firmware, must also address built-in drivers, and allow
for updates.

> >  But by having user space actually say
> > "put this module together with this firmware".

Keep in mind this isn't about just firmware anymore, and we have to consider
built-in drivers as well. But if we were to just consider firmware... for the
sake of following with examples.

How would this registry work?

We already have MODULE_FIRMWARE(), we could have MODULE_FIRMWARE_REQ() or
something like it to help annotate the the driver was only functional with the
firmware, punt things to kmod to deal with the requirements.  kmod would only
load the driver if the firmware is present as well, otherwise it could just
return a new -ENOFIRMWARE and try to defer module loading for a later time, it
would load the driver once and if the firmware becomes available... This all
seems rather hacky.

For built-in drivers.. the vmlinux would have the associated firmware reqs, it
would still need a way to let userspace know when such firmware is ready. What
kernel <-> userspace API would we want to use for this ? Or as you note we
could just prevent such drivers to be built-in. I'd be happy with this, however
I don't think the distributions using non-modular kernels will be.

Now whatever we come up with recall this isn't just about firmware anymore.
Which is why I ended up bundling all these requirements up into one generic
"kernel reads file from filesystem" requirement. I can see the syfs approach
being considered hacky -- but I gladly welcome an actual alternative
suggestion.

> > If you just put the two pieces together, then the module "will just work".
> >
> > And quite frankly, that sounds like a much better maintenance practice
> > anyway. If some module doesn't work without the firmware, then dammit,
> > the two *should* go together. Putting them in two different places
> > would be *INSANE*.
> 
> Quite often it does until it does not. Most of the touch controllers
> work just fine until some event (abrupt cutting of power for example)
> where nvram gets corrupted and they come up in bootloader mode. It is
> just an example.

You want to also opt-in for updates, you don't want to require re-building
a driver for a firmware fix, for instance.

 Luis

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-09-06 21:52                           ` Luis R. Rodriguez
@ 2016-09-06 22:28                             ` Bjorn Andersson
  2016-09-06 23:14                               ` Luis R. Rodriguez
  0 siblings, 1 reply; 80+ messages in thread
From: Bjorn Andersson @ 2016-09-06 22:28 UTC (permalink / raw)
  To: Luis R. Rodriguez
  Cc: Dmitry Torokhov, Linus Torvalds, open list:DOCUMENTATION,
	Jacek Anaszewski, David Woodhouse, Christian Lamparter,
	Julia Lawall, Andrew Morton, linuxppc-dev, Mimi Zohar,
	Andy Lutomirski, Richard Purdie, Wu Fengguang, Johannes Berg,
	Michal Marek, Hauke Mehrtens, Mark Brown, Jiri Slaby, Ming Lei,
	Daniel Vetter, Kevin Cernekee, Jeff Mahoney, Greg KH,
	Jonathan Corbet, Felix Fietkau, David Howells, Vikram Mulukutla,
	Alessandro Rubini, Tom Gundersen, Kees Cook, Takashi Iwai,
	Gilles.Muller, linux-serial, Roman Pen, Kay Sievers,
	Stephen Boyd, nicolas.palix, Abhay_Salunke,
	Linux Kernel Mailing List, Thierry Martinez, Josh Boyer

On Tue 06 Sep 14:52 PDT 2016, Luis R. Rodriguez wrote:

> We already have MODULE_FIRMWARE(), we could have MODULE_FIRMWARE_REQ() or
> something like it to help annotate the the driver was only functional with the
> firmware, punt things to kmod to deal with the requirements.

That implies that a single driver will only use a single version of the
firmware. There are cases where we want a single driver to load firmware
depending on e.g. hardware revisions, or previous firmware version and
there are cases where we want to load firmware based on requested
use-cases.

Regards,
Bjorn

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-09-06 18:32                   ` Linus Torvalds
  2016-09-06 21:11                     ` Bjorn Andersson
@ 2016-09-06 22:32                     ` Luis R. Rodriguez
  1 sibling, 0 replies; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-09-06 22:32 UTC (permalink / raw)
  To: Linus Torvalds
  Cc: Bjorn Andersson, Luis R. Rodriguez, Daniel Vetter, Mimi Zohar,
	Felix Fietkau, David Woodhouse, Roman Pen, Ming Lei,
	Andrew Morton, Michal Marek, Greg KH, Linux Kernel Mailing List,
	Vikram Mulukutla, Stephen Boyd, Mark Brown, Takashi Iwai,
	Johannes Berg, Christian Lamparter, Hauke Mehrtens, Josh Boyer,
	Dmitry Torokhov, Jiri Slaby, Andy Lutomirski, Wu Fengguang,
	Richard Purdie, Jeff Mahoney, Jacek Anaszewski, Abhay_Salunke,
	Julia Lawall, Gilles.Muller, nicolas.palix, Tom Gundersen,
	Kay Sievers, David Howells, Alessandro Rubini, Kevin Cernekee,
	Kees Cook, Jonathan Corbet, Thierry Martinez, linux-serial,
	open list:DOCUMENTATION, linuxppc-dev

On Tue, Sep 06, 2016 at 11:32:05AM -0700, Linus Torvalds wrote:
> On Tue, Sep 6, 2016 at 10:46 AM, Bjorn Andersson
> <bjorn.andersson@linaro.org> wrote:
> >
> > Linus, I reversed the order of your questions/answers to fit my answer
> > better.
> 
> Nobody has actually answered the "why don't we just tie the firmware

This is not about firmware anymore. The fact that we were reading files from
the filesystem in different ways called for a generic API, that was done by
Mimi with the new generic kernel_read_file_from_path(), the fact that there
were race concerns with firmware means such races are also in theory possible
outside of firmware.

> and module together" question.

In terms of the firmware, licensing is one reason I'm aware of. Another reason
is updates to firmware files may not implicate a driver update -- its pointless
to rebuild a kernel if all you need is a firmware update.

For both modules and built-in we already have the option to bundle firmware
into initramfs, and CONFIG_EXTRA_FIRMWARE, some folks do not want to use these.
Bjorn noted a few reasons why. Here's the full list of reasons I've heard why
folks shy away from these:

  o Licensing
  o You still want the ability to do updates
  o The size of the files can be huge (remoteproc is talking about 10 MiB files)

Do we want to bring the firmware closer together than what we allow for modules? If
so it may make sense to use a modpost command to try to bundle module and
firmware together, the build system could use the MODULE_FIRMWARE() for that.
It does mean you now have a build dependency for linux-firmware on the kernel.
Do we want that?

> Really. If the driver doesn't work without the firmware, then why the
> hell is it separated from it in the first place?

This I agree with, although you still have to consider you may want to enable
updates for firmware without a driver update.

  Luis

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-09-06 21:50                       ` Linus Torvalds
@ 2016-09-06 23:04                         ` Luis R. Rodriguez
  2016-09-24  2:51                           ` Herbert, Marc
  0 siblings, 1 reply; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-09-06 23:04 UTC (permalink / raw)
  To: Linus Torvalds
  Cc: Bjorn Andersson, Luis R. Rodriguez, Daniel Vetter, Mimi Zohar,
	Felix Fietkau, David Woodhouse, Roman Pen, Ming Lei,
	Andrew Morton, Michal Marek, Greg KH, Linux Kernel Mailing List,
	Vikram Mulukutla, Stephen Boyd, Mark Brown, Takashi Iwai,
	Johannes Berg, Christian Lamparter, Hauke Mehrtens, Josh Boyer,
	Dmitry Torokhov, Jiri Slaby, Andy Lutomirski, Wu Fengguang,
	Richard Purdie, Jeff Mahoney, Jacek Anaszewski, Abhay_Salunke,
	Julia Lawall, Gilles.Muller, nicolas.palix, Tom Gundersen,
	Kay Sievers, David Howells, Alessandro Rubini, Kevin Cernekee,
	Kees Cook, Jonathan Corbet, Thierry Martinez, linux-serial,
	open list:DOCUMENTATION, linuxppc-dev

On Tue, Sep 06, 2016 at 02:50:51PM -0700, Linus Torvalds wrote:
> On Tue, Sep 6, 2016 at 2:11 PM, Bjorn Andersson
> <bjorn.andersson@linaro.org> wrote:
> > On Tue 06 Sep 11:32 PDT 2016, Linus Torvalds wrote:
> >
> >> On Tue, Sep 6, 2016 at 10:46 AM, Bjorn Andersson
> >> Nobody has actually answered the "why don't we just tie the firmware
> >> and module together" question.
> >
> > The answer to this depends on the details of the suggestion; but
> > generally there's a much stronger bond between the kernel and the driver
> > than between the driver and the firmware in my cases.
> 
> I call BS.
> 
> Let me be very clear. I'm not applying that shit-for-brains stupid
> patch, and will not be pulling it unless somebody tricks me into it.

Great thanks. Please note that the only reason I proposed this was to
get the ball rolling in terms of finding a solution to the problem for why
some folks claim they *need* the firmware usermode helper. Granted, upstream
we only have 2 explicit users left, I'm told some out-of-tree users still
need and use the usermode helper.

They claim that without it there is the race between /lib/firmware being ready
and driver asking for the firmware. I was told there were quite a bit
of out-of-tree hacks to address this without using the usermode helper,
the goal of this patch was to create the discussion needed to a proper
resolution to this.

Given that upon inspection -- I've determined that even if you stuff the
firmware into initramfs you may still run into the race (the commit log of this
patch explains that) and that using initramfs was the alternative solution we
expected folks to use -- the only current deterministic sure way a driver can
depend on a firmware and avoid the race is to use CONFIG_EXTRA_FIRMWARE. This
is an issue.

> Because all these arguments make no sense at all.
> 
> If the driver doesn't work without the firmware, then anybody who
> distributes the driver binary without a firmware is just
> *fundamentally* doing something insane.

Some companies only redistribute firmware binaries to specific entities due to
avoid expanding to whom a patent grant is given to. That is, not every company
writing drivers or pushing out binary drivers is willing to dish out the
firmware as per the terms in linux-firmware.

> You may do it for *development* purposes, but doing so for actual *use* would
> be entirely pointless.

To be fair we haven't been explicit about our requirements for firmware_class
and expectations about what we expect for firmware for driver in Linux. This
has all been rather loose.

Furthermore the race issues we have found in firmware_class have only come about
through introspection, and I've been slowly documenting all this tribal knowledge.

> See my point? If a distribution is distributing the driver without the
> firmware, then what the hell is the point of such a thing?

Agreed.

> But even if you decide to do that for some odd reason, the patch is
> still just stupid. Instead of adding some crazy infrastructure for
> "now I've mounted everything", you could equally well just
> 
>  (a) make the driver fail the module load if it cannot find a firmware binary

Not sure if its clear but:

0) this is not just about firmware anymore
1) there is a race between using kernel_read_file_from_path() and
   having the filesystem that should be present on be ready;
2) Only *userspace* can know for sure what the real valid filesystem for
   files read from kernel_read_file_from_path() can be ready, so only userspace
   can tell the kernel if a read from kernel_read_file_from_path() is
   at a certain point in time valid.

>  (b) after user space has mounted everything, just do "insmod -a"
> again (or insmod just that driver).

I'm happy to document this as the resolution... I have a feeling some folks
will not like it. We also have built-in drivers to consider, what do we
advise for that? Keep in mind only CONFIG_EXTRA_FIRMWARE is deterministically
safe.

> See? The point is, this "generic" hacky interface is just stupid. It's
> not adding any value. If you add user space "I'm ready now" points
> anyway, you might as well make those points do the right thing and
> just load the module that is now loadable.

This is not about firmware anymore though, and we need to address built-in.

> We could mark such "late loading" modules explicitly if people want
> to, so that you can automate the whole thing about delaying the
> loading in user space.

Now *that* could actually help, for instance add another late
init call which would be called after initramfs and friends -- perhaps
way after prepare_namespace() -- by then we would ensure userspace has
all critical fs mounted. The problem though is we'd still need a way
for userspace to tell the kernel that all critical fs are mounted
as only userspace can know this for sure.

When is that done? How would the kernel know?

We do have PROBE_PREFER_ASYNCHRONOUS, but that does not provide any
guarantees over ready filesystems.

> At no point does it make sense to say "I have now mounted all the
> important filesystems". Maybe the firmware is extracted later by user
> space downloading it from the internet, and the module will then work
> only after that point"./
> 
> This whole "I have mounted important filesystems" is just pure and
> utter garbage. Stop pushing this shit.

Happy to do so, we do however need to document what we do expect users and
developers to do about this race for both drivers and built-in, keeping
in mind this is also for any users of kernel_read_file_from_path() as well.

  Luis

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-09-06 22:28                             ` Bjorn Andersson
@ 2016-09-06 23:14                               ` Luis R. Rodriguez
  0 siblings, 0 replies; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-09-06 23:14 UTC (permalink / raw)
  To: Bjorn Andersson
  Cc: Luis R. Rodriguez, Dmitry Torokhov, Linus Torvalds,
	open list:DOCUMENTATION, Jacek Anaszewski, David Woodhouse,
	Christian Lamparter, Julia Lawall, Andrew Morton, linuxppc-dev,
	Mimi Zohar, Andy Lutomirski, Richard Purdie, Wu Fengguang,
	Johannes Berg, Michal Marek, Hauke Mehrtens, Mark Brown,
	Jiri Slaby, Ming Lei, Daniel Vetter, Kevin Cernekee,
	Jeff Mahoney, Greg KH, Jonathan Corbet, Felix Fietkau,
	David Howells, Vikram Mulukutla, Alessandro Rubini,
	Tom Gundersen, Kees Cook, Takashi Iwai, Gilles.Muller,
	linux-serial, Roman Pen, Kay Sievers, Stephen Boyd,
	nicolas.palix, Abhay_Salunke, Linux Kernel Mailing List,
	Thierry Martinez, Josh Boyer

On Tue, Sep 06, 2016 at 03:28:47PM -0700, Bjorn Andersson wrote:
> On Tue 06 Sep 14:52 PDT 2016, Luis R. Rodriguez wrote:
> 
> > We already have MODULE_FIRMWARE(), we could have MODULE_FIRMWARE_REQ() or
> > something like it to help annotate the the driver was only functional with the
> > firmware, punt things to kmod to deal with the requirements.
> 
> That implies that a single driver will only use a single version of the
> firmware. 

Today firmware requests are done manually by the driver, in the future it should
be possible to specify just an array of firmwares and on that list specify
which firmware is optional, and perhaps if you need at last one. Then you treat
optional firmware upgrades as optional -- and only treat fatal conditions as
such. Today drivers manage all this on their own.  This is something we can
later do as we have the flexible firmware API in place, but for now -- you are
right. There is no clear way to extract the semantics of what firmware
requirements really are in an easy way.

> There are cases where we want a single driver to load firmware
> depending on e.g. hardware revisions,

Dynamic firmware names -- indeed. Good point.

> or previous firmware version and
> there are cases where we want to load firmware based on requested
> use-cases.

True.

  Luis

^ permalink raw reply	[flat|nested] 80+ messages in thread

* [PATCH v4 0/5] firmware: add SmPL grammar to avoid issues
  2016-08-24  0:45 ` [PATCH v3 " mcgrof
                     ` (4 preceding siblings ...)
  2016-08-24  0:45   ` [PATCH v3 5/5] firmware: fix fw cache to avoid usermode helper on suspend mcgrof
@ 2016-09-07  0:42   ` Luis R. Rodriguez
  2016-09-07  0:42     ` [PATCH v4 1/5] MAINTAINERS: extend firmware_class maintainer list Luis R. Rodriguez
                       ` (4 more replies)
  5 siblings, 5 replies; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-09-07  0:42 UTC (permalink / raw)
  To: ming.lei, akpm, gregkh
  Cc: daniel.wagner, mmarek, linux-kernel, markivx, stephen.boyd,
	zohar, broonie, tiwai, johannes, chunkeey, hauke, jwboyer,
	dmitry.torokhov, dwmw2, jslaby, torvalds, luto, fengguang.wu,
	rpurdie, j.anaszewski, Abhay_Salunke, Julia.Lawall,
	Gilles.Muller, nicolas.palix, teg, dhowells, bjorn.andersson,
	arend.vanspriel, kvalo, Luis R. Rodriguez

Greg,

this v4 includes only two further changes since my last respin:

  o As noted by Daniel Wagner, this removes the timeout reset for
    when the firmware cache is used
  o Typo fix as noted by Gabriel Paubert

Luis R. Rodriguez (5):
  MAINTAINERS: extend firmware_class maintainer list
  firmware: annotate thou shalt not request fw on init or probe
  firmware: update usermode helper docs and add SmPL report
  firmware: add usermode helper DECLARE_FW_LOADER_USER() annotation
  firmware: fix fw cache to avoid usermode helper on suspend

 Documentation/firmware_class/README                |  61 +++++++++-
 MAINTAINERS                                        |   1 +
 drivers/base/Kconfig                               |   2 +-
 drivers/base/firmware_class.c                      |  16 +--
 drivers/firmware/dell_rbu.c                        |   1 +
 drivers/leds/leds-lp55xx-common.c                  |   1 +
 include/linux/firmware.h                           |   7 ++
 .../request_firmware-avoid-init-probe-init.cocci   | 130 +++++++++++++++++++++
 .../coccinelle/api/request_firmware-usermode.cocci |  44 +++++++
 9 files changed, 242 insertions(+), 21 deletions(-)
 create mode 100644 scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
 create mode 100644 scripts/coccinelle/api/request_firmware-usermode.cocci

-- 
2.9.2

^ permalink raw reply	[flat|nested] 80+ messages in thread

* [PATCH v4 1/5] MAINTAINERS: extend firmware_class maintainer list
  2016-09-07  0:42   ` [PATCH v4 0/5] firmware: add SmPL grammar to avoid issues Luis R. Rodriguez
@ 2016-09-07  0:42     ` Luis R. Rodriguez
  2016-09-07  6:43       ` Greg KH
  2016-09-07  0:42     ` [PATCH v4 2/5] firmware: annotate thou shalt not request fw on init or probe Luis R. Rodriguez
                       ` (3 subsequent siblings)
  4 siblings, 1 reply; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-09-07  0:42 UTC (permalink / raw)
  To: ming.lei, akpm, gregkh
  Cc: daniel.wagner, mmarek, linux-kernel, markivx, stephen.boyd,
	zohar, broonie, tiwai, johannes, chunkeey, hauke, jwboyer,
	dmitry.torokhov, dwmw2, jslaby, torvalds, luto, fengguang.wu,
	rpurdie, j.anaszewski, Abhay_Salunke, Julia.Lawall,
	Gilles.Muller, nicolas.palix, teg, dhowells, bjorn.andersson,
	arend.vanspriel, kvalo, Luis R. Rodriguez

I've been reviewing changes proactively, and plan on doing
more of this work. I'm doing this early as I should be getting
e-mailed about proposed changes.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 MAINTAINERS | 1 +
 1 file changed, 1 insertion(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index fcb387135242..af64eb999ef4 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4891,6 +4891,7 @@ F:	tools/firewire/
 
 FIRMWARE LOADER (request_firmware)
 M:	Ming Lei <ming.lei@canonical.com>
+M:	Luis R. Rodriguez <mcgrof@kernel.org>
 L:	linux-kernel@vger.kernel.org
 S:	Maintained
 F:	Documentation/firmware_class/
-- 
2.9.2

^ permalink raw reply	[flat|nested] 80+ messages in thread

* [PATCH v4 2/5] firmware: annotate thou shalt not request fw on init or probe
  2016-09-07  0:42   ` [PATCH v4 0/5] firmware: add SmPL grammar to avoid issues Luis R. Rodriguez
  2016-09-07  0:42     ` [PATCH v4 1/5] MAINTAINERS: extend firmware_class maintainer list Luis R. Rodriguez
@ 2016-09-07  0:42     ` Luis R. Rodriguez
  2016-09-07  0:42     ` [PATCH v4 3/5] firmware: update usermode helper docs and add SmPL report Luis R. Rodriguez
                       ` (2 subsequent siblings)
  4 siblings, 0 replies; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-09-07  0:42 UTC (permalink / raw)
  To: ming.lei, akpm, gregkh
  Cc: daniel.wagner, mmarek, linux-kernel, markivx, stephen.boyd,
	zohar, broonie, tiwai, johannes, chunkeey, hauke, jwboyer,
	dmitry.torokhov, dwmw2, jslaby, torvalds, luto, fengguang.wu,
	rpurdie, j.anaszewski, Abhay_Salunke, Julia.Lawall,
	Gilles.Muller, nicolas.palix, teg, dhowells, bjorn.andersson,
	arend.vanspriel, kvalo, Luis R. Rodriguez, Alessandro Rubini,
	Kevin Cernekee, Daniel Vetter, Kees Cook, Jonathan Corbet,
	Thierry Martinez, cocci, linux-serial, linux-doc, linuxppc-dev

Thou shalt not make firmware calls early on init or probe.

systemd already ripped support out for the usermode helper
a while ago, there are still users that require the usermode helper,
however systemd's use of the usermode helper exacerbated a long
lasting issue of the fact that we have many drivers that load
firmware on init or probe. Independent of the usermode helper
loading firmware on init or probe is a bad idea for a few reasons.

When the firmware is read directly from the filesystem by the kernel,
if the driver is built-in to the kernel the firmware may not yet be
available, for such uses one could use initramfs and stuff the firmware
inside, or one also use CONFIG_EXTRA_FIRMWARE; however not all distributions
use this, as such generally one cannot count on this. There is another
corner cases to consider, since we are accessing the firmware directly folks
cannot expect new found firmware on a filesystem after we switch off from
an initramfs with pivot_root().

Supporting such situations is possible today but fixing it for good is
really complex due to the large amount of variablity in the boot up
process.

Instead just document the expectations properly and add a grammar rule to
enable folks to check / validate / police if drivers are using the request
firmware API early on init or probe.

The SmPL rule used to check for the probe routine is loose and is
currently defined through a regexp, that can easily be extended to any
other known bus probe routine names. It also uses the new Python
iteration support which allows us to backtrack from a request_firmware*()
call back to a possible probe or init, iteratively. Without iteration
we would only be able to get reports for callers who directly use the
request_firmware*() API on the initial probe or init routine.

There are 4 offenders at this time:

mcgrof@ergon ~/linux-next (git::20160609)$ export COCCI=scripts/coccinelle/api/request_firmware.cocci
mcgrof@ergon ~/linux-next (git::20160609)$ make coccicheck MODE=report

drivers/fmc/fmc-fakedev.c: ERROR: driver call request firmware call on its init routine on line 321.
drivers/fmc/fmc-write-eeprom.c: ERROR: driver call request firmware call on its probe routine on line 136.
drivers/tty/serial/rp2.c: ERROR: driver call request firmware call on its probe routine on line 796.
drivers/tty/serial/ucc_uart.c: ERROR: driver call request firmware call on its probe routine on line 1246.

I checked and verified all these are valid reports. This list also matches
the same list as in 20150805, so we have fortunately not gotten worse.
Let's keep it that way and also fix these drivers.

v3:

Clarify that if initramfs or builtin_fw cannot be used the race
between driver probe and /lib/firmware/ being ready must be addressed.

v2: changes from v1 [0]:

o This now supports iteration, this extends our coverage on the report

o Update documentation and commit log to accept the fate of not being
  able to remove the usermode helper.

[0] https://lkml.kernel.org/r/1440811107-861-1-git-send-email-mcgrof@do-not-panic.com

Cc: Alessandro Rubini <rubini@gnudd.com>
Cc: Kevin Cernekee <cernekee@gmail.com>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: Mimi Zohar <zohar@linux.vnet.ibm.com>
Cc: David Woodhouse <dwmw2@infradead.org>
Cc: Kees Cook <keescook@chromium.org>
Cc: Dmitry Torokhov <dmitry.torokhov@gmail.com>
Cc: Ming Lei <ming.lei@canonical.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Julia Lawall <Julia.Lawall@lip6.fr>
Cc: Gilles Muller <Gilles.Muller@lip6.fr>
Cc: Nicolas Palix <nicolas.palix@imag.fr>
Cc: Thierry Martinez <martinez@nsup.org>
Cc: Michal Marek <mmarek@suse.com>
Cc: cocci@systeme.lip6.fr
Cc: Alessandro Rubini <rubini@gnudd.com>
Cc: Kevin Cernekee <cernekee@gmail.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Jiri Slaby <jslaby@suse.com>
Cc: linux-serial@vger.kernel.org
Cc: linux-doc@vger.kernel.org
Cc: linux-serial@vger.kernel.org
Cc: linuxppc-dev@lists.ozlabs.org
Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 Documentation/firmware_class/README                |  22 ++++
 drivers/base/Kconfig                               |   2 +-
 .../request_firmware-avoid-init-probe-init.cocci   | 130 +++++++++++++++++++++
 3 files changed, 153 insertions(+), 1 deletion(-)
 create mode 100644 scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci

diff --git a/Documentation/firmware_class/README b/Documentation/firmware_class/README
index cafdca8b3b15..20aca5901785 100644
--- a/Documentation/firmware_class/README
+++ b/Documentation/firmware_class/README
@@ -93,6 +93,28 @@
    user contexts to request firmware asynchronously, but can't be called
    in atomic contexts.
 
+Requirements:
+=============
+
+You should avoid at all costs requesting firmware on both init and probe paths
+of your device driver. Reason for this is loading firmware and using it can
+often delay boot and we can have races early in boot, between a device's probe
+and having the real /lib/firmware ready.
+
+Drivers that really need firmware early should stuff the firmware in
+initramfs or consider using CONFIG_EXTRA_FIRMWARE. Using initramfs is much
+more portable to more distributions as not all distributions wish to enable
+CONFIG_EXTRA_FIRMWARE. Should a driver require the firmware being built-in
+it should depend on CONFIG_EXTRA_FIRMWARE. There is no current annotation for
+requiring a firmware on initramfs.
+
+If drivers cannot use CONFIG_EXTRA_FIRMWARE or initramfs the races between
+probe and having /lib/firmware ready needs to be addressed first.
+
+If you're a maintainer you can help police this with:
+
+$ export COCCI=scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
+$ make coccicheck MODE=report
 
  about in-kernel persistence:
  ---------------------------
diff --git a/drivers/base/Kconfig b/drivers/base/Kconfig
index fdf44cac08e6..d493e7f45805 100644
--- a/drivers/base/Kconfig
+++ b/drivers/base/Kconfig
@@ -152,7 +152,7 @@ config FW_LOADER_USER_HELPER
 	bool
 
 config FW_LOADER_USER_HELPER_FALLBACK
-	bool "Fallback user-helper invocation for firmware loading"
+	bool "Fallback user-helper invocation for firmware loading (avoid)"
 	depends on FW_LOADER
 	select FW_LOADER_USER_HELPER
 	help
diff --git a/scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci b/scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
new file mode 100644
index 000000000000..cf180c59e042
--- /dev/null
+++ b/scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
@@ -0,0 +1,130 @@
+///
+/// Thou shalt not request firmware on init or probe
+///
+// Confidence: High
+// Copyright: (C) 2016 Luis R. Rodriguez <mcgrof@kernel.org>
+// Copyright: (C) 2015 Julia Lawall, Inria/LIP6.
+//
+// GPLv2.
+// URL: http://coccinelle.lip6.fr/
+// Options: --no-includes --include-headers --no-show-diff
+// Requires: 1.0.5
+//
+// Coccinelle 1.0.5 is required given that this uses the shiny new
+// python iteration feature.
+
+virtual report
+virtual after_start
+
+// -------------------------------------------------------------------------
+
+@initialize:python@
+@@
+
+seen = set()
+
+def add_if_not_present(f, file, starter_var):
+    if (f, file, starter_var) not in seen:
+        seen.add((f, file, starter_var))
+        it = Iteration()
+        if file != None:
+            it.set_files([file])
+    it.add_virtual_rule(after_start)
+    it.add_virtual_rule(report)
+    it.add_virtual_identifier(fn, f)
+    it.add_virtual_identifier(starter, starter_var)
+    it.register()
+
+reported = set()
+
+def print_err(str, file, line):
+    if (file, line) not in reported:
+        reported.add((file, line))
+	print "%s: ERROR: driver call request firmware call on its %s routine on line %d." % (file, str, line)
+
+@ defines_module_init@
+declarer name module_init;
+identifier init;
+@@
+
+module_init(init);
+
+@ has_probe depends on defines_module_init@
+identifier drv_calls, drv_probe;
+type bus_driver;
+identifier probe_op =~ "(probe)";
+@@
+
+bus_driver drv_calls = {
+	.probe_op = drv_probe,
+};
+
+@hascall depends on !after_start && defines_module_init@
+position p;
+@@
+
+(
+request_firmware@p(...)
+|
+request_firmware_nowait@p(...)
+|
+request_firmware_direct@p(...)
+)
+
+@script:python@
+init << defines_module_init.init;
+p << hascall.p;
+@@
+
+if p[0].current_element == init:
+    print_err("init", p[0].file, int(p[0].line))
+    cocci.include_match(False)
+
+@script:python@
+drv_probe << has_probe.drv_probe;
+p << hascall.p;
+@@
+
+if p[0].current_element == drv_probe:
+    print_err("probe", p[0].file, int(p[0].line))
+    cocci.include_match(False)
+
+@script:python@
+p << hascall.p;
+@@
+
+add_if_not_present(p[0].current_element, p[0].file, p[0].line)
+
+@hasrecall depends on after_start@
+position p;
+identifier virtual.fn;
+@@
+
+fn@p(...)
+
+@script:python@
+init << defines_module_init.init;
+p << hasrecall.p;
+starter << virtual.starter;
+@@
+
+if p[0].current_element == init:
+    print_err("init", p[0].file, int(starter))
+    cocci.include_match(False)
+
+@script:python@
+drv_probe << has_probe.drv_probe;
+p << hasrecall.p;
+starter << virtual.starter;
+@@
+
+if p[0].current_element == drv_probe:
+    print_err("probe", p[0].file, int(starter))
+    cocci.include_match(False)
+
+@script:python@
+p << hasrecall.p;
+starter << virtual.starter;
+@@
+
+add_if_not_present(p[0].current_element, p[0].file, starter)
-- 
2.9.2

^ permalink raw reply	[flat|nested] 80+ messages in thread

* [PATCH v4 3/5] firmware: update usermode helper docs and add SmPL report
  2016-09-07  0:42   ` [PATCH v4 0/5] firmware: add SmPL grammar to avoid issues Luis R. Rodriguez
  2016-09-07  0:42     ` [PATCH v4 1/5] MAINTAINERS: extend firmware_class maintainer list Luis R. Rodriguez
  2016-09-07  0:42     ` [PATCH v4 2/5] firmware: annotate thou shalt not request fw on init or probe Luis R. Rodriguez
@ 2016-09-07  0:42     ` Luis R. Rodriguez
  2016-09-07  0:42     ` [PATCH v4 4/5] firmware: add usermode helper DECLARE_FW_LOADER_USER() annotation Luis R. Rodriguez
  2016-09-07  0:42     ` [PATCH v4 5/5] firmware: fix fw cache to avoid usermode helper on suspend Luis R. Rodriguez
  4 siblings, 0 replies; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-09-07  0:42 UTC (permalink / raw)
  To: ming.lei, akpm, gregkh
  Cc: daniel.wagner, mmarek, linux-kernel, markivx, stephen.boyd,
	zohar, broonie, tiwai, johannes, chunkeey, hauke, jwboyer,
	dmitry.torokhov, dwmw2, jslaby, torvalds, luto, fengguang.wu,
	rpurdie, j.anaszewski, Abhay_Salunke, Julia.Lawall,
	Gilles.Muller, nicolas.palix, teg, dhowells, bjorn.andersson,
	arend.vanspriel, kvalo, Luis R. Rodriguez, linux-leds

We've determined that very sadly we cannot nuke the usermode helper.
The firmware code is a bit hard to follow, and so is the history,
document the reasons for why we cannot remove the usermode helper and
the only thing we can do is compartamentalize it.

While it, add a Coccinelle SmPL patch to help maintainers police
for *infidels* still using the usermode helper. Its accepted that perhaps
we can't get rid of all use cases, as otherwise we'd break userspace, so
best we can do is go on a hunt and fix incorrect users of it. Drivers
can only be transitioned out of the usermode helper once we know old
userspace cannot be not be broken by a kernel change.

The current SmPL patch reports:

$ export COCCI=scripts/coccinelle/api/request_firmware-usermode.cocci
$ make coccicheck MODE=report

drivers/leds/leds-lp55xx-common.c:227:8-31: WARNING: please check if user really needs the usermode helper
drivers/firmware/dell_rbu.c:622:17-40: WARNING: please check if user really needs the usermode helper

Cc: Richard Purdie <rpurdie@rpsys.net>
Cc: Jacek Anaszewski <j.anaszewski@samsung.com>
Cc: linux-leds@vger.kernel.org
Cc: Abhay Salunke <Abhay_Salunke@dell.com>
Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 Documentation/firmware_class/README                | 36 ++++++++++++++++++---
 .../coccinelle/api/request_firmware-usermode.cocci | 37 ++++++++++++++++++++++
 2 files changed, 68 insertions(+), 5 deletions(-)
 create mode 100644 scripts/coccinelle/api/request_firmware-usermode.cocci

diff --git a/Documentation/firmware_class/README b/Documentation/firmware_class/README
index 20aca5901785..9f8b2daf4c7c 100644
--- a/Documentation/firmware_class/README
+++ b/Documentation/firmware_class/README
@@ -33,7 +33,7 @@
 	than 256, user should pass 'firmware_class.path=$CUSTOMIZED_PATH'
 	if firmware_class is built in kernel(the general situation)
 
- 2), userspace:
+ 2), userspace: (AVOID)
  	- /sys/class/firmware/xxx/{loading,data} appear.
 	- hotplug gets called with a firmware identifier in $FIRMWARE
 	  and the usual hotplug environment.
@@ -41,14 +41,14 @@
 
  3), kernel: Discard any previous partial load.
 
- 4), userspace:
+ 4), userspace: (AVOID)
 		- hotplug: cat appropriate_firmware_image > \
 					/sys/class/firmware/xxx/data
 
  5), kernel: grows a buffer in PAGE_SIZE increments to hold the image as it
 	 comes in.
 
- 6), userspace:
+ 6), userspace: (AVOID)
 		- hotplug: echo 0 > /sys/class/firmware/xxx/loading
 
  7), kernel: request_firmware() returns and the driver has the firmware
@@ -66,8 +66,8 @@
 	 	copy_fw_to_device(fw_entry->data, fw_entry->size);
 	 release_firmware(fw_entry);
 
- Sample/simple hotplug script:
- ============================
+ Sample/simple hotplug script: (AVOID)
+ ==========================================
 
 	# Both $DEVPATH and $FIRMWARE are already provided in the environment.
 
@@ -116,6 +116,32 @@ If you're a maintainer you can help police this with:
 $ export COCCI=scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
 $ make coccicheck MODE=report
 
+Usermode helper
+===============
+
+The firmware API supports a usermode helper that lets userspace fetch and
+hand off firmware to a driver through sysfs. While in theory this was a
+nice feature it also presented many issues, so in practice it should be
+avoided at all costs now.
+
+Only drivers that have a really good reason for a usermode helper should
+use it. This use case must be well documented. You should try really hard
+to avoid it, at all costs.
+
+CONFIG_FW_LOADER_USER_HELPER_FALLBACK: disabled by most distributions now
+CONFIG_FW_LOADER_USER_HELPER: still enabled by most disributions
+
+When CONFIG_FW_LOADER_USER_HELPER_FALLBACK is enabled request_firmware()
+*always* calls the usermode helpers. When CONFIG_FW_LOADER_USER_HELPER is
+enabled, only if you specifically ask for it wil the usermode helper be
+called. If CONFIG_FW_LOADER_USER_HELPER_FALLBACK is disabled drivers can
+still require the usermode helper by using request_firmware_nowait().
+
+To help police for user of the usermode helper you can use:
+
+$ export COCCI=scripts/coccinelle/api/request_firmware-avoid-init-probe-init.cocci
+$ make coccicheck MODE=report
+
  about in-kernel persistence:
  ---------------------------
  Under some circumstances, as explained below, it would be interesting to keep
diff --git a/scripts/coccinelle/api/request_firmware-usermode.cocci b/scripts/coccinelle/api/request_firmware-usermode.cocci
new file mode 100644
index 000000000000..94ab95cb7c75
--- /dev/null
+++ b/scripts/coccinelle/api/request_firmware-usermode.cocci
@@ -0,0 +1,37 @@
+// Avoid the usermode helper at all costs
+//
+// request_firmware_nowait() API enables explicit request for the
+// usermode helper. Chances are high its use is just a copy and
+// paste bug. Before you fix the driver be sure to *verify* no
+// usermode helper tool exists that would otherwise break if
+// we replace the driver to use a non-usermode helper variant.
+//
+// Confidence: High
+//
+// Reason for low confidence:
+//
+// Copyright: (C) 2016 Luis R. Rodriguez <mcgrof@kernel.org> GPLv2.
+//
+// Options: --include-headers
+
+virtual report
+virtual context
+
+@ r1 depends on report || context @
+expression mod, name, dev, gfp, drv, cb;
+position p;
+@@
+
+(
+*request_firmware_nowait@p(mod, false, name, dev, gfp, drv, cb)
+|
+*request_firmware_nowait@p(mod, 0, name, dev, gfp, drv, cb)
+|
+*request_firmware_nowait@p(mod, FW_ACTION_NOHOTPLUG, name, dev, gfp, drv, cb)
+)
+
+@script:python depends on report@
+p << r1.p;
+@@
+
+coccilib.report.print_report(p[0], "WARNING: please check if user really needs the usermode helper")
-- 
2.9.2

^ permalink raw reply	[flat|nested] 80+ messages in thread

* [PATCH v4 4/5] firmware: add usermode helper DECLARE_FW_LOADER_USER() annotation
  2016-09-07  0:42   ` [PATCH v4 0/5] firmware: add SmPL grammar to avoid issues Luis R. Rodriguez
                       ` (2 preceding siblings ...)
  2016-09-07  0:42     ` [PATCH v4 3/5] firmware: update usermode helper docs and add SmPL report Luis R. Rodriguez
@ 2016-09-07  0:42     ` Luis R. Rodriguez
  2016-09-07  0:42     ` [PATCH v4 5/5] firmware: fix fw cache to avoid usermode helper on suspend Luis R. Rodriguez
  4 siblings, 0 replies; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-09-07  0:42 UTC (permalink / raw)
  To: ming.lei, akpm, gregkh
  Cc: daniel.wagner, mmarek, linux-kernel, markivx, stephen.boyd,
	zohar, broonie, tiwai, johannes, chunkeey, hauke, jwboyer,
	dmitry.torokhov, dwmw2, jslaby, torvalds, luto, fengguang.wu,
	rpurdie, j.anaszewski, Abhay_Salunke, Julia.Lawall,
	Gilles.Muller, nicolas.palix, teg, dhowells, bjorn.andersson,
	arend.vanspriel, kvalo, Luis R. Rodriguez, linux-leds

We need to ensure no one else adds *anything* that requests the
usermode helper without really meaning it, to police it we now have
an SmPL script but no formal annotation is present to help us ensure
the call has been validated.

Add a dummy DECLARE_FW_LOADER_USER() which we can use to annotate
validated calls and also clearly outline the documentation over where
the required usermode helper is documented.

All other future callers can be reported via 0-day by making use of the
Coccinelle SmPL patch.

Cc: Fengguang Wu <fengguang.wu@intel.com>
Cc: Richard Purdie <rpurdie@rpsys.net>
Cc: Jacek Anaszewski <j.anaszewski@samsung.com>
Cc: linux-leds@vger.kernel.org
Cc: Abhay Salunke <Abhay_Salunke@dell.com>
Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 drivers/firmware/dell_rbu.c                            | 1 +
 drivers/leds/leds-lp55xx-common.c                      | 1 +
 include/linux/firmware.h                               | 7 +++++++
 scripts/coccinelle/api/request_firmware-usermode.cocci | 9 ++++++++-
 4 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/drivers/firmware/dell_rbu.c b/drivers/firmware/dell_rbu.c
index 2f452f1f7c8a..53b3869c0900 100644
--- a/drivers/firmware/dell_rbu.c
+++ b/drivers/firmware/dell_rbu.c
@@ -586,6 +586,7 @@ static ssize_t read_rbu_image_type(struct file *filp, struct kobject *kobj,
 	return size;
 }
 
+DECLARE_FW_LOADER_USER("Documentation/dell_rbu.txt");
 static ssize_t write_rbu_image_type(struct file *filp, struct kobject *kobj,
 				    struct bin_attribute *bin_attr,
 				    char *buffer, loff_t pos, size_t count)
diff --git a/drivers/leds/leds-lp55xx-common.c b/drivers/leds/leds-lp55xx-common.c
index 5377f22ff994..ce1d087979a1 100644
--- a/drivers/leds/leds-lp55xx-common.c
+++ b/drivers/leds/leds-lp55xx-common.c
@@ -219,6 +219,7 @@ out:
 	release_firmware(chip->fw);
 }
 
+DECLARE_FW_LOADER_USER("Documentation/leds/leds-lp55xx.txt");
 static int lp55xx_request_firmware(struct lp55xx_chip *chip)
 {
 	const char *name = chip->cl->name;
diff --git a/include/linux/firmware.h b/include/linux/firmware.h
index b1f9f0ccb8ac..147b3062039b 100644
--- a/include/linux/firmware.h
+++ b/include/linux/firmware.h
@@ -8,6 +8,13 @@
 #define FW_ACTION_NOHOTPLUG 0
 #define FW_ACTION_HOTPLUG 1
 
+/*
+ * Helper for scripts/coccinelle/api/request_firmware-usermode.cocci
+ * and so users can also easily search for the respectively needed
+ * usermode helper.
+ */
+#define DECLARE_FW_LOADER_USER(__usermode_helper)
+
 struct firmware {
 	size_t size;
 	const u8 *data;
diff --git a/scripts/coccinelle/api/request_firmware-usermode.cocci b/scripts/coccinelle/api/request_firmware-usermode.cocci
index 94ab95cb7c75..1775fd3067e8 100644
--- a/scripts/coccinelle/api/request_firmware-usermode.cocci
+++ b/scripts/coccinelle/api/request_firmware-usermode.cocci
@@ -17,6 +17,13 @@
 virtual report
 virtual context
 
+@ r0 depends on report || context @
+declarer name DECLARE_FW_LOADER_USER;
+expression E;
+@@
+
+DECLARE_FW_LOADER_USER(E);
+
 @ r1 depends on report || context @
 expression mod, name, dev, gfp, drv, cb;
 position p;
@@ -30,7 +37,7 @@ position p;
 *request_firmware_nowait@p(mod, FW_ACTION_NOHOTPLUG, name, dev, gfp, drv, cb)
 )
 
-@script:python depends on report@
+@script:python depends on report && !r0 @
 p << r1.p;
 @@
 
-- 
2.9.2

^ permalink raw reply	[flat|nested] 80+ messages in thread

* [PATCH v4 5/5] firmware: fix fw cache to avoid usermode helper on suspend
  2016-09-07  0:42   ` [PATCH v4 0/5] firmware: add SmPL grammar to avoid issues Luis R. Rodriguez
                       ` (3 preceding siblings ...)
  2016-09-07  0:42     ` [PATCH v4 4/5] firmware: add usermode helper DECLARE_FW_LOADER_USER() annotation Luis R. Rodriguez
@ 2016-09-07  0:42     ` Luis R. Rodriguez
  4 siblings, 0 replies; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-09-07  0:42 UTC (permalink / raw)
  To: ming.lei, akpm, gregkh
  Cc: daniel.wagner, mmarek, linux-kernel, markivx, stephen.boyd,
	zohar, broonie, tiwai, johannes, chunkeey, hauke, jwboyer,
	dmitry.torokhov, dwmw2, jslaby, torvalds, luto, fengguang.wu,
	rpurdie, j.anaszewski, Abhay_Salunke, Julia.Lawall,
	Gilles.Muller, nicolas.palix, teg, dhowells, bjorn.andersson,
	arend.vanspriel, kvalo, Luis R. Rodriguez

The firmware cache purposely kills all non-udev (usermode helper)
pending requests prior to suspend with kill_requests_without_uevent()
right before it calls out to request for firmware for the fw cache.
It is pointless to again run into the possible issue of queing up
further usermode helpers during suspend, furthermore its actually
buggy to have required the usermode helper in some cases where
clearly the driver originally had not wanted that. Fix this by
simply using the direct call.

This doesn't fix any known bug however if it should be an optimization
for suspend/resume. While at it extend documentation to ensure folks of
the usermode helper are aware that they must cache the firmware on their
own for suspend / resume.

v3: patch introduced in this series

v4: as per Daniel Wagner since we're removing the usermode helper
    caller on the cache we can remove the timeout resetting on the
    cache code now as its is pointless.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 Documentation/firmware_class/README |  3 +++
 drivers/base/firmware_class.c       | 16 +---------------
 2 files changed, 4 insertions(+), 15 deletions(-)

diff --git a/Documentation/firmware_class/README b/Documentation/firmware_class/README
index 9f8b2daf4c7c..173234005f33 100644
--- a/Documentation/firmware_class/README
+++ b/Documentation/firmware_class/README
@@ -174,3 +174,6 @@ $ make coccicheck MODE=report
  resume callback, and callers needn't cache the firmware by
  themselves any more for dealing with firmware loss during system
  resume.
+
+The firmware cache is only for non-user mode helper users. Drivers that
+require the usermode helper must deal with caching on their own.
diff --git a/drivers/base/firmware_class.c b/drivers/base/firmware_class.c
index 22d1760a4278..960f8f7c7aa2 100644
--- a/drivers/base/firmware_class.c
+++ b/drivers/base/firmware_class.c
@@ -1412,7 +1412,7 @@ static int cache_firmware(const char *fw_name)
 
 	pr_debug("%s: %s\n", __func__, fw_name);
 
-	ret = request_firmware(&fw, fw_name, NULL);
+	ret = request_firmware_direct(&fw, fw_name, NULL);
 	if (!ret)
 		kfree(fw);
 
@@ -1622,7 +1622,6 @@ static void __device_uncache_fw_images(void)
 static void device_cache_fw_images(void)
 {
 	struct firmware_cache *fwc = &fw_cache;
-	int old_timeout;
 	DEFINE_WAIT(wait);
 
 	pr_debug("%s\n", __func__);
@@ -1630,17 +1629,6 @@ static void device_cache_fw_images(void)
 	/* cancel uncache work */
 	cancel_delayed_work_sync(&fwc->work);
 
-	/*
-	 * use small loading timeout for caching devices' firmware
-	 * because all these firmware images have been loaded
-	 * successfully at lease once, also system is ready for
-	 * completing firmware loading now. The maximum size of
-	 * firmware in current distributions is about 2M bytes,
-	 * so 10 secs should be enough.
-	 */
-	old_timeout = loading_timeout;
-	loading_timeout = 10;
-
 	mutex_lock(&fw_lock);
 	fwc->state = FW_LOADER_START_CACHE;
 	dpm_for_each_dev(NULL, dev_cache_fw_image);
@@ -1648,8 +1636,6 @@ static void device_cache_fw_images(void)
 
 	/* wait for completion of caching firmware for all devices */
 	async_synchronize_full_domain(&fw_cache_domain);
-
-	loading_timeout = old_timeout;
 }
 
 /**
-- 
2.9.2

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v4 1/5] MAINTAINERS: extend firmware_class maintainer list
  2016-09-07  0:42     ` [PATCH v4 1/5] MAINTAINERS: extend firmware_class maintainer list Luis R. Rodriguez
@ 2016-09-07  6:43       ` Greg KH
  2016-09-08 14:58         ` Luis R. Rodriguez
  2016-09-08 15:25         ` Ming Lei
  0 siblings, 2 replies; 80+ messages in thread
From: Greg KH @ 2016-09-07  6:43 UTC (permalink / raw)
  To: Luis R. Rodriguez
  Cc: ming.lei, akpm, daniel.wagner, mmarek, linux-kernel, markivx,
	stephen.boyd, zohar, broonie, tiwai, johannes, chunkeey, hauke,
	jwboyer, dmitry.torokhov, dwmw2, jslaby, torvalds, luto,
	fengguang.wu, rpurdie, j.anaszewski, Abhay_Salunke, Julia.Lawall,
	Gilles.Muller, nicolas.palix, teg, dhowells, bjorn.andersson,
	arend.vanspriel, kvalo

On Tue, Sep 06, 2016 at 05:42:06PM -0700, Luis R. Rodriguez wrote:
> I've been reviewing changes proactively, and plan on doing
> more of this work. I'm doing this early as I should be getting
> e-mailed about proposed changes.
> 
> Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
> ---
>  MAINTAINERS | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index fcb387135242..af64eb999ef4 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -4891,6 +4891,7 @@ F:	tools/firewire/
>  
>  FIRMWARE LOADER (request_firmware)
>  M:	Ming Lei <ming.lei@canonical.com>
> +M:	Luis R. Rodriguez <mcgrof@kernel.org>

What does Ming think of this?  :)

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v4 1/5] MAINTAINERS: extend firmware_class maintainer list
  2016-09-07  6:43       ` Greg KH
@ 2016-09-08 14:58         ` Luis R. Rodriguez
  2016-09-08 15:25         ` Ming Lei
  1 sibling, 0 replies; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-09-08 14:58 UTC (permalink / raw)
  To: Greg KH, ming.lei
  Cc: Luis R. Rodriguez, akpm, daniel.wagner, mmarek, linux-kernel,
	markivx, stephen.boyd, zohar, broonie, tiwai, johannes, chunkeey,
	hauke, jwboyer, dmitry.torokhov, dwmw2, jslaby, torvalds, luto,
	fengguang.wu, rpurdie, j.anaszewski, Abhay_Salunke, Julia.Lawall,
	Gilles.Muller, nicolas.palix, teg, dhowells, bjorn.andersson,
	arend.vanspriel, kvalo

On Wed, Sep 07, 2016 at 08:43:11AM +0200, Greg KH wrote:
> On Tue, Sep 06, 2016 at 05:42:06PM -0700, Luis R. Rodriguez wrote:
> > I've been reviewing changes proactively, and plan on doing
> > more of this work. I'm doing this early as I should be getting
> > e-mailed about proposed changes.
> > 
> > Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
> > ---
> >  MAINTAINERS | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index fcb387135242..af64eb999ef4 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -4891,6 +4891,7 @@ F:	tools/firewire/
> >  
> >  FIRMWARE LOADER (request_firmware)
> >  M:	Ming Lei <ming.lei@canonical.com>
> > +M:	Luis R. Rodriguez <mcgrof@kernel.org>
> 
> What does Ming think of this?  :)

Ming?

  Luis

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v4 1/5] MAINTAINERS: extend firmware_class maintainer list
  2016-09-07  6:43       ` Greg KH
  2016-09-08 14:58         ` Luis R. Rodriguez
@ 2016-09-08 15:25         ` Ming Lei
  1 sibling, 0 replies; 80+ messages in thread
From: Ming Lei @ 2016-09-08 15:25 UTC (permalink / raw)
  To: Greg KH
  Cc: Luis R. Rodriguez, Andrew Morton, Daniel Wagner, mmarek,
	Linux Kernel Mailing List, Vikram Mulukutla, Stephen Boyd,
	Mimi Zohar, Mark Brown, Takashi Iwai, Johannes Berg, chunkeey,
	hauke, Josh Boyer, Dmitry Torokhov, David Woodhouse, Jiri Slaby,
	Linus Torvalds, Andy Lutomirski, Wu Fengguang, rpurdie,
	j.anaszewski, Abhay Salunke, Julia Lawall, Gilles.Muller,
	nicolas.palix, Tom Gundersen, David Howells, bjorn.andersson,
	arend.vanspriel, kvalo

On Wed, Sep 7, 2016 at 2:43 PM, Greg KH <gregkh@linuxfoundation.org> wrote:
> On Tue, Sep 06, 2016 at 05:42:06PM -0700, Luis R. Rodriguez wrote:
>> I've been reviewing changes proactively, and plan on doing
>> more of this work. I'm doing this early as I should be getting
>> e-mailed about proposed changes.
>>
>> Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
>> ---
>>  MAINTAINERS | 1 +
>>  1 file changed, 1 insertion(+)
>>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index fcb387135242..af64eb999ef4 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -4891,6 +4891,7 @@ F:      tools/firewire/
>>
>>  FIRMWARE LOADER (request_firmware)
>>  M:   Ming Lei <ming.lei@canonical.com>
>> +M:   Luis R. Rodriguez <mcgrof@kernel.org>
>
> What does Ming think of this?  :)

Acked-by: Ming Lei <ming.lei@canonical.com>

Thanks,

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-09-03  0:20             ` [RFC] fs: add userspace critical mounts event support Luis R. Rodriguez
  2016-09-03  4:11               ` Linus Torvalds
@ 2016-09-14  2:38               ` Rob Landley
  2016-10-05 18:00                 ` Luis R. Rodriguez
  1 sibling, 1 reply; 80+ messages in thread
From: Rob Landley @ 2016-09-14  2:38 UTC (permalink / raw)
  To: Luis R. Rodriguez, Daniel Vetter, Mimi Zohar, Felix Fietkau,
	David Woodhouse, Roman Pen, Bjorn Andersson, Ming Lei,
	Andrew Morton, Michal Marek, Greg KH
  Cc: Linux Kernel Mailing List, Vikram Mulukutla, Stephen Boyd,
	Mark Brown, Takashi Iwai, Johannes Berg, Christian Lamparter,
	hauke, Josh Boyer, Dmitry Torokhov, jslaby, Linus Torvalds,
	Andy Lutomirski, Wu Fengguang, rpurdie, Jeff Mahoney,
	j.anaszewski, Abhay_Salunke, Julia Lawall, Gilles.Muller,
	nicolas.palix, Tom Gundersen, Kay Sievers, David Howells,
	Alessandro Rubini, Kevin Cernekee, Kees Cook, Jonathan Corbet,
	Thierry Martinez, cocci, linux-serial, linux-doc, linuxppc-dev

On 09/02/2016 07:20 PM, Luis R. Rodriguez wrote:
> kernel_read_file_from_path() can try to read a file from
> the system's filesystem. This is typically done for firmware
> for instance, which lives in /lib/firmware. One issue with
> this is that the kernel cannot know for sure when the real
> final /lib/firmare/ is ready, and even if you use initramfs
> drivers are currently initialized *first* prior to the initramfs
> kicking off.

Why?

> During init we run through all init calls first
> (do_initcalls()) and finally the initramfs is processed via
> prepare_namespace():

What's the downside of moving initramfs cpio extraction earlier in the boot?

I did some shuffling around of those code to make initmpfs work, does
anybody know why initramfs extraction _before_ we initialize drivers
would be a bad thing? (The cpio is in memory, either linked into the
kernel or from the bootloader. No drivers are needed to extract it,
that's sort of the point.)

The only things I can think of are memory churn (large contiguous
physical page allocations), or if a driver somehow got us access to more
physical memory?

Rob

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-09-03 18:10                         ` Dmitry Torokhov
  2016-09-06 21:52                           ` Luis R. Rodriguez
@ 2016-09-24  1:37                           ` Herbert, Marc
  2016-09-24 17:41                             ` Dmitry Torokhov
  1 sibling, 1 reply; 80+ messages in thread
From: Herbert, Marc @ 2016-09-24  1:37 UTC (permalink / raw)
  To: linux-kernel; +Cc: linux-doc, cocci, linuxppc-dev, linux-serial

On 03/09/2016 11:10, Dmitry Torokhov wrote:
> I was thinking if we kernel could post
> "conditions" (maybe simple stings) that it waits for, and userspace
> could unlock these "conditions". One of them might be "firmware
> available".

On idea offered by Josh Triplett that seems to overlap with this one
is to have something similar to the (deprecated) userhelper with
*per-blob* requests and notifications except for one major difference:
userspace would not anymore be in charge of *providing* the blob but
would instead only *signal* when a given blob becomes available and is
either found or found missing. Then the kernel loads the blob _by
itself_; unlike the userhelper. No new “critical filesystem” concept
and a *per-blob basis*, allowing any variation of blob locations
across any number of initramfs and filesystems.

Could this one fly?

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-09-06 23:04                         ` Luis R. Rodriguez
@ 2016-09-24  2:51                           ` Herbert, Marc
  2016-10-04 23:28                             ` Luis R. Rodriguez
  0 siblings, 1 reply; 80+ messages in thread
From: Herbert, Marc @ 2016-09-24  2:51 UTC (permalink / raw)
  To: Luis R. Rodriguez, Linus Torvalds
  Cc: Bjorn Andersson, Daniel Vetter, Mimi Zohar, Felix Fietkau,
	David Woodhouse, Roman Pen, Ming Lei, Andrew Morton,
	Michal Marek, Greg KH, Linux Kernel Mailing List,
	Vikram Mulukutla, Stephen Boyd, Mark Brown, Takashi Iwai,
	Johannes Berg, Christian Lamparter, Hauke Mehrtens

On 06/09/2016 16:04, Luis R. Rodriguez wrote:
> They claim that without it there is the race between /lib/firmware
> being ready and driver asking for the firmware.

Hope it's understood by now.

> I was told there were quite a bit of out-of-tree hacks to address
> this without using the usermode helper,

There are:
https://chromium-review.googlesource.com/#/c/354089/
 wait until SYSTEM_RUNNING before loading DMC firmware.

> the goal of this patch was to create the discussion needed to a
> proper resolution to this.

Sincere thanks.

>>> On Tue 06 Sep 11:32 PDT 2016, Linus Torvalds wrote:
>>>
>>>> On Tue, Sep 6, 2016 at 10:46 AM, Bjorn Andersson
>>>> Nobody has actually answered the "why don't we just tie the
>>>> firmware and module together" question.
>>>
>>> The answer to this depends on the details of the suggestion; but
>>> generally there's a much stronger bond between the kernel and the
>>> driver than between the driver and the firmware in my cases.

Indeed.

The i915 DMC firmware is an interesting example. First of all it’s
_optional_!  It’s critical for battery-powered systems but the i915
driver works without it.

Dan wrote:
> Plus all gpu drivers which need firmware. And yes we must load them
> at probe because people are generally pissed when they boot their
> machine and the screen goes black. On top of that a lot of people
> want their gpu drivers to be built-in, but can't ship the firmware
> blobs in the kernel image because gpl. Yep, there's a bit a
> contradiction there ...

Eppur si muove:
1) As Dan just wrote, users expect the screen to light up as soon as they
press the power button so the i915 driver is built-in
2) ... yet they’ll never notice the nanojoules of battery loss caused
by the DMC firmware being on a filesystem and loaded a tiny bit later.

SoCs and platforms have become some new kind of distributed systems
where other processors run their own, specific software/OS/firmware.
>From this perspective the kernel plays a role similar to a boot server;
and choke point. Granted: booting various and heterogeneous
distributed systems doesn’t look like a simple problem to solve
generically. Yet at the moment the kernel  doesn’t help by not
even supporting something as basic as being told when the files it’s
(unfortunately) in charge to deploy to other nodes become available and
ready to deploy.

It can’t be assumed that the driver and the firmware are two parts of
the same software piece whereas they actually run on two different
processors, are most likely developed and validated by completely
different teams and released on different lifecycles. Especially in
the Linux case.

I hope this distributed systems analogy captures the essence of the
examples and rationales detailed elsewhere in this thread.

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-09-24  1:37                           ` Herbert, Marc
@ 2016-09-24 17:41                             ` Dmitry Torokhov
  2016-10-05  0:00                               ` Luis R. Rodriguez
  0 siblings, 1 reply; 80+ messages in thread
From: Dmitry Torokhov @ 2016-09-24 17:41 UTC (permalink / raw)
  To: Herbert, Marc
  Cc: Linus Torvalds, open list:DOCUMENTATION, cocci, Jacek Anaszewski,
	David Woodhouse, Christian Lamparter, Julia Lawall,
	Andrew Morton, linuxppc-dev, Mimi Zohar, Andy Lutomirski,
	Richard Purdie, Wu Fengguang, Johannes Berg, Luis R. Rodriguez,
	Michal Marek, Hauke Mehrtens, Mark Brown, Jiri Slaby, Ming Lei,
	Daniel Vetter, Bjorn Andersson, Felix Fietkau, Roman Pen,
	Greg KH, Linux Kernel Mailing List, Vikram Mulukutla,
	Stephen Boyd, Takashi Iwai

On Fri, Sep 23, 2016 at 6:37 PM, Herbert, Marc <marc.herbert@intel.com> wrote:
> On 03/09/2016 11:10, Dmitry Torokhov wrote:
>> I was thinking if we kernel could post
>> "conditions" (maybe simple stings) that it waits for, and userspace
>> could unlock these "conditions". One of them might be "firmware
>> available".
>
> On idea offered by Josh Triplett that seems to overlap with this one
> is to have something similar to the (deprecated) userhelper with
> *per-blob* requests and notifications except for one major difference:
> userspace would not anymore be in charge of *providing* the blob but
> would instead only *signal* when a given blob becomes available and is
> either found or found missing. Then the kernel loads the blob _by
> itself_; unlike the userhelper. No new “critical filesystem” concept
> and a *per-blob basis*, allowing any variation of blob locations
> across any number of initramfs and filesystems.
>

Really, I do not quite understand why people have issues with usermode
helper/uevents. It used to work reasonably well (if you were using
request_firmware_nowait()), as the kernel would post the request and
then, when userspace was ready[^Hier], uevents would be processed and
firmware would be loaded. We had a timeout of 60(?) seconds by
default, but that would be adjusted as systems needed.

Unfortunately it all broke when udev started insisting [1] on
servicing some uevents in strict sequence, which resulted in boot
stalls. Maybe the ultimate answer is to write a firmware loading
daemon that would also listen to netlink events and do properly what
udev refused to be doing? The distribution would know when it is ready
to service firmware requests (and thus when to start this daemon), and
we would have the freedom of having drivers both built-in and as
modules and bulding firmware into kernel, intiramfs or keep on a
"real" fs available at later time.

Thanks.

-- 
Dmitry

[1] https://lwn.net/Articles/518942/

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-09-24  2:51                           ` Herbert, Marc
@ 2016-10-04 23:28                             ` Luis R. Rodriguez
  0 siblings, 0 replies; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-10-04 23:28 UTC (permalink / raw)
  To: Herbert, Marc
  Cc: Luis R. Rodriguez, Linus Torvalds, Bjorn Andersson,
	Daniel Vetter, Mimi Zohar, Felix Fietkau, David Woodhouse,
	Roman Pen, Ming Lei, Andrew Morton, Michal Marek, Greg KH,
	Linux Kernel Mailing List, Vikram Mulukutla, Stephen Boyd,
	Mark Brown, Takashi Iwai, Johannes Berg, Christian Lamparter,
	Hauke Mehrtens, Jeff Mahoney

On Fri, Sep 23, 2016 at 07:51:41PM -0700, Herbert, Marc wrote:
> On 06/09/2016 16:04, Luis R. Rodriguez wrote:
> > They claim that without it there is the race between /lib/firmware
> > being ready and driver asking for the firmware.
> 
> Hope it's understood by now.
> 
> > I was told there were quite a bit of out-of-tree hacks to address
> > this without using the usermode helper,
> 
> There are:
> https://chromium-review.googlesource.com/#/c/354089/
>  wait until SYSTEM_RUNNING before loading DMC firmware.

Jeesh. Good thing its not merged yet upstream, but indeed
I can understand why out of tree kernels are picking these
sorts of solutions up in the meantime.

> > the goal of this patch was to create the discussion needed to a
> > proper resolution to this.
> 
> Sincere thanks.
> 
> >>> On Tue 06 Sep 11:32 PDT 2016, Linus Torvalds wrote:
> >>>
> >>>> On Tue, Sep 6, 2016 at 10:46 AM, Bjorn Andersson
> >>>> Nobody has actually answered the "why don't we just tie the
> >>>> firmware and module together" question.
> >>>
> >>> The answer to this depends on the details of the suggestion; but
> >>> generally there's a much stronger bond between the kernel and the
> >>> driver than between the driver and the firmware in my cases.
> 
> Indeed.
> 
> The i915 DMC firmware is an interesting example. First of all it’s
> _optional_!  It’s critical for battery-powered systems but the i915
> driver works without it.

You obviously may want to upgrade firmware too, and a driver may
want to provide support for a series of old and new firmware.

An alternative idea hinted to me recently was a new system call for
drivers that need firmware early, so we'd have system call deal with
loading *both* the module and firmware -- but indeed optional firmware is one
possible issue that throws a wrench into this. We can surely add a flags
option but not yet sure if that alone would suffice for most of our needs.
You may need code to generate the firmware name dynamically as well, so a
system call would only be useful for a few cases where firmware requirement
information can be inferred by userspace by just looking at the module object.

> Dan wrote:
> > Plus all gpu drivers which need firmware. And yes we must load them
> > at probe because people are generally pissed when they boot their
> > machine and the screen goes black.

Thanks for the clarification BTW.

> > On top of that a lot of people
> > want their gpu drivers to be built-in, but can't ship the firmware
> > blobs in the kernel image because gpl. Yep, there's a bit a
> > contradiction there ...
> 
> Eppur si muove:
> 1) As Dan just wrote, users expect the screen to light up as soon as they
> press the power button so the i915 driver is built-in
> 2) ... yet they’ll never notice the nanojoules of battery loss caused
> by the DMC firmware being on a filesystem and loaded a tiny bit later.
> 
> SoCs and platforms have become some new kind of distributed systems
> where other processors run their own, specific software/OS/firmware.
> From this perspective the kernel plays a role similar to a boot server;
> and choke point. Granted: booting various and heterogeneous
> distributed systems doesn’t look like a simple problem to solve
> generically. Yet at the moment the kernel  doesn’t help by not
> even supporting something as basic as being told when the files it’s
> (unfortunately) in charge to deploy to other nodes become available and
> ready to deploy.

When you consider the problem more from a directed acyclic graph point of view
you soon realize the issue is really about the *need* for certain files upon
driver load and the lack of of semantics for a deterministic assurance that
when we look for files its a valid hunt. What you describe in terms of
SoCs is just that the complexity of the DAG increases considerably.

> It can’t be assumed that the driver and the firmware are two parts of
> the same software piece whereas they actually run on two different
> processors, are most likely developed and validated by completely
> different teams and released on different lifecycles. Especially in
> the Linux case.
> 
> I hope this distributed systems analogy captures the essence of the
> examples and rationales detailed elsewhere in this thread.

You also need to upgrade firmware, and users should be able to opt-in
for firmware, and pick any firmware, or roll back to older versions
as they see fit.

  Luis

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-09-24 17:41                             ` Dmitry Torokhov
@ 2016-10-05  0:00                               ` Luis R. Rodriguez
  2016-10-05  0:12                                 ` Linus Torvalds
  0 siblings, 1 reply; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-10-05  0:00 UTC (permalink / raw)
  To: Dmitry Torokhov
  Cc: Herbert, Marc, Linus Torvalds, open list:DOCUMENTATION,
	Jacek Anaszewski, David Woodhouse, Christian Lamparter,
	Julia Lawall, Andrew Morton, linuxppc-dev, Mimi Zohar,
	Andy Lutomirski, Richard Purdie, Wu Fengguang, Johannes Berg,
	Luis R. Rodriguez, Michal Marek, Hauke Mehrtens, Mark Brown,
	Jiri Slaby, Ming Lei, Daniel Vetter, Bjorn Andersson,
	Felix Fietkau, Roman Pen, Greg KH, Linux Kernel Mailing List,
	Vikram Mulukutla, Stephen Boyd, Takashi Iwai, Jeff Mahoney,
	Hariprasad S, Benjamin Poirier, Josh Triplett

On Sat, Sep 24, 2016 at 10:41:46AM -0700, Dmitry Torokhov wrote:
> On Fri, Sep 23, 2016 at 6:37 PM, Herbert, Marc <marc.herbert@intel.com> wrote:
> > On 03/09/2016 11:10, Dmitry Torokhov wrote:
> >> I was thinking if we kernel could post
> >> "conditions" (maybe simple stings) that it waits for, and userspace
> >> could unlock these "conditions". One of them might be "firmware
> >> available".
> >
> > On idea offered by Josh Triplett that seems to overlap with this one
> > is to have something similar to the (deprecated) userhelper with
> > *per-blob* requests and notifications except for one major difference:
> > userspace would not anymore be in charge of *providing* the blob but
> > would instead only *signal* when a given blob becomes available and is
> > either found or found missing. Then the kernel loads the blob _by
> > itself_; unlike the userhelper. No new “critical filesystem” concept
> > and a *per-blob basis*, allowing any variation of blob locations
> > across any number of initramfs and filesystems.
> >
> 
> Really, I do not quite understand why people have issues with usermode
> helper/uevents.

One reason is you'd have to implement your own cache for suspend/resume.

> It used to work reasonably well (if you were using
> request_firmware_nowait()), as the kernel would post the request and
> then, when userspace was ready[^Hier], uevents would be processed and
> firmware would be loaded. We had a timeout of 60(?) seconds by
> default, but that would be adjusted as systems needed.

The issue with the timeout was kernel developers *assumed* module init
and probe were detached, and saying 'thou shall not load firmware on
probe' seems actually like a more radical change than just saying
'thou shall load firmware on init'. I'll note that as it stands
its the right thing to complain about these users only because we
lack the semantics to ensure correctness if used on init or probe.
The timeout incurred huge latencies for optional firmwares, and
while we had a new API added to avoid the wait on optional firmware,
that obviously still leaved the races as possible. We now have async
probe which *does* enable some original misconceptions by kernel
developers, but by now other issues have also been found on the
usermode helper, the cache was one, another one was a recent discusion
over the user of the UMH lock with the assumption this was providing
a sort of safeguard on early boot use -- it does not, for the same
exact reasons why a UMH lock does not suffice to avoid all possible
rootfs races. For this later issue refer to a recent discussion in
review with Daniel Wagner's patches.

> Unfortunately it all broke when udev started insisting [1] on
> servicing some uevents in strict sequence, which resulted in boot
> stalls.

That was not the only issue... another implicit issue was that
you are reducing the number of possible supported number of
devices Linux supports per module by the timeout, it would
depend on the combine time it takes to both init and probe.
Some drivers are super complex and even if you *don't* have
firmware requirements and say burn the firmware onto a device
we found that *probe* alone was taking a long long time on some
device drivers -- check out cxgb4 driver, where one device actually
ends up loading about 4 subdevices underneath it. Yes that's a mess
and the driver needs a major rewrite to address this in a clean way
but that takes time. Its no trivial pursuit. The umh timeout then
would not be implicated anymore *but* since systemd implemented the
timeout in general for kmod loading it did mean system was limiting
them Linux drivers and how much devices a driver can support
depending on this timeout value. At SUSE we solved this by lifting
this timeout for kmod workers for now. A long term goal here, which
could help, is also to just detach init and probes, so we give to
system what it originally thought. Summary of this all is here:

http://www.do-not-panic.com/2015/12/linux-asynchronous-probe.html

I have some code that starts to enable some of this on systemd/kmod
but it still needs some more testing before I post.

> Maybe the ultimate answer is to write a firmware loading
> daemon that would also listen to netlink events and do properly what
> udev refused to be doing?

Meh, in the wireless subsystem we devised our own file loader,
check CRDA. That worked for us since we needed to optionally
enable digital RSA signed file checking, but long term our
experience is that this is pointless. So we're going to phase
that out in favor of using the firmware API for the file loading
of this file, and support then digital signatures on the firmware.

I am not sure how/why a firmware loading daemon would be a better
idea now. What Marc describes that Josh proposed with signals for
userspcae seems more aligned with what we likely need -- but note
that since we now use a shared common API for kernel reads from a
path via kernel_read_file_from_path() we'd probably want something
like a notifier for any kernel_read_file_from_path() user. The ability
for the kernel to register a generic userspace notifier seems worthy
of consideration, but I'd be surprised if we don't already have
something quite like this already?

> The distribution would know when it is ready
> to service firmware requests (and thus when to start this daemon), and
> we would have the freedom of having drivers both built-in and as
> modules and bulding firmware into kernel, intiramfs or keep on a
> "real" fs available at later time.

What difference would there be if we just used notifications to
guarantee to the kernel the file in question is now available?

  Luis

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-10-05  0:00                               ` Luis R. Rodriguez
@ 2016-10-05  0:12                                 ` Linus Torvalds
  2016-10-05  0:24                                   ` Luis R. Rodriguez
  2016-10-05  1:48                                   ` Josh Triplett
  0 siblings, 2 replies; 80+ messages in thread
From: Linus Torvalds @ 2016-10-05  0:12 UTC (permalink / raw)
  To: Luis R. Rodriguez
  Cc: Dmitry Torokhov, Herbert, Marc, open list:DOCUMENTATION,
	Jacek Anaszewski, David Woodhouse, Christian Lamparter,
	Julia Lawall, Andrew Morton, linuxppc-dev, Mimi Zohar,
	Andy Lutomirski, Richard Purdie, Wu Fengguang, Johannes Berg,
	Michal Marek, Hauke Mehrtens, Mark Brown, Jiri Slaby, Ming Lei,
	Daniel Vetter, Bjorn Andersson, Felix Fietkau, Roman Pen,
	Greg KH, Linux Kernel Mailing List, Vikram Mulukutla,
	Stephen Boyd, Takashi Iwai, Jeff Mahoney, Hariprasad S,
	Benjamin Poirier, Josh Triplett

On Tue, Oct 4, 2016 at 5:00 PM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
>
> I am not sure how/why a firmware loading daemon would be a better
> idea now. What Marc describes that Josh proposed with signals for
> userspcae seems more aligned with what we likely need

Quite frankly, I doubt you want a signal.

You will want to have some way to specify where the firmware files
are. Right now we have "fw_path[]" which is hardcoded except for the
first entry that can be set as a module parameter. But you'd probably
want to expand on that, which implies some /sys or /proc interface.

And once you do that, wouldn't it make more sense to just make the
"update the firmware path /proc/sys/kernel/fw_path file" make things
re-search for firmware?

In other words, the interface has to be something *sensible*. Not some
idiotic ad-hoc "send a signal" (of which that stupid original patch
was just a very odd example).

                      Linus

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-10-05  0:12                                 ` Linus Torvalds
@ 2016-10-05  0:24                                   ` Luis R. Rodriguez
  2016-10-05  0:32                                     ` Linus Torvalds
  2016-10-05  1:48                                   ` Josh Triplett
  1 sibling, 1 reply; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-10-05  0:24 UTC (permalink / raw)
  To: Linus Torvalds
  Cc: Dmitry Torokhov, Herbert, Marc, open list:DOCUMENTATION,
	Jacek Anaszewski, David Woodhouse, Christian Lamparter,
	Julia Lawall, Andrew Morton, linuxppc-dev, Mimi Zohar,
	Andy Lutomirski, Richard Purdie, Wu Fengguang, Johannes Berg,
	Michal Marek, Hauke Mehrtens, Mark Brown, Jiri Slaby, Ming Lei,
	Daniel Vetter, Bjorn Andersson, Felix Fietkau, Roman Pen,
	Greg KH, Linux Kernel Mailing List, Vikram Mulukutla,
	Stephen Boyd, Takashi Iwai, Jeff Mahoney, Hariprasad S,
	Benjamin Poirier, Josh Triplett

On Tue, Oct 4, 2016 at 5:12 PM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
> On Tue, Oct 4, 2016 at 5:00 PM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
>>
>> I am not sure how/why a firmware loading daemon would be a better
>> idea now. What Marc describes that Josh proposed with signals for
>> userspcae seems more aligned with what we likely need
>
> Quite frankly, I doubt you want a signal.
>
> You will want to have some way to specify where the firmware files
> are. Right now we have "fw_path[]" which is hardcoded except for the
> first entry that can be set as a module parameter. But you'd probably
> want to expand on that, which implies some /sys or /proc interface.
>
> And once you do that, wouldn't it make more sense to just make the
> "update the firmware path /proc/sys/kernel/fw_path file" make things
> re-search for firmware?

We can, but re-searching for firmware assumes we cache pending
firmware, we currently don't, we just either process sync or async
firmware requests.

> In other words, the interface has to be something *sensible*. Not some
> idiotic ad-hoc "send a signal" (of which that stupid original patch
> was just a very odd example).

Note that the races are beyond firmware, so all
kernel_read_file_from_path() users, as such re-using such old /sys/
interafeces for firmware will not suffice to cover all ground now for
the same race for other possible users.

 Luis

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-10-05  0:24                                   ` Luis R. Rodriguez
@ 2016-10-05  0:32                                     ` Linus Torvalds
  2016-10-05 17:38                                       ` Luis R. Rodriguez
  0 siblings, 1 reply; 80+ messages in thread
From: Linus Torvalds @ 2016-10-05  0:32 UTC (permalink / raw)
  To: Luis R. Rodriguez
  Cc: Dmitry Torokhov, Herbert, Marc, open list:DOCUMENTATION,
	Jacek Anaszewski, David Woodhouse, Christian Lamparter,
	Julia Lawall, Andrew Morton, linuxppc-dev, Mimi Zohar,
	Andy Lutomirski, Richard Purdie, Wu Fengguang, Johannes Berg,
	Michal Marek, Hauke Mehrtens, Mark Brown, Jiri Slaby, Ming Lei,
	Daniel Vetter, Bjorn Andersson, Felix Fietkau, Roman Pen,
	Greg KH, Linux Kernel Mailing List, Vikram Mulukutla,
	Stephen Boyd, Takashi Iwai, Jeff Mahoney, Hariprasad S,
	Benjamin Poirier, Josh Triplett

On Tue, Oct 4, 2016 at 5:24 PM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
>
> Note that the races are beyond firmware, so all
> kernel_read_file_from_path() users, as such re-using such old /sys/
> interafeces for firmware will not suffice to cover all ground now for
> the same race for other possible users.

Blah blah blah.

The reason I've hated this whole discussion is that it's full of
"let's re-architect everything", and then it has these horribly warty
interfaces. It's classic second-system syndrome.

Just do *one* thing, and do it well. Don't change anything else. Don't
force existign drivers to use new interfaces. Don't over-architect,
and don't do stupid interfaces.

If user-space mounts a new filesystem (or just unpacks files from a
tar-file that has firmware images in it, for chissake), that is not
some magical "critical mount event". The whole concept is just stupid.
Is it a "mount event" when the user downloads a new firmware image
from the internet?

HELL NO.

But what is equally stupid is to then dismiss simple models because
some totally unrelated "beyond firmware" issue.

Anything that is "beyond firmware" shouldn't even be discussed, for
chrissake! It has nothing what-so-ever to do with firmware loading. If
there ends up being some common helper functions, and shared code,
that *still* doesn't make it so.

Basic rules of thumb:

 (a) don't over-design

 (b) don't have stupid illogical interfaces

 (c) don't conflate different issues just because you think they may
have shared code.

 (4) be consistent. Don't make up new interfaces, and most certainly
do *NOT* dismiss something just because it's what we have done before.

That's it.

                Linus

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-10-05  0:12                                 ` Linus Torvalds
  2016-10-05  0:24                                   ` Luis R. Rodriguez
@ 2016-10-05  1:48                                   ` Josh Triplett
  2016-10-05  1:58                                     ` Linus Torvalds
  1 sibling, 1 reply; 80+ messages in thread
From: Josh Triplett @ 2016-10-05  1:48 UTC (permalink / raw)
  To: Linus Torvalds
  Cc: Luis R. Rodriguez, Dmitry Torokhov, Herbert, Marc,
	open list:DOCUMENTATION, Jacek Anaszewski, David Woodhouse,
	Christian Lamparter, Julia Lawall, Andrew Morton, linuxppc-dev,
	Mimi Zohar, Andy Lutomirski, Richard Purdie, Wu Fengguang,
	Johannes Berg, Michal Marek, Hauke Mehrtens, Mark Brown,
	Jiri Slaby, Ming Lei, Daniel Vetter, Bjorn Andersson,
	Felix Fietkau, Roman Pen, Greg KH, Linux Kernel Mailing List,
	Vikram Mulukutla, Stephen Boyd, Takashi Iwai, Jeff Mahoney,
	Hariprasad S, Benjamin Poirier, keescook

On Tue, Oct 04, 2016 at 05:12:58PM -0700, Linus Torvalds wrote:
> On Tue, Oct 4, 2016 at 5:00 PM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> > I am not sure how/why a firmware loading daemon would be a better
> > idea now. What Marc describes that Josh proposed with signals for
> > userspcae seems more aligned with what we likely need
> 
> Quite frankly, I doubt you want a signal.
> 
> You will want to have some way to specify where the firmware files
> are. Right now we have "fw_path[]" which is hardcoded except for the
> first entry that can be set as a module parameter. But you'd probably
> want to expand on that, which implies some /sys or /proc interface.
> 
> And once you do that, wouldn't it make more sense to just make the
> "update the firmware path /proc/sys/kernel/fw_path file" make things
> re-search for firmware?

That could work, but it seems like overkill to allow changing the path,
rather than the simpler interface of just telling the one driver "go
ahead and direct-load your firmware now".  I definitely don't think it
should be a system-wide "mount event"; it should be a per-device "go
direct-load your firmware" poke from userspace.  That would solve the
"build-in the driver so it can start waking up slow monitors, but wait
to load the firmware until you have a filesystem" problem.  (And it
would avoid creating some unusual driver-specific late-firmware-load
mechanism.)

That said, the Chrome OS folks apparently have some mechanism where they
mount a tmpfs over /lib/firmware to let userspace choose firmware at
runtime, so perhaps the path-changing mechanism would help there.  Kees?

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-10-05  1:48                                   ` Josh Triplett
@ 2016-10-05  1:58                                     ` Linus Torvalds
  0 siblings, 0 replies; 80+ messages in thread
From: Linus Torvalds @ 2016-10-05  1:58 UTC (permalink / raw)
  To: Josh Triplett
  Cc: Luis R. Rodriguez, Dmitry Torokhov, Herbert, Marc,
	open list:DOCUMENTATION, Jacek Anaszewski, David Woodhouse,
	Christian Lamparter, Julia Lawall, Andrew Morton, linuxppc-dev,
	Mimi Zohar, Andy Lutomirski, Richard Purdie, Wu Fengguang,
	Johannes Berg, Michal Marek, Hauke Mehrtens, Mark Brown,
	Jiri Slaby, Ming Lei, Daniel Vetter, Bjorn Andersson,
	Felix Fietkau, Roman Pen, Greg KH, Linux Kernel Mailing List,
	Vikram Mulukutla, Stephen Boyd, Takashi Iwai, Jeff Mahoney,
	Hariprasad S, Benjamin Poirier, Kees Cook

On Tue, Oct 4, 2016 at 6:48 PM, Josh Triplett <josh@joshtriplett.org> wrote:
>
>    I definitely don't think it
> should be a system-wide "mount event"; it should be a per-device "go
> direct-load your firmware" poke from userspace.

I don't disagree with that kind of interface. We already have things
like "rescan" for PCI bus devices to force a bus rescan. Iit's a
simple device attribute. Having a similar thing to trigger firmware
reload for a driver sounds entirely sane.

              Linus

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-10-05  0:32                                     ` Linus Torvalds
@ 2016-10-05 17:38                                       ` Luis R. Rodriguez
  0 siblings, 0 replies; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-10-05 17:38 UTC (permalink / raw)
  To: Linus Torvalds
  Cc: Luis R. Rodriguez, Dmitry Torokhov, Herbert, Marc,
	open list:DOCUMENTATION, Jacek Anaszewski, David Woodhouse,
	Christian Lamparter, Julia Lawall, Andrew Morton, linuxppc-dev,
	Mimi Zohar, Andy Lutomirski, Richard Purdie, Wu Fengguang,
	Johannes Berg, Michal Marek, Hauke Mehrtens, Mark Brown,
	Jiri Slaby, Ming Lei, Daniel Vetter, Bjorn Andersson,
	Felix Fietkau, Roman Pen, Greg KH, Linux Kernel Mailing List,
	Vikram Mulukutla, Stephen Boyd, Takashi Iwai, Jeff Mahoney,
	Hariprasad S, Benjamin Poirier, Josh Triplett, Kees Cook

On Tue, Oct 04, 2016 at 05:32:22PM -0700, Linus Torvalds wrote:
> On Tue, Oct 4, 2016 at 5:24 PM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> >
> > Note that the races are beyond firmware, so all
> > kernel_read_file_from_path() users, as such re-using such old /sys/
> > interafeces for firmware will not suffice to cover all ground now for
> > the same race for other possible users.
> 
> Blah blah blah.
> 
> The reason I've hated this whole discussion is that it's full of
> "let's re-architect everything", and then it has these horribly warty
> interfaces.

To be clear, kernel_read_file_from_path() was an agreed upon strategy
about 1 year ago at the Linux Security summit as we found different
kernel implementations for the same exact task, reading files from
the filesystem -- my point here was simply that acknowledging that the
race on early init and driver's init / probe for firmware is implicating
that the race is *also* possible for the other kernel-read-from-fs points.
Its not clear to me what your grudge here is other than the proposal
for a solution in this patch is not what we want.

> It's classic second-system syndrome.
> 
> Just do *one* thing, and do it well. Don't change anything else. Don't
> force existign drivers to use new interfaces. Don't over-architect,
> and don't do stupid interfaces.

If there is a race for the other users and we want to avoid wrapping
a solution for it to the other callers without doing any vetting for
correctness then so be it, but to disregard completely seems error-prone.
I accept that thinking about such other users may complicate a solution
for firmware and if you prefer we just separate the race solution for
both that's fine.

> If user-space mounts a new filesystem (or just unpacks files from a
> tar-file that has firmware images in it, for chissake), that is not
> some magical "critical mount event". The whole concept is just stupid.
> Is it a "mount event" when the user downloads a new firmware image
> from the internet?
> 
> HELL NO.

We've gotten passed that the original implementation proposed is not what we
want, let's move on.

> But what is equally stupid is to then dismiss simple models because
> some totally unrelated "beyond firmware" issue.

I have not heard back from the other stakeholders using
kernel_read_file_from_path() and possible races for them. You seem to suggest
to ignore those possible theoretical races in the name of a simple solution for
firmware. Fine.

> Anything that is "beyond firmware" shouldn't even be discussed, for
> chrissake! It has nothing what-so-ever to do with firmware loading. If
> there ends up being some common helper functions, and shared code,
> that *still* doesn't make it so.

My point was to raise the flag of the possible races on the other call sites
where we read files directly from the kernel, that's all, if we agree we really
don't care for that fine.

> Basic rules of thumb:
> 
>  (a) don't over-design
> 
>  (b) don't have stupid illogical interfaces
> 
>  (c) don't conflate different issues just because you think they may
> have shared code.
> 
>  (4) be consistent. Don't make up new interfaces, and most certainly
> do *NOT* dismiss something just because it's what we have done before.
> 
> That's it.

OK..

  Luis

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-09-14  2:38               ` Rob Landley
@ 2016-10-05 18:00                 ` Luis R. Rodriguez
  2016-10-05 18:08                   ` Linus Torvalds
  0 siblings, 1 reply; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-10-05 18:00 UTC (permalink / raw)
  To: Rob Landley
  Cc: Luis R. Rodriguez, Daniel Vetter, Mimi Zohar, Felix Fietkau,
	David Woodhouse, Roman Pen, Bjorn Andersson, Ming Lei,
	Andrew Morton, Michal Marek, Greg KH, Linux Kernel Mailing List,
	Vikram Mulukutla, Stephen Boyd, Mark Brown, Takashi Iwai,
	Johannes Berg, Christian Lamparter, hauke, Josh Boyer,
	Dmitry Torokhov, jslaby, Linus Torvalds, Andy Lutomirski,
	Wu Fengguang, rpurdie, Jeff Mahoney, j.anaszewski, Abhay_Salunke,
	Julia Lawall, Gilles.Muller, nicolas.palix, Tom Gundersen,
	Kay Sievers, David Howells, Alessandro Rubini, Kevin Cernekee,
	Kees Cook, Jonathan Corbet, Thierry Martinez, cocci,
	linux-serial, linux-doc, linuxppc-dev, Josh Triplett

On Tue, Sep 13, 2016 at 09:38:17PM -0500, Rob Landley wrote:
> On 09/02/2016 07:20 PM, Luis R. Rodriguez wrote:
> > kernel_read_file_from_path() can try to read a file from
> > the system's filesystem. This is typically done for firmware
> > for instance, which lives in /lib/firmware. One issue with
> > this is that the kernel cannot know for sure when the real
> > final /lib/firmare/ is ready, and even if you use initramfs
> > drivers are currently initialized *first* prior to the initramfs
> > kicking off.
> 
> Why?

do_initcalls() is called prior to prepare_namespace(), other than that
we have no strict rules over where the real rootfs should be, and since
we have pivot_root() its up to userspace to decide when/how the real
rootfs goes. This and the fact that its also up to userspace to design
what files to place in initramfs of further rootfs -- only userspace
will know for sure when all firmware for all drivers is really ready.

> > During init we run through all init calls first
> > (do_initcalls()) and finally the initramfs is processed via
> > prepare_namespace():
> 
> What's the downside of moving initramfs cpio extraction earlier in the boot?

That would help users of initrafms, some folks seem to not want to use
initramfs, one of such users are that of the large firmwares for remote-proc
(Documentation/remoteproc.txt), we're talking about over 200 MiB for some
firmware for example.

> I did some shuffling around of those code to make initmpfs work, does
> anybody know why initramfs extraction _before_ we initialize drivers
> would be a bad thing?

No, but it seems sensible to me, if its done before do_initcalls()
that should resolve the race for initramfs users but -- so long
as the drivers that need firmware early are dumped into initramfs.
We have no assurances/warnings for this, but we can add such things
if we want them. This would not resolve the race for non-initramfs
users / pivot_root() changes.

  Luis

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-10-05 18:00                 ` Luis R. Rodriguez
@ 2016-10-05 18:08                   ` Linus Torvalds
  2016-10-05 19:46                     ` Luis R. Rodriguez
  0 siblings, 1 reply; 80+ messages in thread
From: Linus Torvalds @ 2016-10-05 18:08 UTC (permalink / raw)
  To: Luis R. Rodriguez
  Cc: Rob Landley, Daniel Vetter, Mimi Zohar, Felix Fietkau,
	David Woodhouse, Roman Pen, Bjorn Andersson, Ming Lei,
	Andrew Morton, Michal Marek, Greg KH, Linux Kernel Mailing List,
	Vikram Mulukutla, Stephen Boyd, Mark Brown, Takashi Iwai,
	Johannes Berg, Christian Lamparter, Hauke Mehrtens, Josh Boyer,
	Dmitry Torokhov, Jiri Slaby, Andy Lutomirski, Wu Fengguang,
	Richard Purdie, Jeff Mahoney, Jacek Anaszewski, Abhay_Salunke,
	Julia Lawall, Gilles.Muller, nicolas.palix, Tom Gundersen,
	Kay Sievers, David Howells, Alessandro Rubini, Kevin Cernekee,
	Kees Cook, Jonathan Corbet, Thierry Martinez, cocci,
	linux-serial, open list:DOCUMENTATION, linuxppc-dev,
	Josh Triplett

On Wed, Oct 5, 2016 at 11:00 AM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> On Tue, Sep 13, 2016 at 09:38:17PM -0500, Rob Landley wrote:
>
>> I did some shuffling around of those code to make initmpfs work, does
>> anybody know why initramfs extraction _before_ we initialize drivers
>> would be a bad thing?
>
> No, but it seems sensible to me, if its done before do_initcalls()
> that should resolve the race for initramfs users

initramfs should already be set up before drivers are. Exactly what is
it that has trouble right now?

The gating issue for initramfs is that technically the filesystem
setup needs to be done, which means that it currently ends up being
populated _fairly_ late in the initcall series, but certainly before
drivers. But since initramfs really only needs very limited filesystem
functionality, I assume Rob had few problems with just moving it
earlier.

Still, what kind of ordering issues did people have? What is it that
needs to load files even before driver init? Some crazy subsystem?

                   Linus

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-10-05 18:08                   ` Linus Torvalds
@ 2016-10-05 19:46                     ` Luis R. Rodriguez
  2016-11-08 22:47                       ` Luis R. Rodriguez
  0 siblings, 1 reply; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-10-05 19:46 UTC (permalink / raw)
  To: Linus Torvalds, Herbert, Marc, Daniel Vetter, Bjorn Andersson
  Cc: Luis R. Rodriguez, Rob Landley, Mimi Zohar, Felix Fietkau,
	David Woodhouse, Roman Pen, Ming Lei, Andrew Morton,
	Michal Marek, Greg KH, Linux Kernel Mailing List,
	Vikram Mulukutla, Stephen Boyd, Mark Brown, Takashi Iwai,
	Johannes Berg, Christian Lamparter, Hauke Mehrtens, Josh Boyer,
	Dmitry Torokhov, Jiri Slaby, Andy Lutomirski, Wu Fengguang,
	Richard Purdie, Jeff Mahoney, Jacek Anaszewski, Abhay_Salunke,
	Julia Lawall, Gilles.Muller, nicolas.palix, Tom Gundersen,
	Kay Sievers, David Howells, Alessandro Rubini, Kevin Cernekee,
	Kees Cook, Jonathan Corbet, Thierry Martinez, linux-serial,
	open list:DOCUMENTATION, linuxppc-dev, Josh Triplett

On Wed, Oct 05, 2016 at 11:08:06AM -0700, Linus Torvalds wrote:
> On Wed, Oct 5, 2016 at 11:00 AM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> > On Tue, Sep 13, 2016 at 09:38:17PM -0500, Rob Landley wrote:
> >
> >> I did some shuffling around of those code to make initmpfs work, does
> >> anybody know why initramfs extraction _before_ we initialize drivers
> >> would be a bad thing?
> >
> > No, but it seems sensible to me, if its done before do_initcalls()
> > that should resolve the race for initramfs users
> 
> initramfs should already be set up before drivers are.

Actually you are right, the issue would only be for old initrd, for initramfs
we populate that via rootfs_initcall(populate_rootfs), so as long as drivers
in question use an init level beyond rootfs's we're good there.

> Exactly what is it that has trouble right now?

It would seem then that the only current stated race possible should
then be non-initramfs users. One example if very large firmware for
remote-proc, whereby an initramfs is just not practical or desirable.

> The gating issue for initramfs is that technically the filesystem
> setup needs to be done, which means that it currently ends up being
> populated _fairly_ late in the initcall series, but certainly before
> drivers. But since initramfs really only needs very limited filesystem
> functionality, I assume Rob had few problems with just moving it
> earlier.
> 
> Still, what kind of ordering issues did people have? What is it that
> needs to load files even before driver init? Some crazy subsystem?

No, I think this is just about non-initramfs users now, if we disregard
old initrd users. Bjorn, Marc, correct me if I'm wrong, as I think its
so far you both who have seemed to run into race issues and have then
ended up trying to look for hacks to address this race or considered using
the usermode helper (which we're trying to minimize users for). Daniel
seems to note a lot of video drivers use firmware on probe as well so
there's a potential issue for those users if they don't use initramfs.

  Luis

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-10-05 19:46                     ` Luis R. Rodriguez
@ 2016-11-08 22:47                       ` Luis R. Rodriguez
  2016-11-09  9:13                         ` Daniel Wagner
                                           ` (2 more replies)
  0 siblings, 3 replies; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-11-08 22:47 UTC (permalink / raw)
  To: Linus Torvalds, Jiri Kosina, Johannes Berg, Jouni Malinen,
	Seth Forshee, Tom Gundersen, Kay Sievers, Bjorn Andersson,
	Daniel Wagner, Daniel Wagner, Rafael J. Wysocki
  Cc: Luis R. Rodriguez, Herbert, Marc, Daniel Vetter, Rob Landley,
	Mimi Zohar, Felix Fietkau, David Woodhouse, Roman Pen, Ming Lei,
	Andrew Morton, Michal Marek, Greg KH, Linux Kernel Mailing List,
	Vikram Mulukutla, Stephen Boyd, Mark Brown, Takashi Iwai,
	Christian Lamparter, Hauke Mehrtens, Josh Boyer, Dmitry Torokhov,
	Jiri Slaby, Andy Lutomirski, Wu Fengguang, Richard Purdie,
	Jeff Mahoney, Jacek Anaszewski, Abhay_Salunke, Julia Lawall,
	Gilles.Muller, nicolas.palix, Tom Gundersen, David Howells,
	Alessandro Rubini, Kevin Cernekee, Kees Cook, Jonathan Corbet,
	Thierry Martinez, linux-serial, open list:DOCUMENTATION,
	linuxppc-dev, Josh Triplett

On Wed, Oct 05, 2016 at 09:46:33PM +0200, Luis R. Rodriguez wrote:
> On Wed, Oct 05, 2016 at 11:08:06AM -0700, Linus Torvalds wrote:
> > On Wed, Oct 5, 2016 at 11:00 AM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> > > On Tue, Sep 13, 2016 at 09:38:17PM -0500, Rob Landley wrote:
> > >
> > >> I did some shuffling around of those code to make initmpfs work, does
> > >> anybody know why initramfs extraction _before_ we initialize drivers
> > >> would be a bad thing?
> > >
> > > No, but it seems sensible to me, if its done before do_initcalls()
> > > that should resolve the race for initramfs users
> > 
> > initramfs should already be set up before drivers are.
> 
> Actually you are right, the issue would only be for old initrd, for initramfs
> we populate that via rootfs_initcall(populate_rootfs), so as long as drivers
> in question use an init level beyond rootfs's we're good there.
> 
> > Exactly what is it that has trouble right now?
> 
> It would seem then that the only current stated race possible should
> then be non-initramfs users.

Or:

a) initramfs users that include a driver but do not include the firmware
into initramfs

b) driver is built-in but firmware is not in initrafms (Johannes reports
   this causes driver failure on intel wireless for instance, and I guess
   you need to reload)

> One example if very large firmware for
> remote-proc, whereby an initramfs is just not practical or desirable.

This issue still stands. At Plumbers Johannes Berg did indicate to me
he had a simple elegant solution in mind. He suggested that since the
usermode helper was available, he had added support to be able to
differentiate async firmware request calls form sync requests, and that
userspace should not return an error *iff* the request made was async and
it can determine we're initramfs. The semantics issue is the same though,
is there a generic way to determine we're initramfs ? What if we move
multiple levels? Anyway -- provided we could figure this out, userspace
would simply yield and wait until the real rootfs is met. Upon pivot_root()
the assumption is all previous udev events pending would be re-triggered
and finally udev could finally confirm/deny if the firmware was present.
This would *also* allow you to stuff your firmware whever, however big
it was. This however relied on the userspace firmware loading support,
it turns out that (I think because of an incorrect negative backlash
back in the day over blaming this over booting issues due to the timeout
whereas the real issue was the kmod timeout was affecting our long
standing serial init()/probe()) the systemd userspace firmware laoding
support was removed from systemd udev in 2014 by Kay via commit
be2ea723b1d023b3d ("udev: remove userspace firmware loading support").

Systemd might *still* be able to provide a solution here, however I will
note Johannes was asking for *all* async firmware requests to always
rely on the kernel syfs UMH fallback -- this suggestion is against the
direction we've been taking to eventually compartamentalize the kernel UMH
code, so whatever we decide to do, lets please take a breather and seriously
address this properly *with* systemd folks.

A side race discussed at Plumbers worth mentioning given its related to the
UMH was inspired by Jiri's talk on the abuse of the freezer for kthreads --
and his suggestion to use freeze_super(). Currently the UMH lock is used
for the UMH but as I have noted in Daniel Wagner's recent patches to
give some love to this code and further compartamentalize the UMH --
the UMH lock was originally added to help avoid drivers use the firmware
API on resume, given the races. The firmware cache solution implemented by
Ming Lei years ago helped address this, whereby devm helpers are used
based on the requested firmware and prior to suspend we cache all required
firmware for drivers so that upon resume calls would work without the
effective race present. This mitigated the actual races / issues with
drivers, but they must not use the firmware API on suspend/resume. Since
this solution *kills* all pending UMH caller on suspend obviously this
means on suspend using request_firmware*() API and expecting it to work
is brutally dumb as we will eventually kill any pending requests. This
is a long winded way to say that if you rely on the UMH for firmware
you must figure out your own proactive firmware cache solution and
must definitely not request firmware on suspend. Two things then:

1) I've been brainstorming with Daniel how to use freeze_super() to
   replace the now invalid UMH lock -- its purpose only helps races
   on boot, for the fallback case to the UMH. But note most distributions
   disable forcing it always on, so these days we *only* rely on the UMH
   as a fallback if the driver explicitly requested it

2) Drivers relying on the UMH very likely have a broken cache solution
   if they are doing this on suspend

Whatever the outcome of this discussion is -- Johannes seemed to *want*
to further use the UMH by default on *all* async alls... even if the
driver did not explicitly requested it -- I'm concerned about this given
all the above and the existing flip/flop on systemd for it. Whatever
we try to dream up here, please consider all the above as well.

> > The gating issue for initramfs is that technically the filesystem
> > setup needs to be done, which means that it currently ends up being
> > populated _fairly_ late in the initcall series, but certainly before
> > drivers. But since initramfs really only needs very limited filesystem
> > functionality, I assume Rob had few problems with just moving it
> > earlier.
> > 
> > Still, what kind of ordering issues did people have? What is it that
> > needs to load files even before driver init? Some crazy subsystem?
> 
> No, I think this is just about non-initramfs users now,

And as Johannes pointed two above to cases.

> if we disregard
> old initrd users. Bjorn, Marc, correct me if I'm wrong, as I think its
> so far you both who have seemed to run into race issues and have then
> ended up trying to look for hacks to address this race or considered using
> the usermode helper (which we're trying to minimize users for). Daniel
> seems to note a lot of video drivers use firmware on probe as well so
> there's a potential issue for those users if they don't use initramfs.

  Luis

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-11-08 22:47                       ` Luis R. Rodriguez
@ 2016-11-09  9:13                         ` Daniel Wagner
  2016-11-09 23:40                         ` Luis R. Rodriguez
  2016-11-15  9:28                         ` Johannes Berg
  2 siblings, 0 replies; 80+ messages in thread
From: Daniel Wagner @ 2016-11-09  9:13 UTC (permalink / raw)
  To: Luis R. Rodriguez, Linus Torvalds, Jiri Kosina, Johannes Berg,
	Jouni Malinen, Seth Forshee, Tom Gundersen, Kay Sievers,
	Bjorn Andersson, Daniel Wagner, Rafael J. Wysocki
  Cc: Herbert, Marc, Daniel Vetter, Rob Landley, Mimi Zohar,
	Felix Fietkau, David Woodhouse, Roman Pen, Ming Lei,
	Andrew Morton, Michal Marek, Greg KH, Linux Kernel Mailing List,
	Vikram Mulukutla, Stephen Boyd, Mark Brown, Takashi Iwai,
	Christian Lamparter, Hauke Mehrtens, Josh Boyer, Dmitry Torokhov,
	Jiri Slaby, Andy Lutomirski, Wu Fengguang, Richard Purdie,
	Jeff Mahoney, Jacek Anaszewski, Abhay_Salunke, Julia Lawall,
	Gilles.Muller, nicolas.palix, David Howells, Alessandro Rubini,
	Kevin Cernekee, Kees Cook, Jonathan Corbet, Thierry Martinez,
	linux-serial, open list:DOCUMENTATION, linuxppc-dev,
	Josh Triplett, Harald Hoyer

[CC: added Harald]

On 11/08/2016 11:47 PM, Luis R. Rodriguez wrote:
> On Wed, Oct 05, 2016 at 09:46:33PM +0200, Luis R. Rodriguez wrote:
>> On Wed, Oct 05, 2016 at 11:08:06AM -0700, Linus Torvalds wrote:
>>> On Wed, Oct 5, 2016 at 11:00 AM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
>>>> On Tue, Sep 13, 2016 at 09:38:17PM -0500, Rob Landley wrote:
>>>>
>>>>> I did some shuffling around of those code to make initmpfs work, does
>>>>> anybody know why initramfs extraction _before_ we initialize drivers
>>>>> would be a bad thing?
>>>>
>>>> No, but it seems sensible to me, if its done before do_initcalls()
>>>> that should resolve the race for initramfs users
>>>
>>> initramfs should already be set up before drivers are.
>>
>> Actually you are right, the issue would only be for old initrd, for initramfs
>> we populate that via rootfs_initcall(populate_rootfs), so as long as drivers
>> in question use an init level beyond rootfs's we're good there.
>>
>>> Exactly what is it that has trouble right now?
>>
>> It would seem then that the only current stated race possible should
>> then be non-initramfs users.
>
> Or:
>
> a) initramfs users that include a driver but do not include the firmware
> into initramfs
>
> b) driver is built-in but firmware is not in initrafms (Johannes reports
>    this causes driver failure on intel wireless for instance, and I guess
>    you need to reload)
>
>> One example if very large firmware for
>> remote-proc, whereby an initramfs is just not practical or desirable.
>
> This issue still stands. At Plumbers Johannes Berg did indicate to me
> he had a simple elegant solution in mind. He suggested that since the
> usermode helper was available, he had added support to be able to
> differentiate async firmware request calls form sync requests, and that
> userspace should not return an error *iff* the request made was async and
> it can determine we're initramfs. The semantics issue is the same though,
> is there a generic way to determine we're initramfs ? What if we move
> multiple levels? Anyway -- provided we could figure this out, userspace
> would simply yield and wait until the real rootfs is met. Upon pivot_root()
> the assumption is all previous udev events pending would be re-triggered
> and finally udev could finally confirm/deny if the firmware was present.
> This would *also* allow you to stuff your firmware whever, however big
> it was. This however relied on the userspace firmware loading support,
> it turns out that (I think because of an incorrect negative backlash
> back in the day over blaming this over booting issues due to the timeout
> whereas the real issue was the kmod timeout was affecting our long
> standing serial init()/probe()) the systemd userspace firmware laoding
> support was removed from systemd udev in 2014 by Kay via commit
> be2ea723b1d023b3d ("udev: remove userspace firmware loading support").
>
> Systemd might *still* be able to provide a solution here, however I will
> note Johannes was asking for *all* async firmware requests to always
> rely on the kernel syfs UMH fallback -- this suggestion is against the
> direction we've been taking to eventually compartamentalize the kernel UMH
> code, so whatever we decide to do, lets please take a breather and seriously
> address this properly *with* systemd folks.
>
> A side race discussed at Plumbers worth mentioning given its related to the
> UMH was inspired by Jiri's talk on the abuse of the freezer for kthreads --
> and his suggestion to use freeze_super(). Currently the UMH lock is used
> for the UMH but as I have noted in Daniel Wagner's recent patches to
> give some love to this code and further compartamentalize the UMH --
> the UMH lock was originally added to help avoid drivers use the firmware
> API on resume, given the races. The firmware cache solution implemented by
> Ming Lei years ago helped address this, whereby devm helpers are used
> based on the requested firmware and prior to suspend we cache all required
> firmware for drivers so that upon resume calls would work without the
> effective race present. This mitigated the actual races / issues with
> drivers, but they must not use the firmware API on suspend/resume. Since
> this solution *kills* all pending UMH caller on suspend obviously this
> means on suspend using request_firmware*() API and expecting it to work
> is brutally dumb as we will eventually kill any pending requests. This
> is a long winded way to say that if you rely on the UMH for firmware
> you must figure out your own proactive firmware cache solution and
> must definitely not request firmware on suspend. Two things then:
>
> 1) I've been brainstorming with Daniel how to use freeze_super() to
>    replace the now invalid UMH lock -- its purpose only helps races
>    on boot, for the fallback case to the UMH. But note most distributions
>    disable forcing it always on, so these days we *only* rely on the UMH
>    as a fallback if the driver explicitly requested it
>
> 2) Drivers relying on the UMH very likely have a broken cache solution
>    if they are doing this on suspend
>
> Whatever the outcome of this discussion is -- Johannes seemed to *want*
> to further use the UMH by default on *all* async alls... even if the
> driver did not explicitly requested it -- I'm concerned about this given
> all the above and the existing flip/flop on systemd for it. Whatever
> we try to dream up here, please consider all the above as well.

As Harald pointed out over a beer yesterday evening, there is at least
one more reason why UMH isn't obsolete. The ordering of the firmware 
loading might be of important. Say you want to greet the user with a 
splash screen really early on, the graphic card firmware should be 
loaded first. Also the automotive world has this fancy requirement that 
rear camera must be on the screen within 2 seconds. So controlling the 
firmware loading order is of importance (e.g. also do not overcommit the 
I/O bandwith not so important firmwares). A user space helper is able to 
prioritize the request accordingly the use case.

>>> The gating issue for initramfs is that technically the filesystem
>>> setup needs to be done, which means that it currently ends up being
>>> populated _fairly_ late in the initcall series, but certainly before
>>> drivers. But since initramfs really only needs very limited filesystem
>>> functionality, I assume Rob had few problems with just moving it
>>> earlier.
>>>
>>> Still, what kind of ordering issues did people have? What is it that
>>> needs to load files even before driver init? Some crazy subsystem?
>>
>> No, I think this is just about non-initramfs users now,
>
> And as Johannes pointed two above to cases.
>
>> if we disregard
>> old initrd users. Bjorn, Marc, correct me if I'm wrong, as I think its
>> so far you both who have seemed to run into race issues and have then
>> ended up trying to look for hacks to address this race or considered using
>> the usermode helper (which we're trying to minimize users for). Daniel
>> seems to note a lot of video drivers use firmware on probe as well so
>> there's a potential issue for those users if they don't use initramfs.

Daniel

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-11-08 22:47                       ` Luis R. Rodriguez
  2016-11-09  9:13                         ` Daniel Wagner
@ 2016-11-09 23:40                         ` Luis R. Rodriguez
  2016-11-15  9:28                         ` Johannes Berg
  2 siblings, 0 replies; 80+ messages in thread
From: Luis R. Rodriguez @ 2016-11-09 23:40 UTC (permalink / raw)
  To: Linus Torvalds, Jiri Kosina, Johannes Berg, Jouni Malinen,
	Seth Forshee, Tom Gundersen, Kay Sievers, Bjorn Andersson,
	Daniel Wagner, Daniel Wagner, Rafael J. Wysocki, Harald Hoyer
  Cc: Luis R. Rodriguez, Herbert, Marc, Daniel Vetter, Rob Landley,
	Mimi Zohar, Felix Fietkau, David Woodhouse, Roman Pen, Ming Lei,
	Andrew Morton, Michal Marek, Greg KH, Linux Kernel Mailing List,
	Vikram Mulukutla, Stephen Boyd, Mark Brown, Takashi Iwai,
	Christian Lamparter, Hauke Mehrtens, Josh Boyer, Dmitry Torokhov,
	Jiri Slaby, Andy Lutomirski, Wu Fengguang, Richard Purdie,
	Jeff Mahoney, Jacek Anaszewski, Abhay Salunke, Julia Lawall,
	Gilles Muller, Nicolas Palix, David Howells, Alessandro Rubini,
	Kevin Cernekee, Kees Cook, Jonathan Corbet, Thierry Martinez,
	linux-serial, open list:DOCUMENTATION, linuxppc-dev,
	Josh Triplett

On Tue, Nov 8, 2016 at 2:47 PM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> Whatever the outcome of this discussion is -- Johannes seemed to *want*
> to further use the UMH by default on *all* async alls... even if the
> driver did not explicitly requested it -- I'm concerned about this given
> all the above and the existing flip/flop on systemd for it. Whatever
> we try to dream up here, please consider all the above as well.

One addition to this: the current API does not always require the UMH
firmware fallback, for most distributions that do not enable
CONFIG_FW_LOADER_USER_HELPER_FALLBACK but do enable
CONFIG_FW_LOADER_USER_HELPER we only require the UMH firmware fallback
*iff* the driver explicitly requests it. For kernels with
CONFIG_FW_LOADER_USER_HELPER_FALLBACK enabled we *always* use the UMH
fallback. By fallback note that this means its used only if the first
direct filesystem request failed. For further details on complexities
of the UMH refer to two ongoing threads [0] [1] about it.

Johannes, you seemed to note you added some uevent classifier for
async requests, I checked and it seems this was with commit
e9045f9178f3e ("firmware class: export nowait to userspace") was this
the change you were referring to ? Even with all these complexities
annotated, do you still believe we need the UMH always for all async
calls ?

[0] https://lkml.kernel.org/r/20161109211741.GI13978@wotan.suse.de
[1] https://lkml.kernel.org/r/20161109220210.GJ13978@wotan.suse.de

  Luis

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [RFC] fs: add userspace critical mounts event support
  2016-11-08 22:47                       ` Luis R. Rodriguez
  2016-11-09  9:13                         ` Daniel Wagner
  2016-11-09 23:40                         ` Luis R. Rodriguez
@ 2016-11-15  9:28                         ` Johannes Berg
  2 siblings, 0 replies; 80+ messages in thread
From: Johannes Berg @ 2016-11-15  9:28 UTC (permalink / raw)
  To: Luis R. Rodriguez, Linus Torvalds, Jiri Kosina, Jouni Malinen,
	Seth Forshee, Tom Gundersen, Kay Sievers, Bjorn Andersson,
	Daniel Wagner, Daniel Wagner, Rafael J. Wysocki
  Cc: Herbert, Marc, Daniel Vetter, Rob Landley, Mimi Zohar,
	Felix Fietkau, David Woodhouse, Roman Pen, Ming Lei,
	Andrew Morton, Michal Marek, Greg KH, Linux Kernel Mailing List,
	Vikram Mulukutla, Stephen Boyd, Mark Brown, Takashi Iwai,
	Christian Lamparter, Hauke Mehrtens, Josh Boyer, Dmitry Torokhov,
	Jiri Slaby, Andy Lutomirski, Wu Fengguang, Richard Purdie,
	Jeff Mahoney, Jacek Anaszewski, Abhay_Salunke, Julia Lawall,
	Gilles.Muller, nicolas.palix, David Howells, Alessandro Rubini,
	Kevin Cernekee, Kees Cook, Jonathan Corbet, Thierry Martinez,
	linux-serial, open list:DOCUMENTATION, linuxppc-dev,
	Josh Triplett

On Tue, 2016-11-08 at 23:47 +0100, Luis R. Rodriguez wrote:

> This issue still stands. At Plumbers Johannes Berg did indicate to me
> he had a simple elegant solution in mind. He suggested that since the
> usermode helper was available, he had added support to be able to
> differentiate async firmware request calls form sync requests,

For reference:

commit e9045f9178f3e3445a3a5b85206f8681b3869562
Author: Johannes Berg <johannes.berg@intel.com>
Date:   Mon Mar 29 17:57:20 2010 +0200

    firmware class: export nowait to userspace

> it can determine we're initramfs. The semantics issue is the same
> though, is there a generic way to determine we're initramfs ? What if
> we move multiple levels? Anyway -- provided we could figure this out,
> userspace would simply yield and wait until the real rootfs is met. 

One way or another we have to have this kind of information somewhere.
I don't actually know how/where though.

> Upon pivot_root() the assumption is all previous udev events pending
> would be re-triggered and finally udev could finally confirm/deny if
> the firmware was present.

The retriggering is already the case, as far as I know, if only to load
modules that weren't part of initramfs.

> note Johannes was asking for *all* async firmware requests to always
> rely on the kernel syfs UMH fallback -- this suggestion is against
> the direction we've been taking to eventually compartamentalize the
> kernel UMH code, so whatever we decide to do, lets please take a
> breather and seriously address this properly *with* systemd folks.

I was saying that because that's the only way you can actually rely on
this functionality as a system integrator. If drivers have to opt in or
can opt out then you'll always end up chasing the drivers around.

My argument basically goes like this:

First, given good drivers (i.e. using request_firmware_nowait())
putting firmware even for a built-in driver into initramfs or not
should be a system integrator decision. If they don't need the device
that early, it should be possible for them to delay it. Or, perhaps, if
the firmware is too big, etc. I'm sure we can all come up with more
examples of why you'd want to do it one way or another.

Second, all of this can be solved in other ways by adding logic to the
kernel, like the rejected proposal to add a "rootfs available" bit
somewhere, that would cause async requests to behave similarly within
the kernel (don't return "not found" until they time out or this bit is
set, and retry loading when the bit gets set)

Third, having this in place can be more friendly to users who play with
kernel compilation, modules, etc. This is a fringe group in some ways,
but it is (was?) actually a relatively common complaint that drivers
built into the kernel wouldn't work - we'd always have to direct users
to do magic steps like rebuilding initramfs with the right options etc.

johannes

^ permalink raw reply	[flat|nested] 80+ messages in thread

end of thread, back to index

Thread overview: 80+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-06-16 22:54 [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues Luis R. Rodriguez
2016-06-16 22:54 ` [PATCH v2 1/5] MAINTAINERS: extend firmware_class maintainer list Luis R. Rodriguez
2016-06-16 22:54 ` [PATCH v2 2/5] firmware: annotate thou shalt not request fw on init or probe Luis R. Rodriguez
2016-08-24  6:55   ` Daniel Vetter
2016-08-24 20:39     ` Luis R. Rodriguez
2016-08-25 11:05       ` Daniel Vetter
2016-08-25 19:41         ` Luis R. Rodriguez
2016-08-25 20:10           ` Daniel Vetter
2016-08-25 20:25             ` Luis R. Rodriguez
2016-08-25 20:30           ` Dmitry Torokhov
2016-09-02 23:59           ` Luis R. Rodriguez
2016-09-03  0:20             ` [RFC] fs: add userspace critical mounts event support Luis R. Rodriguez
2016-09-03  4:11               ` Linus Torvalds
2016-09-03  4:20                 ` Dmitry Torokhov
     [not found]                   ` <CA+55aFz4q5peXAeY9h8o3he7R=wXrBSYkOjMM9TehOw=pPoS+Q@mail.gmail.com>
2016-09-03 17:49                     ` Dmitry Torokhov
2016-09-03 18:01                       ` Linus Torvalds
2016-09-03 18:10                         ` Dmitry Torokhov
2016-09-06 21:52                           ` Luis R. Rodriguez
2016-09-06 22:28                             ` Bjorn Andersson
2016-09-06 23:14                               ` Luis R. Rodriguez
2016-09-24  1:37                           ` Herbert, Marc
2016-09-24 17:41                             ` Dmitry Torokhov
2016-10-05  0:00                               ` Luis R. Rodriguez
2016-10-05  0:12                                 ` Linus Torvalds
2016-10-05  0:24                                   ` Luis R. Rodriguez
2016-10-05  0:32                                     ` Linus Torvalds
2016-10-05 17:38                                       ` Luis R. Rodriguez
2016-10-05  1:48                                   ` Josh Triplett
2016-10-05  1:58                                     ` Linus Torvalds
2016-09-06 17:46                 ` Bjorn Andersson
2016-09-06 18:32                   ` Linus Torvalds
2016-09-06 21:11                     ` Bjorn Andersson
2016-09-06 21:50                       ` Linus Torvalds
2016-09-06 23:04                         ` Luis R. Rodriguez
2016-09-24  2:51                           ` Herbert, Marc
2016-10-04 23:28                             ` Luis R. Rodriguez
2016-09-06 22:32                     ` Luis R. Rodriguez
2016-09-14  2:38               ` Rob Landley
2016-10-05 18:00                 ` Luis R. Rodriguez
2016-10-05 18:08                   ` Linus Torvalds
2016-10-05 19:46                     ` Luis R. Rodriguez
2016-11-08 22:47                       ` Luis R. Rodriguez
2016-11-09  9:13                         ` Daniel Wagner
2016-11-09 23:40                         ` Luis R. Rodriguez
2016-11-15  9:28                         ` Johannes Berg
2016-06-16 22:54 ` [PATCH v2 3/5] firmware: update usermode helper docs and add SmPL report Luis R. Rodriguez
2016-06-16 22:54 ` [PATCH v2 4/5] firmware: add usermode helper DECLARE_FW_LOADER_USER() annotation Luis R. Rodriguez
2016-06-16 22:54 ` [PATCH v2 5/5] firmware: fix fw cache to avoid usermode helper on suspend Luis R. Rodriguez
2016-07-07  0:56 ` [PATCH v2 0/5] firmware: add SmPL grammar to avoid issues Luis R. Rodriguez
2016-07-13 21:47   ` Luis R. Rodriguez
2016-07-28  0:41     ` Luis R. Rodriguez
2016-08-03 14:50       ` Luis R. Rodriguez
2016-08-03 15:04         ` Greg KH
2016-08-03 17:06           ` Luis R. Rodriguez
2016-08-03 19:32             ` Greg KH
2016-08-03 19:46               ` Luis R. Rodriguez
2016-07-13 23:52   ` Fengguang Wu
2016-07-14  2:15     ` Luis R. Rodriguez
2016-07-14  2:23       ` Fengguang Wu
2016-07-14  3:08         ` Luis R. Rodriguez
2016-07-14  3:35           ` Fengguang Wu
2016-08-24  0:45 ` [PATCH v3 " mcgrof
2016-08-24  0:45   ` [PATCH v3 1/5] MAINTAINERS: extend firmware_class maintainer list mcgrof
2016-08-24  0:45   ` [PATCH v3 2/5] firmware: annotate thou shalt not request fw on init or probe mcgrof
2016-08-24  8:17     ` Gabriel Paubert
2016-09-02 18:26       ` Luis R. Rodriguez
2016-08-24  0:45   ` [PATCH v3 3/5] firmware: update usermode helper docs and add SmPL report mcgrof
2016-08-24  0:45   ` [PATCH v3 4/5] firmware: add usermode helper DECLARE_FW_LOADER_USER() annotation mcgrof
2016-08-24  0:45   ` [PATCH v3 5/5] firmware: fix fw cache to avoid usermode helper on suspend mcgrof
2016-08-31  7:03     ` Daniel Wagner
2016-09-02 18:13       ` Luis R. Rodriguez
2016-09-07  0:42   ` [PATCH v4 0/5] firmware: add SmPL grammar to avoid issues Luis R. Rodriguez
2016-09-07  0:42     ` [PATCH v4 1/5] MAINTAINERS: extend firmware_class maintainer list Luis R. Rodriguez
2016-09-07  6:43       ` Greg KH
2016-09-08 14:58         ` Luis R. Rodriguez
2016-09-08 15:25         ` Ming Lei
2016-09-07  0:42     ` [PATCH v4 2/5] firmware: annotate thou shalt not request fw on init or probe Luis R. Rodriguez
2016-09-07  0:42     ` [PATCH v4 3/5] firmware: update usermode helper docs and add SmPL report Luis R. Rodriguez
2016-09-07  0:42     ` [PATCH v4 4/5] firmware: add usermode helper DECLARE_FW_LOADER_USER() annotation Luis R. Rodriguez
2016-09-07  0:42     ` [PATCH v4 5/5] firmware: fix fw cache to avoid usermode helper on suspend Luis R. Rodriguez

LKML Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git
	git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git
	git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git
	git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git
	git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git
	git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git
	git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git
	git clone --mirror https://lore.kernel.org/lkml/7 lkml/git/7.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \
		linux-kernel@vger.kernel.org linux-kernel@archiver.kernel.org
	public-inbox-index lkml


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel


AGPL code for this site: git clone https://public-inbox.org/ public-inbox