[PATCH] netconsole: Initialize after all core networking drivers

[PATCH] netconsole: Initialize after all core networking drivers

[Calvin Owens]

With built-in netconsole and IXGBE, configuring netconsole via the kernel
cmdline results in the following panic at boot:

    netpoll: netconsole: device eth0 not up yet, forcing it
    usb 2-1: new high-speed USB device number 2 using ehci-pci
    ixgbe 0000:03:00.0: registered PHC device on eth0
    BUG: unable to handle kernel NULL pointer dereference at 0000000000000810
    <snip>
    Call Trace:
     [<ffffffff81578661>] ? vxlan_get_rx_port+0x41/0xa0
     [<ffffffff81586828>] ixgbe_open+0x4e8/0x540
     [<ffffffff8168045c>] __dev_open+0xac/0x120
     [<ffffffff81680506>] dev_open+0x36/0x70
     [<ffffffff8169abec>] netpoll_setup+0x23c/0x300
     [<ffffffff8169a66a>] ? netpoll_parse_options+0x19a/0x200
     [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
     [<ffffffff81d79882>] init_netconsole+0xda/0x262
     [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
     [<ffffffff810003a8>] do_one_initcall+0x88/0x1b0
     [<ffffffff81d31144>] kernel_init_freeable+0x14a/0x1e3
     [<ffffffff81d308f1>] ? do_early_param+0x8c/0x8c
     [<ffffffff81778610>] ? rest_init+0x80/0x80
     [<ffffffff8177861e>] kernel_init+0xe/0xe0
     [<ffffffff8177dc5f>] ret_from_fork+0x3f/0x70
     [<ffffffff81778610>] ? rest_init+0x80/0x80

This happens because IXGBE assumes that vxlan has already been initialized.
The cleanest way to fix this is to just initialize netconsole after all the
other core networking stuff has completed.

Signed-off-by: Calvin Owens <calvinowens@fb.com>
---
 drivers/net/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/Makefile b/drivers/net/Makefile
index 900b0c5..31557d0 100644
--- a/drivers/net/Makefile
+++ b/drivers/net/Makefile
@@ -15,7 +15,6 @@ obj-$(CONFIG_MACVTAP) += macvtap.o
 obj-$(CONFIG_MII) += mii.o
 obj-$(CONFIG_MDIO) += mdio.o
 obj-$(CONFIG_NET) += Space.o loopback.o
-obj-$(CONFIG_NETCONSOLE) += netconsole.o
 obj-$(CONFIG_PHYLIB) += phy/
 obj-$(CONFIG_RIONET) += rionet.o
 obj-$(CONFIG_NET_TEAM) += team/
@@ -26,6 +25,7 @@ obj-$(CONFIG_VXLAN) += vxlan.o
 obj-$(CONFIG_GENEVE) += geneve.o
 obj-$(CONFIG_NLMON) += nlmon.o
 obj-$(CONFIG_NET_VRF) += vrf.o
+obj-$(CONFIG_NETCONSOLE) += netconsole.o
 
 #
 # Networking Drivers
-- 
2.5.0

Re: [PATCH] netconsole: Initialize after all core networking drivers

[Eric Dumazet]

On Thu, 2015-12-17 at 15:52 -0800, Calvin Owens wrote:
> With built-in netconsole and IXGBE, configuring netconsole via the kernel
> cmdline results in the following panic at boot:
> 
>     netpoll: netconsole: device eth0 not up yet, forcing it
>     usb 2-1: new high-speed USB device number 2 using ehci-pci
>     ixgbe 0000:03:00.0: registered PHC device on eth0
>     BUG: unable to handle kernel NULL pointer dereference at 0000000000000810
>     <snip>
>     Call Trace:
>      [<ffffffff81578661>] ? vxlan_get_rx_port+0x41/0xa0
>      [<ffffffff81586828>] ixgbe_open+0x4e8/0x540
>      [<ffffffff8168045c>] __dev_open+0xac/0x120
>      [<ffffffff81680506>] dev_open+0x36/0x70
>      [<ffffffff8169abec>] netpoll_setup+0x23c/0x300
>      [<ffffffff8169a66a>] ? netpoll_parse_options+0x19a/0x200
>      [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
>      [<ffffffff81d79882>] init_netconsole+0xda/0x262
>      [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
>      [<ffffffff810003a8>] do_one_initcall+0x88/0x1b0
>      [<ffffffff81d31144>] kernel_init_freeable+0x14a/0x1e3
>      [<ffffffff81d308f1>] ? do_early_param+0x8c/0x8c
>      [<ffffffff81778610>] ? rest_init+0x80/0x80
>      [<ffffffff8177861e>] kernel_init+0xe/0xe0
>      [<ffffffff8177dc5f>] ret_from_fork+0x3f/0x70
>      [<ffffffff81778610>] ? rest_init+0x80/0x80
> 
> This happens because IXGBE assumes that vxlan has already been initialized.
> The cleanest way to fix this is to just initialize netconsole after all the
> other core networking stuff has completed.
> 
> Signed-off-by: Calvin Owens <calvinowens@fb.com>
> ---
>  drivers/net/Makefile | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/net/Makefile b/drivers/net/Makefile
> index 900b0c5..31557d0 100644
> --- a/drivers/net/Makefile
> +++ b/drivers/net/Makefile
> @@ -15,7 +15,6 @@ obj-$(CONFIG_MACVTAP) += macvtap.o
>  obj-$(CONFIG_MII) += mii.o
>  obj-$(CONFIG_MDIO) += mdio.o
>  obj-$(CONFIG_NET) += Space.o loopback.o
> -obj-$(CONFIG_NETCONSOLE) += netconsole.o
>  obj-$(CONFIG_PHYLIB) += phy/
>  obj-$(CONFIG_RIONET) += rionet.o
>  obj-$(CONFIG_NET_TEAM) += team/
> @@ -26,6 +25,7 @@ obj-$(CONFIG_VXLAN) += vxlan.o
>  obj-$(CONFIG_GENEVE) += geneve.o
>  obj-$(CONFIG_NLMON) += nlmon.o
>  obj-$(CONFIG_NET_VRF) += vrf.o
> +obj-$(CONFIG_NETCONSOLE) += netconsole.o
>  
>  #
>  # Networking Drivers


Looks odd to rely on link order, but we might already rely on this...

Have you considered using device_initcall() instead of late_initcall()
in vxlan ?

In any case, a comment would really be good to avoid future mistakes.

Re: [PATCH] netconsole: Initialize after all core networking drivers

[Calvin Owens]

On Thursday 12/17 at 17:08 -0800, Eric Dumazet wrote:
> On Thu, 2015-12-17 at 15:52 -0800, Calvin Owens wrote:
> > With built-in netconsole and IXGBE, configuring netconsole via the kernel
> > cmdline results in the following panic at boot:
> > 
> >     netpoll: netconsole: device eth0 not up yet, forcing it
> >     usb 2-1: new high-speed USB device number 2 using ehci-pci
> >     ixgbe 0000:03:00.0: registered PHC device on eth0
> >     BUG: unable to handle kernel NULL pointer dereference at 0000000000000810
> >     <snip>
> >     Call Trace:
> >      [<ffffffff81578661>] ? vxlan_get_rx_port+0x41/0xa0
> >      [<ffffffff81586828>] ixgbe_open+0x4e8/0x540
> >      [<ffffffff8168045c>] __dev_open+0xac/0x120
> >      [<ffffffff81680506>] dev_open+0x36/0x70
> >      [<ffffffff8169abec>] netpoll_setup+0x23c/0x300
> >      [<ffffffff8169a66a>] ? netpoll_parse_options+0x19a/0x200
> >      [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
> >      [<ffffffff81d79882>] init_netconsole+0xda/0x262
> >      [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
> >      [<ffffffff810003a8>] do_one_initcall+0x88/0x1b0
> >      [<ffffffff81d31144>] kernel_init_freeable+0x14a/0x1e3
> >      [<ffffffff81d308f1>] ? do_early_param+0x8c/0x8c
> >      [<ffffffff81778610>] ? rest_init+0x80/0x80
> >      [<ffffffff8177861e>] kernel_init+0xe/0xe0
> >      [<ffffffff8177dc5f>] ret_from_fork+0x3f/0x70
> >      [<ffffffff81778610>] ? rest_init+0x80/0x80
> > 
> > This happens because IXGBE assumes that vxlan has already been initialized.
> > The cleanest way to fix this is to just initialize netconsole after all the
> > other core networking stuff has completed.
> > 
> > Signed-off-by: Calvin Owens <calvinowens@fb.com>
> > ---
> >  drivers/net/Makefile | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/drivers/net/Makefile b/drivers/net/Makefile
> > index 900b0c5..31557d0 100644
> > --- a/drivers/net/Makefile
> > +++ b/drivers/net/Makefile
> > @@ -15,7 +15,6 @@ obj-$(CONFIG_MACVTAP) += macvtap.o
> >  obj-$(CONFIG_MII) += mii.o
> >  obj-$(CONFIG_MDIO) += mdio.o
> >  obj-$(CONFIG_NET) += Space.o loopback.o
> > -obj-$(CONFIG_NETCONSOLE) += netconsole.o
> >  obj-$(CONFIG_PHYLIB) += phy/
> >  obj-$(CONFIG_RIONET) += rionet.o
> >  obj-$(CONFIG_NET_TEAM) += team/
> > @@ -26,6 +25,7 @@ obj-$(CONFIG_VXLAN) += vxlan.o
> >  obj-$(CONFIG_GENEVE) += geneve.o
> >  obj-$(CONFIG_NLMON) += nlmon.o
> >  obj-$(CONFIG_NET_VRF) += vrf.o
> > +obj-$(CONFIG_NETCONSOLE) += netconsole.o
> >  
> >  #
> >  # Networking Drivers
> 
> 
> Looks odd to rely on link order, but we might already rely on this...
> 
> Have you considered using device_initcall() instead of late_initcall()
> in vxlan ?

I'll look. As-is though, I think a similar problem would happen if you
tried to use a virtio_net device with netconsole= cmdline (although that
is admittedly a bizarre use case). The Makefile patch seemed like the
best way to ensure this can't recur elsewhere.

> In any case, a comment would really be good to avoid future mistakes.

Good point, I'll add something.

Re: [PATCH] netconsole: Initialize after all core networking drivers

[Calvin Owens]

On Thursday 12/17 at 17:46 -0800, Calvin Owens wrote:
> On Thursday 12/17 at 17:08 -0800, Eric Dumazet wrote:
> > On Thu, 2015-12-17 at 15:52 -0800, Calvin Owens wrote:
> > > With built-in netconsole and IXGBE, configuring netconsole via the kernel
> > > cmdline results in the following panic at boot:
> > > 
> > >     netpoll: netconsole: device eth0 not up yet, forcing it
> > >     usb 2-1: new high-speed USB device number 2 using ehci-pci
> > >     ixgbe 0000:03:00.0: registered PHC device on eth0
> > >     BUG: unable to handle kernel NULL pointer dereference at 0000000000000810
> > >     <snip>
> > >     Call Trace:
> > >      [<ffffffff81578661>] ? vxlan_get_rx_port+0x41/0xa0
> > >      [<ffffffff81586828>] ixgbe_open+0x4e8/0x540
> > >      [<ffffffff8168045c>] __dev_open+0xac/0x120
> > >      [<ffffffff81680506>] dev_open+0x36/0x70
> > >      [<ffffffff8169abec>] netpoll_setup+0x23c/0x300
> > >      [<ffffffff8169a66a>] ? netpoll_parse_options+0x19a/0x200
> > >      [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
> > >      [<ffffffff81d79882>] init_netconsole+0xda/0x262
> > >      [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
> > >      [<ffffffff810003a8>] do_one_initcall+0x88/0x1b0
> > >      [<ffffffff81d31144>] kernel_init_freeable+0x14a/0x1e3
> > >      [<ffffffff81d308f1>] ? do_early_param+0x8c/0x8c
> > >      [<ffffffff81778610>] ? rest_init+0x80/0x80
> > >      [<ffffffff8177861e>] kernel_init+0xe/0xe0
> > >      [<ffffffff8177dc5f>] ret_from_fork+0x3f/0x70
> > >      [<ffffffff81778610>] ? rest_init+0x80/0x80
> > > 
> > > This happens because IXGBE assumes that vxlan has already been initialized.
> > > The cleanest way to fix this is to just initialize netconsole after all the
> > > other core networking stuff has completed.
> > > 
> > > Signed-off-by: Calvin Owens <calvinowens@fb.com>
> > > ---
> > >  drivers/net/Makefile | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/drivers/net/Makefile b/drivers/net/Makefile
> > > index 900b0c5..31557d0 100644
> > > --- a/drivers/net/Makefile
> > > +++ b/drivers/net/Makefile
> > > @@ -15,7 +15,6 @@ obj-$(CONFIG_MACVTAP) += macvtap.o
> > >  obj-$(CONFIG_MII) += mii.o
> > >  obj-$(CONFIG_MDIO) += mdio.o
> > >  obj-$(CONFIG_NET) += Space.o loopback.o
> > > -obj-$(CONFIG_NETCONSOLE) += netconsole.o
> > >  obj-$(CONFIG_PHYLIB) += phy/
> > >  obj-$(CONFIG_RIONET) += rionet.o
> > >  obj-$(CONFIG_NET_TEAM) += team/
> > > @@ -26,6 +25,7 @@ obj-$(CONFIG_VXLAN) += vxlan.o
> > >  obj-$(CONFIG_GENEVE) += geneve.o
> > >  obj-$(CONFIG_NLMON) += nlmon.o
> > >  obj-$(CONFIG_NET_VRF) += vrf.o
> > > +obj-$(CONFIG_NETCONSOLE) += netconsole.o
> > >  
> > >  #
> > >  # Networking Drivers
> > 
> > 
> > Looks odd to rely on link order, but we might already rely on this...
> > 
> > Have you considered using device_initcall() instead of late_initcall()
> > in vxlan ?
> 
> I'll look.

So this does work, but commit 7332a13b038be05c explicitly changed it to
late_initcall() because of dependencies on IPv6:

  When vxlan is compiled as builtin, its init code
  runs before IPv6 init, this could cause problems
  if we create IPv6 socket in the latter patch.

So I guess something like the following patch is needed to go that
route? It's ugly, IMHO the Makefile patch is cleaner...

Stephen / Cong, what do you think?

> As-is though, I think a similar problem would happen if you
> tried to use a virtio_net device with netconsole= cmdline (although that
> is admittedly a bizarre use case). The Makefile patch seemed like the
> best way to ensure this can't recur elsewhere.

I misunderstood this, it works fine as is.


---8<---
From: Calvin Owens <calvinowens@fb.com>
Subject: [PATCH] vxlan: Properly depend on ipv6 and revert to module_init()

Commit 7332a13b038be05c ("vxlan: defer vxlan init as late as possible")
changed vxlan to use late_initcall(), because vxlan relies on ipv6 being
loaded when a new device is opened.

This causes netconsole to panic at boot when configured via the kernel
cmdline on an IXGBE NIC, because ixgbe_open() assumes that vxlan has
already been initialized:

  netpoll: netconsole: device eth0 not up yet, forcing it
  ixgbe 0000:03:00.0: registered PHC device on eth0
  BUG: unable to handle kernel NULL pointer dereference at 0000000000000810
  <snip>
  Call Trace:
  [<ffffffff81578661>] ? vxlan_get_rx_port+0x41/0xa0
  [<ffffffff81586828>] ixgbe_open+0x4e8/0x540
  [<ffffffff8168045c>] __dev_open+0xac/0x120
  [<ffffffff81680506>] dev_open+0x36/0x70
  [<ffffffff8169abec>] netpoll_setup+0x23c/0x300
  [<ffffffff8169a66a>] ? netpoll_parse_options+0x19a/0x200
  [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
  [<ffffffff81d79882>] init_netconsole+0xda/0x262
  [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
  [<ffffffff810003a8>] do_one_initcall+0x88/0x1b0
  [<ffffffff81d31144>] kernel_init_freeable+0x14a/0x1e3
  [<ffffffff81d308f1>] ? do_early_param+0x8c/0x8c
  [<ffffffff81778610>] ? rest_init+0x80/0x80
  [<ffffffff8177861e>] kernel_init+0xe/0xe0
  [<ffffffff8177dc5f>] ret_from_fork+0x3f/0x70
  [<ffffffff81778610>] ? rest_init+0x80/0x80

This patch addresses the issue cited in 7332a13b038be05c by making vxlan
actually check if ipv6 is loaded, and reverts it to module_init() so
that it becomes device_initcall() when built-in, eliminating the
netconsole issue.

The ipv6 module is permanent, so there's no need to actually do the
usual module_get/module_put dance: once we find it loaded, we can just
assume that it always will be.

AFAICS, nothing actually ends up calling vxlan_open() during initcalls,
so in the (IPV6=y && VXLAN=y) case we can't end up there before ipv6 has
initialized.

Signed-off-by: Calvin Owens <calvinowens@fb.com>
---
 drivers/net/vxlan.c | 35 ++++++++++++++++++++++++++++++++++-
 1 file changed, 34 insertions(+), 1 deletion(-)

diff --git a/drivers/net/vxlan.c b/drivers/net/vxlan.c
index ba363ce..e3061b7 100644
--- a/drivers/net/vxlan.c
+++ b/drivers/net/vxlan.c
@@ -158,6 +158,36 @@ static int vxlan_nla_put_addr(struct sk_buff *skb, int attr,
 		return nla_put_in_addr(skb, attr, ip->sin.sin_addr.s_addr);
 }
 
+#if IS_MODULE(CONFIG_IPV6)
+MODULE_SOFTDEP("pre: ipv6");
+
+/*
+ * IPv6 is permanent, so once we find it loaded we can just assume it always
+ * will be, and avoid walking the module_list every time we open a new vxlan
+ * device.
+ */
+static bool vxlan_ipv6_is_loaded(void)
+{
+	static bool ipv6_loaded = false;
+
+	if (!ipv6_loaded) {
+		mutex_lock(&module_mutex);
+		ipv6_loaded = !!find_module("ipv6");
+		mutex_unlock(&module_mutex);
+	}
+
+	return ipv6_loaded;
+}
+
+#elif IS_BUILTIN(CONFIG_IPV6)
+
+static bool vxlan_ipv6_is_loaded(void)
+{
+	return true;
+}
+
+#endif /* IS_{MODULE,BUILTIN}(CONFIG_IPV6) */
+
 #else /* !CONFIG_IPV6 */
 
 static inline
@@ -2628,6 +2658,9 @@ static struct socket *vxlan_create_sock(struct net *net, bool ipv6,
 	memset(&udp_conf, 0, sizeof(udp_conf));
 
 	if (ipv6) {
+		if (!vxlan_ipv6_is_loaded())
+			return ERR_PTR(-EAFNOSUPPORT);
+
 		udp_conf.family = AF_INET6;
 		udp_conf.use_udp6_rx_checksums =
 		    !(flags & VXLAN_F_UDP_ZERO_CSUM6_RX);
@@ -3259,7 +3292,7 @@ out1:
 	destroy_workqueue(vxlan_wq);
 	return rc;
 }
-late_initcall(vxlan_init_module);
+module_init(vxlan_init_module);
 
 static void __exit vxlan_cleanup_module(void)
 {
-- 
2.5.0

Re: [PATCH] vxlan: Properly depend on ipv6 and revert to module_init()

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 1628 bytes --]

Hi Calvin,

[auto build test ERROR on net-next/master]
[also build test ERROR on v4.4-rc6 next-20151223]

url:    https://github.com/0day-ci/linux/commits/Calvin-Owens/vxlan-Properly-depend-on-ipv6-and-revert-to-module_init/20151224-070557
config: i386-randconfig-b0-12241357 (attached as .config)
reproduce:
        # save the attached .config to linux build tree
        make ARCH=i386 

All error/warnings (new ones prefixed by >>):

   In file included from include/linux/linkage.h:4:0,
                    from include/linux/kernel.h:6,
                    from drivers/net/vxlan.c:13:
   drivers/net/vxlan.c: In function 'vxlan_create_sock':
>> drivers/net/vxlan.c:2661:8: error: implicit declaration of function 'vxlan_ipv6_is_loaded' [-Werror=implicit-function-declaration]
      if (!vxlan_ipv6_is_loaded())
           ^
   include/linux/compiler.h:147:28: note: in definition of macro '__trace_if'
     if (__builtin_constant_p((cond)) ? !!(cond) :   \
                               ^
>> drivers/net/vxlan.c:2661:3: note: in expansion of macro 'if'
      if (!vxlan_ipv6_is_loaded())
      ^
   cc1: some warnings being treated as errors

vim +/vxlan_ipv6_is_loaded +2661 drivers/net/vxlan.c

  2655		struct udp_port_cfg udp_conf;
  2656		int err;
  2657	
  2658		memset(&udp_conf, 0, sizeof(udp_conf));
  2659	
  2660		if (ipv6) {
> 2661			if (!vxlan_ipv6_is_loaded())
  2662				return ERR_PTR(-EAFNOSUPPORT);
  2663	
  2664			udp_conf.family = AF_INET6;

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/octet-stream, Size: 26768 bytes --]

Re: [PATCH] netconsole: Initialize after all core networking drivers

[Hannes Frederic Sowa]

Hi,

On 24.12.2015 00:03, Calvin Owens wrote:
> On Thursday 12/17 at 17:46 -0800, Calvin Owens wrote:
>> On Thursday 12/17 at 17:08 -0800, Eric Dumazet wrote:
>>> On Thu, 2015-12-17 at 15:52 -0800, Calvin Owens wrote:
>>>> With built-in netconsole and IXGBE, configuring netconsole via the kernel
>>>> cmdline results in the following panic at boot:
>>>>
>>>>     netpoll: netconsole: device eth0 not up yet, forcing it
>>>>     usb 2-1: new high-speed USB device number 2 using ehci-pci
>>>>     ixgbe 0000:03:00.0: registered PHC device on eth0
>>>>     BUG: unable to handle kernel NULL pointer dereference at 0000000000000810
>>>>     <snip>
>>>>     Call Trace:
>>>>      [<ffffffff81578661>] ? vxlan_get_rx_port+0x41/0xa0
>>>>      [<ffffffff81586828>] ixgbe_open+0x4e8/0x540
>>>>      [<ffffffff8168045c>] __dev_open+0xac/0x120
>>>>      [<ffffffff81680506>] dev_open+0x36/0x70
>>>>      [<ffffffff8169abec>] netpoll_setup+0x23c/0x300
>>>>      [<ffffffff8169a66a>] ? netpoll_parse_options+0x19a/0x200
>>>>      [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
>>>>      [<ffffffff81d79882>] init_netconsole+0xda/0x262
>>>>      [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
>>>>      [<ffffffff810003a8>] do_one_initcall+0x88/0x1b0
>>>>      [<ffffffff81d31144>] kernel_init_freeable+0x14a/0x1e3
>>>>      [<ffffffff81d308f1>] ? do_early_param+0x8c/0x8c
>>>>      [<ffffffff81778610>] ? rest_init+0x80/0x80
>>>>      [<ffffffff8177861e>] kernel_init+0xe/0xe0
>>>>      [<ffffffff8177dc5f>] ret_from_fork+0x3f/0x70
>>>>      [<ffffffff81778610>] ? rest_init+0x80/0x80
>>>>
>>>> This happens because IXGBE assumes that vxlan has already been initialized.
>>>> The cleanest way to fix this is to just initialize netconsole after all the
>>>> other core networking stuff has completed.
>>>>
>>>> Signed-off-by: Calvin Owens <calvinowens@fb.com>
>>>> ---
>>>>  drivers/net/Makefile | 2 +-
>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>
>>>> diff --git a/drivers/net/Makefile b/drivers/net/Makefile
>>>> index 900b0c5..31557d0 100644
>>>> --- a/drivers/net/Makefile
>>>> +++ b/drivers/net/Makefile
>>>> @@ -15,7 +15,6 @@ obj-$(CONFIG_MACVTAP) += macvtap.o
>>>>  obj-$(CONFIG_MII) += mii.o
>>>>  obj-$(CONFIG_MDIO) += mdio.o
>>>>  obj-$(CONFIG_NET) += Space.o loopback.o
>>>> -obj-$(CONFIG_NETCONSOLE) += netconsole.o
>>>>  obj-$(CONFIG_PHYLIB) += phy/
>>>>  obj-$(CONFIG_RIONET) += rionet.o
>>>>  obj-$(CONFIG_NET_TEAM) += team/
>>>> @@ -26,6 +25,7 @@ obj-$(CONFIG_VXLAN) += vxlan.o
>>>>  obj-$(CONFIG_GENEVE) += geneve.o
>>>>  obj-$(CONFIG_NLMON) += nlmon.o
>>>>  obj-$(CONFIG_NET_VRF) += vrf.o
>>>> +obj-$(CONFIG_NETCONSOLE) += netconsole.o
>>>>  
>>>>  #
>>>>  # Networking Drivers
>>>
>>>
>>> Looks odd to rely on link order, but we might already rely on this...
>>>
>>> Have you considered using device_initcall() instead of late_initcall()
>>> in vxlan ?
>>
>> I'll look.
> 
> So this does work, but commit 7332a13b038be05c explicitly changed it to
> late_initcall() because of dependencies on IPv6:
> 
>   When vxlan is compiled as builtin, its init code
>   runs before IPv6 init, this could cause problems
>   if we create IPv6 socket in the latter patch.
> 
> So I guess something like the following patch is needed to go that
> route? It's ugly, IMHO the Makefile patch is cleaner...
> 
> Stephen / Cong, what do you think?
> 
>> As-is though, I think a similar problem would happen if you
>> tried to use a virtio_net device with netconsole= cmdline (although that
>> is admittedly a bizarre use case). The Makefile patch seemed like the
>> best way to ensure this can't recur elsewhere.
> 
> I misunderstood this, it works fine as is.
> 
> 
> ---8<---
> From: Calvin Owens <calvinowens@fb.com>
> Subject: [PATCH] vxlan: Properly depend on ipv6 and revert to module_init()
> 
> Commit 7332a13b038be05c ("vxlan: defer vxlan init as late as possible")
> changed vxlan to use late_initcall(), because vxlan relies on ipv6 being
> loaded when a new device is opened.
> 
> This causes netconsole to panic at boot when configured via the kernel
> cmdline on an IXGBE NIC, because ixgbe_open() assumes that vxlan has
> already been initialized:
> 
>   netpoll: netconsole: device eth0 not up yet, forcing it
>   ixgbe 0000:03:00.0: registered PHC device on eth0
>   BUG: unable to handle kernel NULL pointer dereference at 0000000000000810
>   <snip>
>   Call Trace:
>   [<ffffffff81578661>] ? vxlan_get_rx_port+0x41/0xa0
>   [<ffffffff81586828>] ixgbe_open+0x4e8/0x540
>   [<ffffffff8168045c>] __dev_open+0xac/0x120
>   [<ffffffff81680506>] dev_open+0x36/0x70
>   [<ffffffff8169abec>] netpoll_setup+0x23c/0x300
>   [<ffffffff8169a66a>] ? netpoll_parse_options+0x19a/0x200
>   [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
>   [<ffffffff81d79882>] init_netconsole+0xda/0x262
>   [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
>   [<ffffffff810003a8>] do_one_initcall+0x88/0x1b0
>   [<ffffffff81d31144>] kernel_init_freeable+0x14a/0x1e3
>   [<ffffffff81d308f1>] ? do_early_param+0x8c/0x8c
>   [<ffffffff81778610>] ? rest_init+0x80/0x80
>   [<ffffffff8177861e>] kernel_init+0xe/0xe0
>   [<ffffffff8177dc5f>] ret_from_fork+0x3f/0x70
>   [<ffffffff81778610>] ? rest_init+0x80/0x80
> 
> This patch addresses the issue cited in 7332a13b038be05c by making vxlan
> actually check if ipv6 is loaded, and reverts it to module_init() so
> that it becomes device_initcall() when built-in, eliminating the
> netconsole issue.
> 
> The ipv6 module is permanent, so there's no need to actually do the
> usual module_get/module_put dance: once we find it loaded, we can just
> assume that it always will be.
> 
> AFAICS, nothing actually ends up calling vxlan_open() during initcalls,
> so in the (IPV6=y && VXLAN=y) case we can't end up there before ipv6 has
> initialized.
> 
> Signed-off-by: Calvin Owens <calvinowens@fb.com>

This architecture just sucks. :(


ixgbe should not have to call into vxlan but vxlan has to call to ixgbe.
Thus the vxlan_get_rx_port is absolutely unnecessary and should be
removed. This also lets ixgbe depend on vxlan which is absurd.

Simply let vxlan_get_rx_port be called from vxlan_notifier_block on
NETDEV_REGISTER or NETDEV_UP events, which is already available.

For the second vxlan_get_rx_port case, which is a
IXGBE_FLAG2_VXLAN_REREG_NEEDED needed event, I would suggest we also
push that over to the vxlan_notifier_block, maybe with a new event type
for the notifiers.

After this change ixgbe would not depend on vxlan module any more.

Thanks,
Hannes

Re: [PATCH] netconsole: Initialize after all core networking drivers

[Cong Wang]

On Thu, Dec 24, 2015 at 2:25 AM, Hannes Frederic Sowa
<hannes@stressinduktion.org> wrote:
> Hi,
>
> On 24.12.2015 00:03, Calvin Owens wrote:
>> This patch addresses the issue cited in 7332a13b038be05c by making vxlan
>> actually check if ipv6 is loaded, and reverts it to module_init() so
>> that it becomes device_initcall() when built-in, eliminating the
>> netconsole issue.
>>
>> The ipv6 module is permanent, so there's no need to actually do the
>> usual module_get/module_put dance: once we find it loaded, we can just
>> assume that it always will be.
>>
>> AFAICS, nothing actually ends up calling vxlan_open() during initcalls,
>> so in the (IPV6=y && VXLAN=y) case we can't end up there before ipv6 has
>> initialized.
>>
>> Signed-off-by: Calvin Owens <calvinowens@fb.com>
>
> This architecture just sucks. :(
>
>
> ixgbe should not have to call into vxlan but vxlan has to call to ixgbe.
> Thus the vxlan_get_rx_port is absolutely unnecessary and should be
> removed. This also lets ixgbe depend on vxlan which is absurd.
>
> Simply let vxlan_get_rx_port be called from vxlan_notifier_block on
> NETDEV_REGISTER or NETDEV_UP events, which is already available.
>
> For the second vxlan_get_rx_port case, which is a
> IXGBE_FLAG2_VXLAN_REREG_NEEDED needed event, I would suggest we also
> push that over to the vxlan_notifier_block, maybe with a new event type
> for the notifiers.
>
> After this change ixgbe would not depend on vxlan module any more.
>

Agreed with this direction, but not yet convinced with this approach.

My feeling is that we should move this piece of code into core
networking, so that both vxlan and ixgbe driver can call it without
worrying about the dependencies. It looks like all we need here
is a per-netns structure about family and ports for all vxlans.
But I could overlook something here.

Thanks.

Re: [PATCH] netconsole: Initialize after all core networking drivers

[Hannes Frederic Sowa]

On 29.12.2015 00:21, Cong Wang wrote:
> On Thu, Dec 24, 2015 at 2:25 AM, Hannes Frederic Sowa
> <hannes@stressinduktion.org> wrote:
>> Hi,
>>
>> On 24.12.2015 00:03, Calvin Owens wrote:
>>> This patch addresses the issue cited in 7332a13b038be05c by making vxlan
>>> actually check if ipv6 is loaded, and reverts it to module_init() so
>>> that it becomes device_initcall() when built-in, eliminating the
>>> netconsole issue.
>>>
>>> The ipv6 module is permanent, so there's no need to actually do the
>>> usual module_get/module_put dance: once we find it loaded, we can just
>>> assume that it always will be.
>>>
>>> AFAICS, nothing actually ends up calling vxlan_open() during initcalls,
>>> so in the (IPV6=y && VXLAN=y) case we can't end up there before ipv6 has
>>> initialized.
>>>
>>> Signed-off-by: Calvin Owens <calvinowens@fb.com>
>>
>> This architecture just sucks. :(
>>
>>
>> ixgbe should not have to call into vxlan but vxlan has to call to ixgbe.
>> Thus the vxlan_get_rx_port is absolutely unnecessary and should be
>> removed. This also lets ixgbe depend on vxlan which is absurd.
>>
>> Simply let vxlan_get_rx_port be called from vxlan_notifier_block on
>> NETDEV_REGISTER or NETDEV_UP events, which is already available.
>>
>> For the second vxlan_get_rx_port case, which is a
>> IXGBE_FLAG2_VXLAN_REREG_NEEDED needed event, I would suggest we also
>> push that over to the vxlan_notifier_block, maybe with a new event type
>> for the notifiers.
>>
>> After this change ixgbe would not depend on vxlan module any more.
>>
>
> Agreed with this direction, but not yet convinced with this approach.
>
> My feeling is that we should move this piece of code into core
> networking, so that both vxlan and ixgbe driver can call it without
> worrying about the dependencies. It looks like all we need here
> is a per-netns structure about family and ports for all vxlans.
> But I could overlook something here.

We actually have that already in terms of net_generic(vxlan_net_id). We 
will also need that for geneve and other offloads supported by hardware.

Your approach seems to be basically the same like mine, just in your 
proposal the driver calls to all the helper functions, while in my 
proposal the vxlan/geneve and other layers can update the offload 
settings on the hardware.

As the notifier_blocks are already available and registered, why not 
just use them?

Thanks,
Hannes

Re: [PATCH] netconsole: Initialize after all core networking drivers

[Cong Wang]

On Wed, Dec 23, 2015 at 3:03 PM, Calvin Owens <calvinowens@fb.com> wrote:
> Commit 7332a13b038be05c ("vxlan: defer vxlan init as late as possible")
> changed vxlan to use late_initcall(), because vxlan relies on ipv6 being
> loaded when a new device is opened.
>
> This causes netconsole to panic at boot when configured via the kernel
> cmdline on an IXGBE NIC, because ixgbe_open() assumes that vxlan has
> already been initialized:

BTW, it is not fair to blame my commit above, since it was merged long before
ixgbe or any hardware driver has vxlan support. Not a big deal, just FYI. ;)

Re: [PATCH] netconsole: Initialize after all core networking drivers

[Stephen Hemminger]

On Thu, 17 Dec 2015 15:52:39 -0800
Calvin Owens <calvinowens@fb.com> wrote:

> With built-in netconsole and IXGBE, configuring netconsole via the kernel
> cmdline results in the following panic at boot:
> 
>     netpoll: netconsole: device eth0 not up yet, forcing it
>     usb 2-1: new high-speed USB device number 2 using ehci-pci
>     ixgbe 0000:03:00.0: registered PHC device on eth0
>     BUG: unable to handle kernel NULL pointer dereference at 0000000000000810
>     <snip>
>     Call Trace:
>      [<ffffffff81578661>] ? vxlan_get_rx_port+0x41/0xa0
>      [<ffffffff81586828>] ixgbe_open+0x4e8/0x540
>      [<ffffffff8168045c>] __dev_open+0xac/0x120
>      [<ffffffff81680506>] dev_open+0x36/0x70
>      [<ffffffff8169abec>] netpoll_setup+0x23c/0x300
>      [<ffffffff8169a66a>] ? netpoll_parse_options+0x19a/0x200
>      [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
>      [<ffffffff81d79882>] init_netconsole+0xda/0x262
>      [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
>      [<ffffffff810003a8>] do_one_initcall+0x88/0x1b0
>      [<ffffffff81d31144>] kernel_init_freeable+0x14a/0x1e3
>      [<ffffffff81d308f1>] ? do_early_param+0x8c/0x8c
>      [<ffffffff81778610>] ? rest_init+0x80/0x80
>      [<ffffffff8177861e>] kernel_init+0xe/0xe0
>      [<ffffffff8177dc5f>] ret_from_fork+0x3f/0x70
>      [<ffffffff81778610>] ? rest_init+0x80/0x80
> 
> This happens because IXGBE assumes that vxlan has already been initialized.
> The cleanest way to fix this is to just initialize netconsole after all the
> other core networking stuff has completed.
> 
> Signed-off-by: Calvin Owens <calvinowens@fb.com>

Fixing this by changing Makefile order is too fragile.
You are depending on the fact that Makefile order determines link order
and that determines initialization order. Down that path demons lie.

Re: [PATCH] netconsole: Initialize after all core networking drivers

[Calvin Owens]

On Thursday 12/17 at 17:10 -0800, Stephen Hemminger wrote:
> On Thu, 17 Dec 2015 15:52:39 -0800
> Calvin Owens <calvinowens@fb.com> wrote:
> 
> > With built-in netconsole and IXGBE, configuring netconsole via the kernel
> > cmdline results in the following panic at boot:
> > 
> >     netpoll: netconsole: device eth0 not up yet, forcing it
> >     usb 2-1: new high-speed USB device number 2 using ehci-pci
> >     ixgbe 0000:03:00.0: registered PHC device on eth0
> >     BUG: unable to handle kernel NULL pointer dereference at 0000000000000810
> >     <snip>
> >     Call Trace:
> >      [<ffffffff81578661>] ? vxlan_get_rx_port+0x41/0xa0
> >      [<ffffffff81586828>] ixgbe_open+0x4e8/0x540
> >      [<ffffffff8168045c>] __dev_open+0xac/0x120
> >      [<ffffffff81680506>] dev_open+0x36/0x70
> >      [<ffffffff8169abec>] netpoll_setup+0x23c/0x300
> >      [<ffffffff8169a66a>] ? netpoll_parse_options+0x19a/0x200
> >      [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
> >      [<ffffffff81d79882>] init_netconsole+0xda/0x262
> >      [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
> >      [<ffffffff810003a8>] do_one_initcall+0x88/0x1b0
> >      [<ffffffff81d31144>] kernel_init_freeable+0x14a/0x1e3
> >      [<ffffffff81d308f1>] ? do_early_param+0x8c/0x8c
> >      [<ffffffff81778610>] ? rest_init+0x80/0x80
> >      [<ffffffff8177861e>] kernel_init+0xe/0xe0
> >      [<ffffffff8177dc5f>] ret_from_fork+0x3f/0x70
> >      [<ffffffff81778610>] ? rest_init+0x80/0x80
> > 
> > This happens because IXGBE assumes that vxlan has already been initialized.
> > The cleanest way to fix this is to just initialize netconsole after all the
> > other core networking stuff has completed.
> > 
> > Signed-off-by: Calvin Owens <calvinowens@fb.com>
> 
> Fixing this by changing Makefile order is too fragile.
> You are depending on the fact that Makefile order determines link order
> and that determines initialization order. Down that path demons lie.

Hmm, include/linux/init.h explicitly states that "Ordering inside the
subsections is determined by link order", and has since before the
beginning of the history in Git.

I agree it seems magic/scary, but as Eric says I would imagine this
behavior is relied upon in other places.