Problem with netconsole and eth0 timing

Problem with netconsole and eth0 timing

[don fisher]

I am trying to use netconsole to examine some problems I am having 
booting Alienware 13 r3 and 14 laptops. I am running opensuse 42.3 with 
generic 4.18.7 kernel configured to include netconsole. After building 
my kernel and including the proper command (as shown below) in the linux 
boot string:
  netconsole=64001 at 192.168.7.60/eth0,64001 at 192.168.7.55/34:e6:d7:01:2a:dd
I receive the following error message, as shown by journalctl:

sudo journalctl -b | grep netconsole
Sep 24 22:01:25 dfpc60 kernel: netpoll: netconsole: local port 64001
Sep 24 22:01:25 dfpc60 kernel: netpoll: netconsole: local IPv4 address 
192.168.7.60
Sep 24 22:01:25 dfpc60 kernel: netpoll: netconsole: interface 'eth0'
Sep 24 22:01:25 dfpc60 kernel: netpoll: netconsole: remote port 64001
Sep 24 22:01:25 dfpc60 kernel: netpoll: netconsole: remote IPv4 address 
192.168.7.55
Sep 24 22:01:25 dfpc60 kernel: netpoll: netconsole: remote ethernet 
address 34:e6:d7:01:2a:dd
Sep 24 22:01:25 dfpc60 kernel: netpoll: netconsole: eth0 doesn't exist, 
aborting
Sep 24 22:01:25 dfpc60 kernel: netconsole: cleaning up
Sep 24 22:01:25 dfpc60 systemd-modules-load[185]: Module 'netconsole' is 
builtin

I examined the timing in more detail and found that the above "eth0: 
doesn't exist" message comes at Sep 24 22:01:25.080603.

The wicked message eth0: up comes at Sep 24 22:02:01.173616. The 
difference is maybe 36 seconds? There is an eth0: avail message at Sep 
24 22:01:34.112744, don't know if that would suffice for netconsole Both 
are after netconsole has bailed out. Any obvious solutions I am missing? 
The documentation is pretty clear on how to set this up, so there must 
be some way to get it to work. I could find nothing on Google.

This is my first post to this list, so if this question is not 
appropriate please let me know without flames:-)

Don

Problem with netconsole and eth0 timing

[valdis.kletnieks at vt.edu]

On Tue, 25 Sep 2018 18:26:06 -0700, don fisher said:

> The wicked message eth0: up comes at Sep 24 22:02:01.173616. The
> difference is maybe 36 seconds? There is an eth0: avail message at Sep
> 24 22:01:34.112744, don't know if that would suffice for netconsole Both
> are after netconsole has bailed out. Any obvious solutions I am missing?
> The documentation is pretty clear on how to set this up, so there must
> be some way to get it to work. I could find nothing on Google.

Here's the big clue:

Sep 24 22:01:25 dfpc60 systemd-modules-load[185]: Module 'netconsole' is builtin

which means that it's going to get initialized during *very* early boot, before
the initramfs gets called. netconsole is able to do the equivalent of
'ifconfig' (or at least enough of it to set the IPs/ports/ARP entries before
the rest of networking comes up), but it can't also get the physical device up and
running if the hardware driver isn't present.

Since it's builtin, this is probably a custom-built kernel.  So make sure that
the driver for eth0 is also builtin.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 486 bytes
Desc: not available
URL: <http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20180925/674d071e/attachment.sig>

Problem with netconsole and eth0 timing

[don fisher]

On 9/25/18 7:26 PM, valdis.kletnieks at vt.edu wrote:
> On Tue, 25 Sep 2018 18:26:06 -0700, don fisher said:
> 
>> The wicked message eth0: up comes at Sep 24 22:02:01.173616. The
>> difference is maybe 36 seconds? There is an eth0: avail message at Sep
>> 24 22:01:34.112744, don't know if that would suffice for netconsole Both
>> are after netconsole has bailed out. Any obvious solutions I am missing?
>> The documentation is pretty clear on how to set this up, so there must
>> be some way to get it to work. I could find nothing on Google.
> 
> Here's the big clue:
> 
> Sep 24 22:01:25 dfpc60 systemd-modules-load[185]: Module 'netconsole' is builtin
> 
> which means that it's going to get initialized during *very* early boot, before
> the initramfs gets called. netconsole is able to do the equivalent of
> 'ifconfig' (or at least enough of it to set the IPs/ports/ARP entries before
> the rest of networking comes up), but it can't also get the physical device up and
> running if the hardware driver isn't present.
> 
> Since it's builtin, this is probably a custom-built kernel.  So make sure that
> the driver for eth0 is also builtin.
> 
Would you tell me how to tell the driver that it is to be eth0, ip 
address etc. Maybe on linux command line, but I do not know the format.

Thanks
Don

Problem with netconsole and eth0 timing

[valdis.kletnieks at vt.edu]

On Wed, 26 Sep 2018 13:25:35 -0700, don fisher said:
> Would you tell me how to tell the driver that it is to be eth0, ip
> address etc. Maybe on linux command line, but I do not know the format.

To quote some guy named Don Fisher:

> my kernel and including the proper command (as shown below) in the linux boot string:
>   netconsole=64001 at 192.168.7.60/eth0,64001 at 192.168.7.55/34:e6:d7:01:2a:dd

That's how. The netconsole command gets the info it needs from there, and tells
the network layer how to configure the ethernet device and the network layer -
although mostly the network layer.  And the devices will auto-name themselves
during boot, so all you need to do is know *which* name the kernel gives to the
port you want to use, and then use that name.

So grovel around in dmesg, and look for lines like (2 examples I have handy here)

 grep eth /var/log/dmesg
[    7.278395] igb 0000:07:00.0: added PHC on eth0
[    7.278398] igb 0000:07:00.0: eth0: (PCIe:5.0Gb/s:Width x2) 24:6e:96:10:db:64
[    7.278916] igb 0000:07:00.0: eth0: PBA No: G61346-000
[    7.368911] igb 0000:07:00.1: added PHC on eth1
[    7.368913] igb 0000:07:00.1: eth1: (PCIe:5.0Gb/s:Width x2) 24:6e:96:10:db:65
[    7.369372] igb 0000:07:00.1: eth1: PBA No: G61346-000

dmesg | grep eth
[    2.642006] e1000e 0000:00:19.0 eth0: (PCI Express:2.5GT/s:Width x1) f0:1f:af:0c:8a:da
[    2.642076] e1000e 0000:00:19.0 eth0: Intel(R) PRO/1000 Network Connection
[    2.642118] e1000e 0000:00:19.0 eth0: MAC: 10, PHY: 11, PBA No: 7011FF-0FF
[    5.071095] e1000e 0000:00:19.0 eno1: renamed from eth0
[   44.516004] e1000e 0000:00:19.0 eth0: renamed from eno1

And find the one that has the MAC address of the port you want to use. Note
that you want the kernel-assigned device name, not the one that udev/systemd
finally assign to the device. So for the second example (my laptop), you'd want
either the eth0 or eno1 name (depending which one was in effect when the
netconsole module initializes.  The initial eth0 is from pci enumeration, the
rename to eno1 is courtesy of the kernel, and then the rename back to eth0 by
systemd.  Some experimentation may be needed (I've got a few servers that have
4 1G ports on the motherboard and multiple 10G/40G dual-port cards, so
sometimes the port wired to our management network ends up at eth6 or eth7...)

So if the port you want to use gets named eth4 by the kernel, you use
netconsole=64001 at 192.168.7.60/eth4,64001 at 192.168.7.55/34:e6:d7:01:2a:dd

(Gory ethernet details follow :)

Remember that strictly speaking, the ethernet device itself doesn't *need* to
know what its IP address is - it only needs to know its own MAC address so it
knows which packets on the wire to accept to hand to the network stack, and
*maybe* a list of other MAC addresses it should accept.

And the hardware already knows its own MAC address.. :)

You can get ethernet devices working with a *very* small set of functions:

0) Tell the kernel your hardware state (link/no link, MAC address, a few other things)
1) Receive packets for your own MAC address
2) Receive broadcast packets
3) Receive packets for another specified MAC address (semi-optional)
4) Receive packets in promiscuous mode (semi-optional)
5) Transmit packet to the MAC address provided

Pretty much everything else can be done in kernelspace (though modern
cards often provide offload of some IP and even TCP handling, interrupt
coalescing, and all sorts of other stuff)

(I learned far too much about minimalist Ethernet when the Clarkson Packet
Drivers were getting created in the cubicle next to mine. :)


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 486 bytes
Desc: not available
URL: <http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20180926/540a2ddf/attachment.sig>

Problem with netconsole and eth0 timing

[don fisher]

On 9/26/18 2:33 PM, valdis.kletnieks at vt.edu wrote:
> On Wed, 26 Sep 2018 13:25:35 -0700, don fisher said:
>> Would you tell me how to tell the driver that it is to be eth0, ip
>> address etc. Maybe on linux command line, but I do not know the format.
> 
> To quote some guy named Don Fisher:
> 
>> my kernel and including the proper command (as shown below) in the linux boot string:
>>    netconsole=64001 at 192.168.7.60/eth0,64001 at 192.168.7.55/34:e6:d7:01:2a:dd
> 
> That's how. The netconsole command gets the info it needs from there, and tells
> the network layer how to configure the ethernet device and the network layer -
> although mostly the network layer.  And the devices will auto-name themselves
> during boot, so all you need to do is know *which* name the kernel gives to the
> port you want to use, and then use that name.
> 
> So grovel around in dmesg, and look for lines like (2 examples I have handy here)
> 
>   grep eth /var/log/dmesg
> [    7.278395] igb 0000:07:00.0: added PHC on eth0
> [    7.278398] igb 0000:07:00.0: eth0: (PCIe:5.0Gb/s:Width x2) 24:6e:96:10:db:64
> [    7.278916] igb 0000:07:00.0: eth0: PBA No: G61346-000
> [    7.368911] igb 0000:07:00.1: added PHC on eth1
> [    7.368913] igb 0000:07:00.1: eth1: (PCIe:5.0Gb/s:Width x2) 24:6e:96:10:db:65
> [    7.369372] igb 0000:07:00.1: eth1: PBA No: G61346-000
> 
> dmesg | grep eth
> [    2.642006] e1000e 0000:00:19.0 eth0: (PCI Express:2.5GT/s:Width x1) f0:1f:af:0c:8a:da
> [    2.642076] e1000e 0000:00:19.0 eth0: Intel(R) PRO/1000 Network Connection
> [    2.642118] e1000e 0000:00:19.0 eth0: MAC: 10, PHY: 11, PBA No: 7011FF-0FF
> [    5.071095] e1000e 0000:00:19.0 eno1: renamed from eth0
> [   44.516004] e1000e 0000:00:19.0 eth0: renamed from eno1
> 
> And find the one that has the MAC address of the port you want to use. Note
> that you want the kernel-assigned device name, not the one that udev/systemd
> finally assign to the device. So for the second example (my laptop), you'd want
> either the eth0 or eno1 name (depending which one was in effect when the
> netconsole module initializes.  The initial eth0 is from pci enumeration, the
> rename to eno1 is courtesy of the kernel, and then the rename back to eth0 by
> systemd.  Some experimentation may be needed (I've got a few servers that have
> 4 1G ports on the motherboard and multiple 10G/40G dual-port cards, so
> sometimes the port wired to our management network ends up at eth6 or eth7...)
> 
> So if the port you want to use gets named eth4 by the kernel, you use
> netconsole=64001 at 192.168.7.60/eth4,64001 at 192.168.7.55/34:e6:d7:01:2a:dd
> 
> (Gory ethernet details follow :)
> 
> Remember that strictly speaking, the ethernet device itself doesn't *need* to
> know what its IP address is - it only needs to know its own MAC address so it
> knows which packets on the wire to accept to hand to the network stack, and
> *maybe* a list of other MAC addresses it should accept.
> 
> And the hardware already knows its own MAC address.. :)
> 
> You can get ethernet devices working with a *very* small set of functions:
> 
> 0) Tell the kernel your hardware state (link/no link, MAC address, a few other things)
> 1) Receive packets for your own MAC address
> 2) Receive broadcast packets
> 3) Receive packets for another specified MAC address (semi-optional)
> 4) Receive packets in promiscuous mode (semi-optional)
> 5) Transmit packet to the MAC address provided
> 
> Pretty much everything else can be done in kernelspace (though modern
> cards often provide offload of some IP and even TCP handling, interrupt
> coalescing, and all sorts of other stuff)
> 
> (I learned far too much about minimalist Ethernet when the Clarkson Packet
> Drivers were getting created in the cubicle next to mine. :)

Thanks. I tried building with the driver embedded in the kernel, but the 
compile failed with a halt. No crash is apparent, just a halt. It turned 
out that this was repeated until I removed the netconsole command during 
boot. System appears stable now. I will try tomorrow to embed the 
driver, then add netconsole option in the command line.

Don

Problem with netconsole and eth0 timing

[valdis.kletnieks at vt.edu]

On Wed, 26 Sep 2018 21:38:27 -0700, don fisher said:

> Thanks. I tried building with the driver embedded in the kernel, but the
> compile failed with a halt. No crash is apparent, just a halt. It turned
> out that this was repeated until I removed the netconsole command during
> boot. System appears stable now. I will try tomorrow to embed the
> driver, then add netconsole option in the command line.

Wait, what?  The *compile* "failed with a halt"?  What the heck does that mean?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 486 bytes
Desc: not available
URL: <http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20180927/59a1bb76/attachment-0001.sig>

Problem with netconsole and eth0 timing

[don fisher]

On 9/27/18 12:01 AM, valdis.kletnieks at vt.edu wrote:
> On Wed, 26 Sep 2018 21:38:27 -0700, don fisher said:
> 
>> Thanks. I tried building with the driver embedded in the kernel, but the
>> compile failed with a halt. No crash is apparent, just a halt. It turned
>> out that this was repeated until I removed the netconsole command during
>> boot. System appears stable now. I will try tomorrow to embed the
>> driver, then add netconsole option in the command line.
> 
> Wait, what?  The *compile* "failed with a halt"?  What the heck does that mean?
> 
> 
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> 
Don't know what it means. The compile just happened to be what I was 
execution. The system just stopped with no output to screen, dmesg or 
journal. With trial an error I discovered that if I eliminated the 
netconsole command from the grub2 linux command line, the system 
appeared stable again. I put that netconsole command in 
/etc/default/grub, so it is sort of a pain to insert and remove it.

Don

Problem with netconsole and eth0 timing

[don fisher]

On 9/27/18 3:16 PM, don fisher wrote:
> On 9/27/18 12:01 AM, valdis.kletnieks at vt.edu wrote:
>> On Wed, 26 Sep 2018 21:38:27 -0700, don fisher said:
>>
>>> Thanks. I tried building with the driver embedded in the kernel, but the
>>> compile failed with a halt. No crash is apparent, just a halt. It turned
>>> out that this was repeated until I removed the netconsole command during
>>> boot. System appears stable now. I will try tomorrow to embed the
>>> driver, then add netconsole option in the command line.
>>
>> Wait, what?? The *compile* "failed with a halt"?? What the heck does 
>> that mean?

> Don't know what it means. The compile just happened to be what I was 
> execution. The system just stopped with no output to screen, dmesg or 
> journal. With trial an error I discovered that if I eliminated the 
> netconsole command from the grub2 linux command line, the system 
> appeared stable again. I put that netconsole command in 
> /etc/default/grub, so it is sort of a pain to insert and remove it.
> 
> Don
As threatened, I rebuilt with the alx driver embedded. I tested this 
kernel just to make sure the alx driver still supported standard 
Ethernet, which it did. I then add the linux netconsole command and 
rebooted. Everything worked well for awhile, but the output remote 
output stopped at 11.909 sec, while dmesg has entries up to 12.627 sec, 
and later up to 206.453 sec. The last message from dmesg is "work still 
pending". Before, at 11.977 sec there was a "No iBFT detected" message. 
This is about the time output terminated. I do not know what iBFT is. 
The netconsole on the receiver, nc -u -l 64001, is still running at 100% 
cpu utilization.

Journal ctrl on  the source gave:

701:Sep 27 16:25:01 dfpc60 kernel: netpoll: netconsole: local port 64001
703:Sep 27 16:25:01 dfpc60 kernel: netpoll: netconsole: local IPv4 
address 192.168.7.60
705:Sep 27 16:25:01 dfpc60 kernel: netpoll: netconsole: interface 'eth0'
706:Sep 27 16:25:01 dfpc60 kernel: netpoll: netconsole: remote port 64001
707:Sep 27 16:25:01 dfpc60 kernel: netpoll: netconsole: remote IPv4 
address 192.168.7.55
709:Sep 27 16:25:01 dfpc60 kernel: netpoll: netconsole: remote ethernet 
address 34:e6:d7:01:2a:dd
711:Sep 27 16:25:01 dfpc60 kernel: netpoll: netconsole: device eth0 not 
up yet, forcing it
719:Sep 27 16:25:01 dfpc60 kernel: netconsole: network logging started
750:Sep 27 16:25:01 dfpc60 systemd-modules-load[186]: Module 
'netconsole' is builtin

Is the forcing it correct?

Don