[RFC PATCH RT] netconsole: enable netconsole on PREEMPT_RT

[RFC PATCH RT] netconsole: enable netconsole on PREEMPT_RT

[Daniel Bristot de Oliveira]

Dear RT Folks,

This patch enables the netconsole on PREEMPT_RT_FULL, netconsole
was disabled on commit:
cb7cb77 kconfig-disable-a-few-options-rt.patch

I tested the netconsole on three machines:

- Intel Core i5 (4 cpus) - Local network
- Intel Xeon (32 cpus)   - Remote network
- KVM VM (on a Core i7)  - Virtual network

On these machines I build the kernel 3.10.34-rt34 + netconsole
built as module. In order to execute some printks, I run a small
benchmark that runs as a module. To analyse the output I captured
the netconsole messages using Wireshark.

On all cases, the tests passed without any error:

- No error messages from the kernel
- No missed messages
- No out of order messages

I ran +1 hour tests on each machine. The test using the Xeon machine
ran by around 10 hours. On this test I received 317149 netconsole's
messages.

I also read the code and ran some traces (using the ftrace's
function_graph). In the traces that I read, the netconsole code never
broke the rt lock assumptions.

Signed-off-by: Daniel Bristot de Oliveira <bristot@redhat.com>
---
 drivers/net/Kconfig | 1 -
 1 file changed, 1 deletion(-)

diff --git a/drivers/net/Kconfig b/drivers/net/Kconfig
index 096de88..3835321 100644
--- a/drivers/net/Kconfig
+++ b/drivers/net/Kconfig
@@ -164,7 +164,6 @@ config VXLAN
 
 config NETCONSOLE
 	tristate "Network console logging support"
-	depends on !PREEMPT_RT_FULL
 	---help---
 	If you want to log kernel messages over the network, enable this.
 	See <file:Documentation/networking/netconsole.txt> for details.
-- 
1.9.0

Re: [RFC PATCH RT] netconsole: enable netconsole on PREEMPT_RT

[Paul Gortmaker]

On 14-04-29 01:55 PM, Daniel Bristot de Oliveira wrote:
> Dear RT Folks,
> 
> This patch enables the netconsole on PREEMPT_RT_FULL, netconsole
> was disabled on commit:
> cb7cb77 kconfig-disable-a-few-options-rt.patch
> 
> I tested the netconsole on three machines:
> 
> - Intel Core i5 (4 cpus) - Local network
> - Intel Xeon (32 cpus)   - Remote network
> - KVM VM (on a Core i7)  - Virtual network

Chances are the two real targets above were using the
same igb network driver.  I wonder if it matters what
underlying nic is used for netconsole?

It is probably worthwhile to do some data mining into why
it was disabled in the 1st place.  I can't say I recall, and
it is possible that the reason has long since become irrelevant,
even though the patch has been carried forward over time.

Or it may be that your testing simply doesn't cover the use
case where netconsole will spew sleeping while atomic...

P.
--

> 
> On these machines I build the kernel 3.10.34-rt34 + netconsole
> built as module. In order to execute some printks, I run a small
> benchmark that runs as a module. To analyse the output I captured
> the netconsole messages using Wireshark.
> 
> On all cases, the tests passed without any error:
> 
> - No error messages from the kernel
> - No missed messages
> - No out of order messages
> 
> I ran +1 hour tests on each machine. The test using the Xeon machine
> ran by around 10 hours. On this test I received 317149 netconsole's
> messages.
> 
> I also read the code and ran some traces (using the ftrace's
> function_graph). In the traces that I read, the netconsole code never
> broke the rt lock assumptions.
> 
> Signed-off-by: Daniel Bristot de Oliveira <bristot@redhat.com>
> ---
>  drivers/net/Kconfig | 1 -
>  1 file changed, 1 deletion(-)
> 
> diff --git a/drivers/net/Kconfig b/drivers/net/Kconfig
> index 096de88..3835321 100644
> --- a/drivers/net/Kconfig
> +++ b/drivers/net/Kconfig
> @@ -164,7 +164,6 @@ config VXLAN
>  
>  config NETCONSOLE
>  	tristate "Network console logging support"
> -	depends on !PREEMPT_RT_FULL
>  	---help---
>  	If you want to log kernel messages over the network, enable this.
>  	See <file:Documentation/networking/netconsole.txt> for details.
> 

Re: [RFC PATCH RT] netconsole: enable netconsole on PREEMPT_RT

[Daniel Bristot de Oliveira]

On 05/01/2014 11:15 AM, Paul Gortmaker wrote:
> Chances are the two real targets above were using the
> same igb network driver.  I wonder if it matters what
> underlying nic is used for netconsole?

These three machines are using different drivers:

+----------+------------+
| Machine  | Nic Module |
+----------+------------+
| Xeon     | tg3        |
| core i5  | r8169      |
| KVM VM   | virtio_net |
+----------+------------+

Netconsole uses the function netpoll_send_udp to send a packet,
and netpool_send_udp calls the driver's ".ndo_start_xmit" function.
So it end up calling the NIC driver, but it's not "hardware 
dependent".
 
> It is probably worthwhile to do some data mining into why
> it was disabled in the 1st place.  I can't say I recall, and
> it is possible that the reason has long since become irrelevant,
> even though the patch has been carried forward over time.

Figure out why it was disabled is one objective of this RFC :-).
I did not found any message about some error of netconsole on RT 
kernel. I enabled the netconsole waiting for some obscure bug,
but surprisingly it always worked fine.

> Or it may be that your testing simply doesn't cover the use
> case where netconsole will spew sleeping while atomic...

In order to test netconsole, I developed a (very simple) module to call
the printk using the sysfs interface [1]. This module creates a sysfs
file that calls printk passing the wrote message as the printk message. 

This module runs in 4 execution modes:
[n]one:     call printk
[i]rq:      call printk inside an irq disabled section
[p]reempt:  call printk inside a preempt disabled section
[x]:        call printk alternating the above modes

I ran the tests injecting printk using this four modes, using
concurrent threads, for example one thread per core injecting printk.

The point is, when the printk is called, firstly it stores the
message in the log buffer, and then it try to grab the console lock to
print the messages to the registered consoles. To grab the console
lock, printk calls the function console_trylock_for_printk. This
function, on a kernel configured with PREEMPT_RT_FULL, will check
if the system is in atomic. If the system is in atomic, it will fail,
and printk will not call the console drivers (this test is
implemented on patch printk-rt-aware.patch). The message will be
be printed to the console after, by a kworker, calling the
netconsole on the preemptive context. This is another good thing
for the RT kernel, as it can't be called inside an atomic section,
and it didn't disables the preemption and IRQs, it does not add a
possible point of latency.

[1] http://bristot.eti.br/files/netconsole/printk_inject.tar.gz

-- Daniel Bristot

Re: [RFC PATCH RT] netconsole: enable netconsole on PREEMPT_RT

[Sebastian Andrzej Siewior]

* Daniel Bristot de Oliveira | 2014-05-05 16:10:51 [-0300]:

>Figure out why it was disabled is one objective of this RFC :-).

The patch first appeared in patch-3.2-rt10.patch. Haven't seen it in
v3.0-rt1 nor in in the first v3.2-rcX-rt1. I would assume that it was
something locking specific.
I haven't seen any lockdep splat in current v3.14-RT. Therefore I'm
going to drop that hunk from the patch so it can be enabled again.

Since I don't really know why it was disabled in v3.2 I think it is
best not to backport this change unless someone figures out the real
reason and provides proof that it works in the older kernel as well.

Sebastian