* [RFC PATCH RT] netconsole: enable netconsole on PREEMPT_RT @ 2014-04-29 17:55 Daniel Bristot de Oliveira 2014-05-01 14:15 ` Paul Gortmaker 0 siblings, 1 reply; 4+ messages in thread From: Daniel Bristot de Oliveira @ 2014-04-29 17:55 UTC (permalink / raw) To: linux-rt-users; +Cc: linux-kernel Dear RT Folks, This patch enables the netconsole on PREEMPT_RT_FULL, netconsole was disabled on commit: cb7cb77 kconfig-disable-a-few-options-rt.patch I tested the netconsole on three machines: - Intel Core i5 (4 cpus) - Local network - Intel Xeon (32 cpus) - Remote network - KVM VM (on a Core i7) - Virtual network On these machines I build the kernel 3.10.34-rt34 + netconsole built as module. In order to execute some printks, I run a small benchmark that runs as a module. To analyse the output I captured the netconsole messages using Wireshark. On all cases, the tests passed without any error: - No error messages from the kernel - No missed messages - No out of order messages I ran +1 hour tests on each machine. The test using the Xeon machine ran by around 10 hours. On this test I received 317149 netconsole's messages. I also read the code and ran some traces (using the ftrace's function_graph). In the traces that I read, the netconsole code never broke the rt lock assumptions. Signed-off-by: Daniel Bristot de Oliveira <bristot@redhat.com> --- drivers/net/Kconfig | 1 - 1 file changed, 1 deletion(-) diff --git a/drivers/net/Kconfig b/drivers/net/Kconfig index 096de88..3835321 100644 --- a/drivers/net/Kconfig +++ b/drivers/net/Kconfig @@ -164,7 +164,6 @@ config VXLAN config NETCONSOLE tristate "Network console logging support" - depends on !PREEMPT_RT_FULL ---help--- If you want to log kernel messages over the network, enable this. See <file:Documentation/networking/netconsole.txt> for details. -- 1.9.0 ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [RFC PATCH RT] netconsole: enable netconsole on PREEMPT_RT 2014-04-29 17:55 [RFC PATCH RT] netconsole: enable netconsole on PREEMPT_RT Daniel Bristot de Oliveira @ 2014-05-01 14:15 ` Paul Gortmaker 2014-05-05 19:10 ` Daniel Bristot de Oliveira 0 siblings, 1 reply; 4+ messages in thread From: Paul Gortmaker @ 2014-05-01 14:15 UTC (permalink / raw) To: Daniel Bristot de Oliveira, linux-rt-users; +Cc: linux-kernel On 14-04-29 01:55 PM, Daniel Bristot de Oliveira wrote: > Dear RT Folks, > > This patch enables the netconsole on PREEMPT_RT_FULL, netconsole > was disabled on commit: > cb7cb77 kconfig-disable-a-few-options-rt.patch > > I tested the netconsole on three machines: > > - Intel Core i5 (4 cpus) - Local network > - Intel Xeon (32 cpus) - Remote network > - KVM VM (on a Core i7) - Virtual network Chances are the two real targets above were using the same igb network driver. I wonder if it matters what underlying nic is used for netconsole? It is probably worthwhile to do some data mining into why it was disabled in the 1st place. I can't say I recall, and it is possible that the reason has long since become irrelevant, even though the patch has been carried forward over time. Or it may be that your testing simply doesn't cover the use case where netconsole will spew sleeping while atomic... P. -- > > On these machines I build the kernel 3.10.34-rt34 + netconsole > built as module. In order to execute some printks, I run a small > benchmark that runs as a module. To analyse the output I captured > the netconsole messages using Wireshark. > > On all cases, the tests passed without any error: > > - No error messages from the kernel > - No missed messages > - No out of order messages > > I ran +1 hour tests on each machine. The test using the Xeon machine > ran by around 10 hours. On this test I received 317149 netconsole's > messages. > > I also read the code and ran some traces (using the ftrace's > function_graph). In the traces that I read, the netconsole code never > broke the rt lock assumptions. > > Signed-off-by: Daniel Bristot de Oliveira <bristot@redhat.com> > --- > drivers/net/Kconfig | 1 - > 1 file changed, 1 deletion(-) > > diff --git a/drivers/net/Kconfig b/drivers/net/Kconfig > index 096de88..3835321 100644 > --- a/drivers/net/Kconfig > +++ b/drivers/net/Kconfig > @@ -164,7 +164,6 @@ config VXLAN > > config NETCONSOLE > tristate "Network console logging support" > - depends on !PREEMPT_RT_FULL > ---help--- > If you want to log kernel messages over the network, enable this. > See <file:Documentation/networking/netconsole.txt> for details. > ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [RFC PATCH RT] netconsole: enable netconsole on PREEMPT_RT 2014-05-01 14:15 ` Paul Gortmaker @ 2014-05-05 19:10 ` Daniel Bristot de Oliveira 2014-05-09 15:55 ` Sebastian Andrzej Siewior 0 siblings, 1 reply; 4+ messages in thread From: Daniel Bristot de Oliveira @ 2014-05-05 19:10 UTC (permalink / raw) To: Paul Gortmaker, linux-rt-users; +Cc: linux-kernel On 05/01/2014 11:15 AM, Paul Gortmaker wrote: > Chances are the two real targets above were using the > same igb network driver. I wonder if it matters what > underlying nic is used for netconsole? These three machines are using different drivers: +----------+------------+ | Machine | Nic Module | +----------+------------+ | Xeon | tg3 | | core i5 | r8169 | | KVM VM | virtio_net | +----------+------------+ Netconsole uses the function netpoll_send_udp to send a packet, and netpool_send_udp calls the driver's ".ndo_start_xmit" function. So it end up calling the NIC driver, but it's not "hardware dependent". > It is probably worthwhile to do some data mining into why > it was disabled in the 1st place. I can't say I recall, and > it is possible that the reason has long since become irrelevant, > even though the patch has been carried forward over time. Figure out why it was disabled is one objective of this RFC :-). I did not found any message about some error of netconsole on RT kernel. I enabled the netconsole waiting for some obscure bug, but surprisingly it always worked fine. > Or it may be that your testing simply doesn't cover the use > case where netconsole will spew sleeping while atomic... In order to test netconsole, I developed a (very simple) module to call the printk using the sysfs interface [1]. This module creates a sysfs file that calls printk passing the wrote message as the printk message. This module runs in 4 execution modes: [n]one: call printk [i]rq: call printk inside an irq disabled section [p]reempt: call printk inside a preempt disabled section [x]: call printk alternating the above modes I ran the tests injecting printk using this four modes, using concurrent threads, for example one thread per core injecting printk. The point is, when the printk is called, firstly it stores the message in the log buffer, and then it try to grab the console lock to print the messages to the registered consoles. To grab the console lock, printk calls the function console_trylock_for_printk. This function, on a kernel configured with PREEMPT_RT_FULL, will check if the system is in atomic. If the system is in atomic, it will fail, and printk will not call the console drivers (this test is implemented on patch printk-rt-aware.patch). The message will be be printed to the console after, by a kworker, calling the netconsole on the preemptive context. This is another good thing for the RT kernel, as it can't be called inside an atomic section, and it didn't disables the preemption and IRQs, it does not add a possible point of latency. [1] http://bristot.eti.br/files/netconsole/printk_inject.tar.gz -- Daniel Bristot ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [RFC PATCH RT] netconsole: enable netconsole on PREEMPT_RT 2014-05-05 19:10 ` Daniel Bristot de Oliveira @ 2014-05-09 15:55 ` Sebastian Andrzej Siewior 0 siblings, 0 replies; 4+ messages in thread From: Sebastian Andrzej Siewior @ 2014-05-09 15:55 UTC (permalink / raw) To: Daniel Bristot de Oliveira Cc: Paul Gortmaker, linux-rt-users, linux-kernel, rostedt * Daniel Bristot de Oliveira | 2014-05-05 16:10:51 [-0300]: >Figure out why it was disabled is one objective of this RFC :-). The patch first appeared in patch-3.2-rt10.patch. Haven't seen it in v3.0-rt1 nor in in the first v3.2-rcX-rt1. I would assume that it was something locking specific. I haven't seen any lockdep splat in current v3.14-RT. Therefore I'm going to drop that hunk from the patch so it can be enabled again. Since I don't really know why it was disabled in v3.2 I think it is best not to backport this change unless someone figures out the real reason and provides proof that it works in the older kernel as well. Sebastian ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2014-05-09 15:55 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-04-29 17:55 [RFC PATCH RT] netconsole: enable netconsole on PREEMPT_RT Daniel Bristot de Oliveira 2014-05-01 14:15 ` Paul Gortmaker 2014-05-05 19:10 ` Daniel Bristot de Oliveira 2014-05-09 15:55 ` Sebastian Andrzej Siewior
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.