* Re: ODEBUG: Out of memory. ODEBUG disabled
@ 2018-11-10 1:45 Qian Cai
2018-11-10 13:59 ` Waiman Long
0 siblings, 1 reply; 10+ messages in thread
From: Qian Cai @ 2018-11-10 1:45 UTC (permalink / raw)
To: Waiman Long
Cc: Joel Fernandes (Google),
Yang Shi, Zhong Jiang, Arnd Bergmann, open list, Thomas Gleixner
> Sent: Friday, November 09, 2018 at 5:08 PM
> From: "Waiman Long" <longman@redhat.com>
> To: "Qian Cai" <cai@gmx.us>, "Yang Shi" <yang.shi@linux.alibaba.com>
> Cc: "open list" <linux-kernel@vger.kernel.org>, "Thomas Gleixner" <tglx@linutronix.de>, "Arnd Bergmann" <arnd@arndb.de>, "Joel Fernandes (Google)" <joel@joelfernandes.org>, "Zhong Jiang" <zhongjiang@huawei.com>
> Subject: Re: ODEBUG: Out of memory. ODEBUG disabled
>
> On 11/09/2018 04:51 PM, Qian Cai wrote:
>>
>>> On Nov 9, 2018, at 4:42 PM, Yang Shi <yang.shi@linux.alibaba.com> wrote:
>>>
>>>
>>>
>>> On 11/9/18 1:36 PM, Qian Cai wrote:
>>>> It is a bit annoying on this aarch64 server with 64 CPUs that is
>>>> booting the latest mainline (3541833fd1f2) causes object debugging
>>>> always running out of memory.
>>> May you please paste the detail failure log?
>> I assume you mean dmesg.
>>
>> Here is the dmesg for 64 CPUs,
>> https://paste.ubuntu.com/p/BnhvXXhn7k/
>>>> I have to boot the kernel with only 16 CPUs instead (nr_cpus=16)
>>>> to make it work. Is it expected that object debugging is not going
>>>> to work with large machines?
>>> I don't think so. I'm supposed it works well with large CPU number on x86.
>> Here is the one with nr_cpus workaround,
>> https://paste.ubuntu.com/p/qMpd2CCPSV/
>
> The debugobjects code have a set of 1024 statically allocated debug
> objects that can be used in early boot before the slab memory allocator
> is initialized. Apparently, the system may have used up all the
> statically allocated objects. Try double ODEBUG_POOL_SIZE to see if it
> helps.
Great, you are right. Doubling the size makes it work. Does it make sense
to have a kconfig option instead?
>
> There are also quite a number of warnings in your console log. So there
> is certainly something wrong with your kernel or config options.
Yes, I am working on all those warnings. This one is found by ODEBUG,
https://lkml.org/lkml/2018/11/10/136
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: ODEBUG: Out of memory. ODEBUG disabled
2018-11-10 1:45 ODEBUG: Out of memory. ODEBUG disabled Qian Cai
@ 2018-11-10 13:59 ` Waiman Long
2018-11-10 14:11 ` Qian Cai
0 siblings, 1 reply; 10+ messages in thread
From: Waiman Long @ 2018-11-10 13:59 UTC (permalink / raw)
To: Qian Cai
Cc: Joel Fernandes (Google),
Yang Shi, Zhong Jiang, Arnd Bergmann, open list, Thomas Gleixner
On 11/09/2018 08:45 PM, Qian Cai wrote:
>> Sent: Friday, November 09, 2018 at 5:08 PM
>> From: "Waiman Long" <longman@redhat.com>
>> To: "Qian Cai" <cai@gmx.us>, "Yang Shi" <yang.shi@linux.alibaba.com>
>> Cc: "open list" <linux-kernel@vger.kernel.org>, "Thomas Gleixner" <tglx@linutronix.de>, "Arnd Bergmann" <arnd@arndb.de>, "Joel Fernandes (Google)" <joel@joelfernandes.org>, "Zhong Jiang" <zhongjiang@huawei.com>
>> Subject: Re: ODEBUG: Out of memory. ODEBUG disabled
>>
>> On 11/09/2018 04:51 PM, Qian Cai wrote:
>>>> On Nov 9, 2018, at 4:42 PM, Yang Shi <yang.shi@linux.alibaba.com> wrote:
>>>>
>>>>
>>>>
>>>> On 11/9/18 1:36 PM, Qian Cai wrote:
>>>>> It is a bit annoying on this aarch64 server with 64 CPUs that is
>>>>> booting the latest mainline (3541833fd1f2) causes object debugging
>>>>> always running out of memory.
>>>> May you please paste the detail failure log?
>>> I assume you mean dmesg.
>>>
>>> Here is the dmesg for 64 CPUs,
>>> https://paste.ubuntu.com/p/BnhvXXhn7k/
>>>>> I have to boot the kernel with only 16 CPUs instead (nr_cpus=16)
>>>>> to make it work. Is it expected that object debugging is not going
>>>>> to work with large machines?
>>>> I don't think so. I'm supposed it works well with large CPU number on x86.
>>> Here is the one with nr_cpus workaround,
>>> https://paste.ubuntu.com/p/qMpd2CCPSV/
>> The debugobjects code have a set of 1024 statically allocated debug
>> objects that can be used in early boot before the slab memory allocator
>> is initialized. Apparently, the system may have used up all the
>> statically allocated objects. Try double ODEBUG_POOL_SIZE to see if it
>> helps.
> Great, you are right. Doubling the size makes it work. Does it make sense
> to have a kconfig option instead?
First, I think you need to figure out what your system needed to use up
so many debug objects in early boot. If there is a legitimate reason for
this behavior, we can talk about having a kconfig option to increase that.
>> There are also quite a number of warnings in your console log. So there
>> is certainly something wrong with your kernel or config options.
> Yes, I am working on all those warnings. This one is found by ODEBUG,
> https://lkml.org/lkml/2018/11/10/136
Cheers,
Longman
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: ODEBUG: Out of memory. ODEBUG disabled
2018-11-10 13:59 ` Waiman Long
@ 2018-11-10 14:11 ` Qian Cai
2018-11-13 4:33 ` Qian Cai
0 siblings, 1 reply; 10+ messages in thread
From: Qian Cai @ 2018-11-10 14:11 UTC (permalink / raw)
To: Waiman Long
Cc: Arnd Bergmann, Yang Shi, Zhong Jiang, open list, Thomas Gleixner,
Joel Fernandes (Google)
On 11/10/18 at 8:59 AM, Waiman Long wrote:
> On 11/09/2018 08:45 PM, Qian Cai wrote:
> >> Sent: Friday, November 09, 2018 at 5:08 PM
> >> From: "Waiman Long" <longman@redhat.com>
> >> To: "Qian Cai" <cai@gmx.us>, "Yang Shi" <yang.shi@linux.alibaba.com>
> >> Cc: "open list" <linux-kernel@vger.kernel.org>, "Thomas Gleixner" <tglx@linutronix.de>, "Arnd Bergmann" <arnd@arndb.de>, "Joel Fernandes (Google)" <joel@joelfernandes.org>, "Zhong Jiang" <zhongjiang@huawei.com>
> >> Subject: Re: ODEBUG: Out of memory. ODEBUG disabled
> >>
> >> On 11/09/2018 04:51 PM, Qian Cai wrote:
> >>>> On Nov 9, 2018, at 4:42 PM, Yang Shi <yang.shi@linux.alibaba.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>> On 11/9/18 1:36 PM, Qian Cai wrote:
> >>>>> It is a bit annoying on this aarch64 server with 64 CPUs that is
> >>>>> booting the latest mainline (3541833fd1f2) causes object debugging
> >>>>> always running out of memory.
> >>>> May you please paste the detail failure log?
> >>> I assume you mean dmesg.
> >>>
> >>> Here is the dmesg for 64 CPUs,
> >>> https://paste.ubuntu.com/p/BnhvXXhn7k/
> >>>>> I have to boot the kernel with only 16 CPUs instead (nr_cpus=16)
> >>>>> to make it work. Is it expected that object debugging is not going
> >>>>> to work with large machines?
> >>>> I don't think so. I'm supposed it works well with large CPU number on x86.
> >>> Here is the one with nr_cpus workaround,
> >>> https://paste.ubuntu.com/p/qMpd2CCPSV/
> >> The debugobjects code have a set of 1024 statically allocated debug
> >> objects that can be used in early boot before the slab memory allocator
> >> is initialized. Apparently, the system may have used up all the
> >> statically allocated objects. Try double ODEBUG_POOL_SIZE to see if it
> >> helps.
> > Great, you are right. Doubling the size makes it work. Does it make sense
> > to have a kconfig option instead?
>
> First, I think you need to figure out what your system needed to use up
> so many debug objects in early boot. If there is a legitimate reason for
> this behavior, we can talk about having a kconfig option to increase that.
Anybody else not getting ODEBUG OOM with more than 64-CPU? As mentioned, restricting to 16-CPU works fine. How can I figure out why the system uses so much debug objects?
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: ODEBUG: Out of memory. ODEBUG disabled
2018-11-10 14:11 ` Qian Cai
@ 2018-11-13 4:33 ` Qian Cai
2018-11-14 0:55 ` Qian Cai
0 siblings, 1 reply; 10+ messages in thread
From: Qian Cai @ 2018-11-13 4:33 UTC (permalink / raw)
To: Waiman Long
Cc: Arnd Bergmann, Yang Shi, Zhong Jiang, linux kernel,
Thomas Gleixner, Joel Fernandes (Google)
> On Nov 10, 2018, at 9:11 AM, Qian Cai <cai@gmx.us> wrote:
>
> On 11/10/18 at 8:59 AM, Waiman Long wrote:
>
>> On 11/09/2018 08:45 PM, Qian Cai wrote:
>>>> Sent: Friday, November 09, 2018 at 5:08 PM
>>>> From: "Waiman Long" <longman@redhat.com>
>>>> To: "Qian Cai" <cai@gmx.us>, "Yang Shi" <yang.shi@linux.alibaba.com>
>>>> Cc: "open list" <linux-kernel@vger.kernel.org>, "Thomas Gleixner" <tglx@linutronix.de>, "Arnd Bergmann" <arnd@arndb.de>, "Joel Fernandes (Google)" <joel@joelfernandes.org>, "Zhong Jiang" <zhongjiang@huawei.com>
>>>> Subject: Re: ODEBUG: Out of memory. ODEBUG disabled
>>>>
>>>> On 11/09/2018 04:51 PM, Qian Cai wrote:
>>>>>> On Nov 9, 2018, at 4:42 PM, Yang Shi <yang.shi@linux.alibaba.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 11/9/18 1:36 PM, Qian Cai wrote:
>>>>>>> It is a bit annoying on this aarch64 server with 64 CPUs that is
>>>>>>> booting the latest mainline (3541833fd1f2) causes object debugging
>>>>>>> always running out of memory.
>>>>>> May you please paste the detail failure log?
>>>>> I assume you mean dmesg.
>>>>>
>>>>> Here is the dmesg for 64 CPUs,
>>>>> https://paste.ubuntu.com/p/BnhvXXhn7k/
>>>>>>> I have to boot the kernel with only 16 CPUs instead (nr_cpus=16)
>>>>>>> to make it work. Is it expected that object debugging is not going
>>>>>>> to work with large machines?
>>>>>> I don't think so. I'm supposed it works well with large CPU number on x86.
>>>>> Here is the one with nr_cpus workaround,
>>>>> https://paste.ubuntu.com/p/qMpd2CCPSV/
>>>> The debugobjects code have a set of 1024 statically allocated debug
>>>> objects that can be used in early boot before the slab memory allocator
>>>> is initialized. Apparently, the system may have used up all the
>>>> statically allocated objects. Try double ODEBUG_POOL_SIZE to see if it
>>>> helps.
>>> Great, you are right. Doubling the size makes it work. Does it make sense
>>> to have a kconfig option instead?
>>
>> First, I think you need to figure out what your system needed to use up
>> so many debug objects in early boot. If there is a legitimate reason for
>> this behavior, we can talk about having a kconfig option to increase that.
> Anybody else not getting ODEBUG OOM with more than 64-CPU? As
> mentioned, restricting to 16-CPU works fine. How can I figure out why the
> system uses so much debug objects?
On another aarch64 server with 256-CPU, even double the size of
ODEBUG_POOL_SIZE, i.e., 2048 will get "ODEBUG: Out of memory. ODEBUG
disabled”.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: ODEBUG: Out of memory. ODEBUG disabled
2018-11-13 4:33 ` Qian Cai
@ 2018-11-14 0:55 ` Qian Cai
0 siblings, 0 replies; 10+ messages in thread
From: Qian Cai @ 2018-11-14 0:55 UTC (permalink / raw)
To: linux kernel
Cc: Jan Stancek, Andrew Morton, Du Changbin, Christian Borntraeger
> On Nov 12, 2018, at 11:33 PM, Qian Cai <cai@gmx.us> wrote:
>
>
>
>> On Nov 10, 2018, at 9:11 AM, Qian Cai <cai@gmx.us> wrote:
>>
>> On 11/10/18 at 8:59 AM, Waiman Long wrote:
>>
>>> On 11/09/2018 08:45 PM, Qian Cai wrote:
>>>>> Sent: Friday, November 09, 2018 at 5:08 PM
>>>>> From: "Waiman Long" <longman@redhat.com>
>>>>> To: "Qian Cai" <cai@gmx.us>, "Yang Shi" <yang.shi@linux.alibaba.com>
>>>>> Cc: "open list" <linux-kernel@vger.kernel.org>, "Thomas Gleixner" <tglx@linutronix.de>, "Arnd Bergmann" <arnd@arndb.de>, "Joel Fernandes (Google)" <joel@joelfernandes.org>, "Zhong Jiang" <zhongjiang@huawei.com>
>>>>> Subject: Re: ODEBUG: Out of memory. ODEBUG disabled
>>>>>
>>>>> On 11/09/2018 04:51 PM, Qian Cai wrote:
>>>>>>> On Nov 9, 2018, at 4:42 PM, Yang Shi <yang.shi@linux.alibaba.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 11/9/18 1:36 PM, Qian Cai wrote:
>>>>>>>> It is a bit annoying on this aarch64 server with 64 CPUs that is
>>>>>>>> booting the latest mainline (3541833fd1f2) causes object debugging
>>>>>>>> always running out of memory.
>>>>>>> May you please paste the detail failure log?
>>>>>> I assume you mean dmesg.
>>>>>>
>>>>>> Here is the dmesg for 64 CPUs,
>>>>>> https://paste.ubuntu.com/p/BnhvXXhn7k/
>>>>>>>> I have to boot the kernel with only 16 CPUs instead (nr_cpus=16)
>>>>>>>> to make it work. Is it expected that object debugging is not going
>>>>>>>> to work with large machines?
>>>>>>> I don't think so. I'm supposed it works well with large CPU number on x86.
>>>>>> Here is the one with nr_cpus workaround,
>>>>>> https://paste.ubuntu.com/p/qMpd2CCPSV/
>>>>> The debugobjects code have a set of 1024 statically allocated debug
>>>>> objects that can be used in early boot before the slab memory allocator
>>>>> is initialized. Apparently, the system may have used up all the
>>>>> statically allocated objects. Try double ODEBUG_POOL_SIZE to see if it
>>>>> helps.
>>>> Great, you are right. Doubling the size makes it work. Does it make sense
>>>> to have a kconfig option instead?
>>>
>>> First, I think you need to figure out what your system needed to use up
>>> so many debug objects in early boot. If there is a legitimate reason for
>>> this behavior, we can talk about having a kconfig option to increase that.
>> Anybody else not getting ODEBUG OOM with more than 64-CPU? As
>> mentioned, restricting to 16-CPU works fine. How can I figure out why the
>> system uses so much debug objects?
> On another aarch64 server with 256-CPU, even double the size of
> ODEBUG_POOL_SIZE, i.e., 2048 will get "ODEBUG: Out of memory. ODEBUG
> disabled”.
OK, here is the problem.
In order to get aarch64 work, the initial ODEBUG_POOL_SIZE on
64-CPU: need 2048
256-CPU: need 8192 (4096 too small)
This commit 97dd552eb23c
+ * Increase the thresholds for allocating and freeing objects
+ * according to the number of possible CPUs available in the system.
+ */
+ debug_objects_pool_size += num_possible_cpus() * 32;
Why magic number 32?
It needs to be bigger than that for aarch64.
(2048 + 64 x 32 - 1024) / 64 = 48 (work on 64-cpu)
(4096 + 256 x 32 - 1024) / 256 = 48 (not work on 256-cpu)
(8196 + 256 x 32 - 1024) / 256 = 60 (work on 256-cpu)
^ permalink raw reply [flat|nested] 10+ messages in thread
* ODEBUG: Out of memory. ODEBUG disabled
@ 2018-11-09 21:36 Qian Cai
2018-11-09 21:42 ` Yang Shi
0 siblings, 1 reply; 10+ messages in thread
From: Qian Cai @ 2018-11-09 21:36 UTC (permalink / raw)
To: open list
Cc: Thomas Gleixner, Yang Shi, Waiman Long, Arnd Bergmann,
Joel Fernandes (Google),
Zhong Jiang
It is a bit annoying on this aarch64 server with 64 CPUs that is
booting the latest mainline (3541833fd1f2) causes object debugging
always running out of memory.
I have to boot the kernel with only 16 CPUs instead (nr_cpus=16)
to make it work. Is it expected that object debugging is not going
to work with large machines?
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: ODEBUG: Out of memory. ODEBUG disabled
2018-11-09 21:36 Qian Cai
@ 2018-11-09 21:42 ` Yang Shi
2018-11-09 21:51 ` Qian Cai
0 siblings, 1 reply; 10+ messages in thread
From: Yang Shi @ 2018-11-09 21:42 UTC (permalink / raw)
To: Qian Cai, open list
Cc: Thomas Gleixner, Waiman Long, Arnd Bergmann,
Joel Fernandes (Google),
Zhong Jiang
On 11/9/18 1:36 PM, Qian Cai wrote:
> It is a bit annoying on this aarch64 server with 64 CPUs that is
> booting the latest mainline (3541833fd1f2) causes object debugging
> always running out of memory.
May you please paste the detail failure log?
>
> I have to boot the kernel with only 16 CPUs instead (nr_cpus=16)
> to make it work. Is it expected that object debugging is not going
> to work with large machines?
I don't think so. I'm supposed it works well with large CPU number on x86.
Thanks,
Yang
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: ODEBUG: Out of memory. ODEBUG disabled
2018-11-09 21:42 ` Yang Shi
@ 2018-11-09 21:51 ` Qian Cai
2018-11-09 22:01 ` Yang Shi
2018-11-09 22:08 ` Waiman Long
0 siblings, 2 replies; 10+ messages in thread
From: Qian Cai @ 2018-11-09 21:51 UTC (permalink / raw)
To: Yang Shi
Cc: open list, Thomas Gleixner, Waiman Long, Arnd Bergmann,
Joel Fernandes (Google),
Zhong Jiang
> On Nov 9, 2018, at 4:42 PM, Yang Shi <yang.shi@linux.alibaba.com> wrote:
>
>
>
> On 11/9/18 1:36 PM, Qian Cai wrote:
>> It is a bit annoying on this aarch64 server with 64 CPUs that is
>> booting the latest mainline (3541833fd1f2) causes object debugging
>> always running out of memory.
>
> May you please paste the detail failure log?
I assume you mean dmesg.
Here is the dmesg for 64 CPUs,
https://paste.ubuntu.com/p/BnhvXXhn7k/
>>
>> I have to boot the kernel with only 16 CPUs instead (nr_cpus=16)
>> to make it work. Is it expected that object debugging is not going
>> to work with large machines?
>
> I don't think so. I'm supposed it works well with large CPU number on x86.
Here is the one with nr_cpus workaround,
https://paste.ubuntu.com/p/qMpd2CCPSV/
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: ODEBUG: Out of memory. ODEBUG disabled
2018-11-09 21:51 ` Qian Cai
@ 2018-11-09 22:01 ` Yang Shi
2018-11-09 22:08 ` Waiman Long
1 sibling, 0 replies; 10+ messages in thread
From: Yang Shi @ 2018-11-09 22:01 UTC (permalink / raw)
To: Qian Cai
Cc: open list, Thomas Gleixner, Waiman Long, Arnd Bergmann,
Joel Fernandes (Google),
Zhong Jiang
On 11/9/18 1:51 PM, Qian Cai wrote:
>
>> On Nov 9, 2018, at 4:42 PM, Yang Shi <yang.shi@linux.alibaba.com> wrote:
>>
>>
>>
>> On 11/9/18 1:36 PM, Qian Cai wrote:
>>> It is a bit annoying on this aarch64 server with 64 CPUs that is
>>> booting the latest mainline (3541833fd1f2) causes object debugging
>>> always running out of memory.
>> May you please paste the detail failure log?
> I assume you mean dmesg.
>
> Here is the dmesg for 64 CPUs,
> https://paste.ubuntu.com/p/BnhvXXhn7k/
I don't see the oom message, it looks it boots up successfully with 64 CPUs.
>>> I have to boot the kernel with only 16 CPUs instead (nr_cpus=16)
>>> to make it work. Is it expected that object debugging is not going
>>> to work with large machines?
>> I don't think so. I'm supposed it works well with large CPU number on x86.
> Here is the one with nr_cpus workaround,
> https://paste.ubuntu.com/p/qMpd2CCPSV/
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: ODEBUG: Out of memory. ODEBUG disabled
2018-11-09 21:51 ` Qian Cai
2018-11-09 22:01 ` Yang Shi
@ 2018-11-09 22:08 ` Waiman Long
1 sibling, 0 replies; 10+ messages in thread
From: Waiman Long @ 2018-11-09 22:08 UTC (permalink / raw)
To: Qian Cai, Yang Shi
Cc: open list, Thomas Gleixner, Arnd Bergmann,
Joel Fernandes (Google),
Zhong Jiang
On 11/09/2018 04:51 PM, Qian Cai wrote:
>
>> On Nov 9, 2018, at 4:42 PM, Yang Shi <yang.shi@linux.alibaba.com> wrote:
>>
>>
>>
>> On 11/9/18 1:36 PM, Qian Cai wrote:
>>> It is a bit annoying on this aarch64 server with 64 CPUs that is
>>> booting the latest mainline (3541833fd1f2) causes object debugging
>>> always running out of memory.
>> May you please paste the detail failure log?
> I assume you mean dmesg.
>
> Here is the dmesg for 64 CPUs,
> https://paste.ubuntu.com/p/BnhvXXhn7k/
>>> I have to boot the kernel with only 16 CPUs instead (nr_cpus=16)
>>> to make it work. Is it expected that object debugging is not going
>>> to work with large machines?
>> I don't think so. I'm supposed it works well with large CPU number on x86.
> Here is the one with nr_cpus workaround,
> https://paste.ubuntu.com/p/qMpd2CCPSV/
The debugobjects code have a set of 1024 statically allocated debug
objects that can be used in early boot before the slab memory allocator
is initialized. Apparently, the system may have used up all the
statically allocated objects. Try double ODEBUG_POOL_SIZE to see if it
helps.
There are also quite a number of warnings in your console log. So there
is certainly something wrong with your kernel or config options.
Cheers,
Longman
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2018-11-14 0:56 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-11-10 1:45 ODEBUG: Out of memory. ODEBUG disabled Qian Cai
2018-11-10 13:59 ` Waiman Long
2018-11-10 14:11 ` Qian Cai
2018-11-13 4:33 ` Qian Cai
2018-11-14 0:55 ` Qian Cai
-- strict thread matches above, loose matches on Subject: below --
2018-11-09 21:36 Qian Cai
2018-11-09 21:42 ` Yang Shi
2018-11-09 21:51 ` Qian Cai
2018-11-09 22:01 ` Yang Shi
2018-11-09 22:08 ` Waiman Long
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).