DPDK-dev Archive on lore.kernel.org
 help / color / Atom feed
From: Jerin Jacob <jerinjacobk@gmail.com>
To: Bruce Richardson <bruce.richardson@intel.com>
Cc: "Jerin Jacob Kollanukkaran" <jerinj@marvell.com>,
	"dev@dpdk.org" <dev@dpdk.org>,
	"Thomas Monjalon" <thomas@monjalon.net>,
	"David Marchand" <david.marchand@redhat.com>,
	"Ferruh Yigit" <ferruh.yigit@intel.com>,
	"Andrew Rybchenko" <arybchenko@solarflare.com>,
	"Ajit Khaparde" <ajit.khaparde@broadcom.com>,
	"Qi Zhang" <qi.z.zhang@intel.com>,
	"Xiaolong Ye" <xiaolong.ye@intel.com>,
	"Raslan Darawsheh" <rasland@mellanox.com>,
	"Maxime Coquelin" <maxime.coquelin@redhat.com>,
	"Tiwei Bie" <tiwei.bie@intel.com>,
	"Akhil Goyal" <akhil.goyal@nxp.com>,
	"Luca Boccassi" <bluca@debian.org>,
	"Kevin Traynor" <ktraynor@redhat.com>,
	"maintainers@dpdk.org" <maintainers@dpdk.org>,
	"John McNamara" <john.mcnamara@intel.com>,
	"Marko Kovacevic" <marko.kovacevic@intel.com>,
	"Ray Kinsella" <mdr@ashroe.eu>,
	"Aaron Conole" <aconole@redhat.com>,
	"Michael Santana" <maicolgabriel@hotmail.com>,
	"Harry van Haaren" <harry.van.haaren@intel.com>,
	"Cristian Dumitrescu" <cristian.dumitrescu@intel.com>,
	"Phil Yang" <phil.yang@arm.com>,
	"Joyce Kong" <joyce.kong@arm.com>,
	"Mattias Rönnblom" <mattias.ronnblom@ericsson.com>,
	"Jan Viktorin" <viktorin@rehivetech.com>,
	"Gavin Hu" <gavin.hu@arm.com>,
	"David Christensen" <drc@linux.vnet.ibm.com>,
	"Konstantin Ananyev" <konstantin.ananyev@intel.com>,
	"Anatoly Burakov" <anatoly.burakov@intel.com>,
	"Harini Ramakrishnan" <harini.ramakrishnan@microsoft.com>,
	"Omar Cardona" <ocardona@microsoft.com>,
	"Anand Rawat" <anand.rawat@intel.com>,
	"Olivier Matz" <olivier.matz@6wind.com>,
	"Gage Eads" <gage.eads@intel.com>,
	"Adrien Mazarguil" <adrien.mazarguil@6wind.com>,
	"Nicolas Chautru" <nicolas.chautru@intel.com>,
	"Declan Doherty" <declan.doherty@intel.com>,
	"Fiona Trahe" <fiona.trahe@intel.com>,
	"Ashish Gupta" <ashishg@marvell.com>,
	"Erik Gabriel Carrillo" <erik.g.carrillo@intel.com>,
	"Abhinandan Gujjar" <abhinandan.gujjar@intel.com>,
	"Hemant Agrawal" <hemant.agrawal@nxp.com>,
	"Artem V. Andreev" <artem.andreev@oktetlabs.ru>,
	"Nithin Kumar Dabilpuram" <ndabilpuram@marvell.com>,
	"Vamsi Krishna Attunuru" <vattunuru@marvell.com>,
	"Rosen Xu" <rosen.xu@intel.com>,
	"Sachin Saxena" <sachin.saxena@nxp.com>,
	"Stephen Hemminger" <sthemmin@microsoft.com>,
	"Chas Williams" <chas3@att.com>,
	"John W. Linville" <linville@tuxdriver.com>,
	"Prasun Kapoor" <pkapoor@marvell.com>,
	"Marcin Wojtas" <mw@semihalf.com>,
	"Michal Krawczyk" <mk@semihalf.com>,
	"Guy Tzalik" <gtzalik@amazon.com>,
	"Evgeny Schemeilin" <evgenys@amazon.com>,
	"Igor Chauskin" <igorch@amazon.com>,
	"Ravi Kumar" <ravi1.kumar@amd.com>,
	"Igor Russkikh" <igor.russkikh@aquantia.com>,
	"Pavel Belous" <pavel.belous@aquantia.com>,
	"Shepard Siegel" <shepard.siegel@atomicrules.com>,
	"Ed Czeck" <ed.czeck@atomicrules.com>,
	"John Miller" <john.miller@atomicrules.com>,
	"Somnath Kotur" <somnath.kotur@broadcom.com>,
	"Maciej Czekaj" <mczekaj@marvell.com>,
	"Shijith Thotton" <sthotton@marvell.com>,
	"Srisivasubramanian Srinivasan" <srinivasan@marvell.com>,
	"Rahul Lakkireddy" <rahul.lakkireddy@chelsio.com>,
	"John Daley" <johndale@cisco.com>,
	"Hyong Youb Kim" <hyonkim@cisco.com>,
	"Wei Hu (Xavier" <xavier.huwei@huawei.com>,
	"Min Hu (Connor" <humin29@huawei.com>,
	"Yisen Zhuang" <yisen.zhuang@huawei.com>,
	"Ziyang Xuan" <xuanziyang2@huawei.com>,
	"Xiaoyun Wang" <cloud.wangxiaoyun@huawei.com>,
	"Guoyang Zhou" <zhouguoyang@huawei.com>,
	"Beilei Xing" <beilei.xing@intel.com>,
	"Xiao Wang" <xiao.w.wang@intel.com>,
	"Jingjing Wu" <jingjing.wu@intel.com>,
	"Wenzhuo Lu" <wenzhuo.lu@intel.com>,
	"Qiming Yang" <qiming.yang@intel.com>,
	"Tomasz Duszynski" <tdu@semihalf.com>,
	"Liron Himi" <lironh@marvell.com>, "Zyta Szpak" <zr@semihalf.com>,
	"Kiran Kumar Kokkilagadda" <kirankumark@marvell.com>,
	"Matan Azrad" <matan@mellanox.com>,
	"Shahaf Shuler" <shahafs@mellanox.com>,
	"Viacheslav Ovsiienko" <viacheslavo@mellanox.com>,
	"K. Y. Srinivasan" <kys@microsoft.com>,
	"Haiyang Zhang" <haiyangz@microsoft.com>,
	"Jan Remes" <remes@netcope.com>,
	"Heinrich Kuhn" <heinrich.kuhn@netronome.com>,
	"Jan Gutter" <jan.gutter@netronome.com>,
	"Gagandeep Singh" <g.singh@nxp.com>,
	"Rasesh Mody" <rmody@marvell.com>,
	"Shahed Shaikh" <shshaikh@marvell.com>,
	"Yong Wang" <yongwang@vmware.com>,
	"Zhihong Wang" <zhihong.wang@intel.com>,
	"Steven Webster" <steven.webster@windriver.com>,
	"Matt Peters" <matt.peters@windriver.com>,
	"Keith Wiles" <keith.wiles@intel.com>,
	"Tetsuya Mukawa" <mtetsuyah@gmail.com>,
	"Jasvinder Singh" <jasvinder.singh@intel.com>,
	"Jakub Grajciar" <jgrajcia@cisco.com>,
	"Ruifeng Wang" <ruifeng.wang@arm.com>,
	"Anoob Joseph" <anoobj@marvell.com>,
	"Fan Zhang" <roy.fan.zhang@intel.com>,
	"Pablo de Lara" <pablo.de.lara.guarch@intel.com>,
	"John Griffin" <john.griffin@intel.com>,
	"Deepak Kumar Jain" <deepak.k.jain@intel.com>,
	"Michael Shamis" <michaelsh@marvell.com>,
	"Nagadheeraj Rottela" <rnagadheeraj@marvell.com>,
	"Srikanth Jampala" <jsrikanth@marvell.com>,
	"Ankur Dwivedi" <adwivedi@marvell.com>,
	"Jay Zhou" <jianjay.zhou@huawei.com>,
	"Lee Daly" <lee.daly@intel.com>,
	"Sunila Sahu" <ssahu@marvell.com>,
	"Nipun Gupta" <nipun.gupta@nxp.com>,
	"Liang Ma" <liang.j.ma@intel.com>,
	"Peter Mccarthy" <peter.mccarthy@intel.com>,
	"Tianfei zhang" <tianfei.zhang@intel.com>,
	"Satha Koteswara Rao Kottidi" <skoteshwar@marvell.com>,
	"Xiaoyun Li" <xiaoyun.li@intel.com>,
	"Bernard Iremonger" <bernard.iremonger@intel.com>,
	"Vladimir Medvedkin" <vladimir.medvedkin@intel.com>,
	"David Hunt" <david.hunt@intel.com>,
	"Reshma Pattan" <reshma.pattan@intel.com>,
	"Byron Marohn" <byron.marohn@intel.com>,
	"Sameh Gobriel" <sameh.gobriel@intel.com>,
	"Yipeng Wang" <yipeng1.wang@intel.com>,
	"Honnappa Nagarahalli" <honnappa.nagarahalli@arm.com>,
	"Robert Sanford" <rsanford@akamai.com>,
	"Kevin Laatz" <kevin.laatz@intel.com>,
	"Maryam Tahhan" <maryam.tahhan@intel.com>,
	"Ori Kam" <orika@mellanox.com>,
	"Radu Nicolau" <radu.nicolau@intel.com>,
	"Tomasz Kantecki" <tomasz.kantecki@intel.com>,
	"Sunil Kumar Kori" <skori@marvell.com>,
	"Pavan Nikhilesh Bhagavatula" <pbhagavatula@marvell.com>,
	"Kirill Rybalchenko" <kirill.rybalchenko@intel.com>,
	"Kadam, Pallavi" <pallavi.kadam@intel.com>
Subject: Re: [dpdk-dev] [RFC] DPDK Trace support
Date: Fri, 17 Jan 2020 10:11:06 +0530
Message-ID: <CALBAE1PcVvGa3e2HSUM4iVgO6pRnhv14rSW-R0RmoDStgoLC-g@mail.gmail.com> (raw)
In-Reply-To: <20200113161259.GE1645@bricha3-MOBL.ger.corp.intel.com>

> > > >
> > > > Yes this is when trace is enabled. If the trace is disabled then it
> > > > will be the only a handful of cycles.
> > > >
> > > Two follow-on questions:
> > > 1. Is the trace enable/disable dynamic at runtime?
> >
> > Yes. See the requirement section.
> >
> > > 2. Have you investigated how low the "handful of cycles" actually is?
> >
> > Yes. it is around 1 to 3 cycles based on the arch. it boils down to
> > mostly a branch hit/miss on a memory location
> > embedded in a C macro.
> >
> That seems impressively low, which is great news!

Does anyone have an objection to have
1) Use CTF as trace format to reuse the opensource tracing tools and
compatibility wth LTTng
2) Have native DPDK CTF trace emitter for better performance for DPDK
fast path tracing and Non-Linux support.

I would like to avoid the situation where once code gets completed and
then starts our basic discussion
on the design decisions.

If someone needs more time to think through or any clarification is
required then please discuss.

See below the original RFC.

-------------------------- 8<----------------------------------

Hi All,

I would like to add tracing support for DPDK.
I am planning to add this support in v20.05 release.

This RFC attempts to get feedback from the community on

a) Tracing Use cases.
b) Tracing Requirements.
b) Implementation choices.
c) Trace format.

- Most of the cases, The DPDK provider will not have access to the
DPDK customer applications.
To debug/analyze the slow path and fast path DPDK API usage from the field,
we need to have integrated trace support in DPDK.

- Need a low overhead Fast path multi-core PMD driver debugging/analysis
infrastructure in DPDK to fix the functional and performance issue(s) of PMD.

- Post trace analysis tools can provide various status across the system such
as cpu_idle() using the timestamp added in the trace.

- Support for Linux, FreeBSD and Windows OS
- Open trace format
- Multi-platform Open source trace viewer
- Absolute low overhead trace API for DPDK fast path tracing/debugging.
- Dynamic enable/disable of trace events

To enable trace support in DPDK, following items need to work out:

a) Add the DPDK trace points in the DPDK source code.

- This includes updating DPDK functions such as,
rte_eth_dev_configure(), rte_eth_dev_start(), rte_eth_dev_rx_burst()
to emit the trace.

b) Choosing suitable serialization-format

- Common Trace Format, CTF, is an open format and language to describe
trace formats.
This enables tool reuse, of which line-textual (babeltrace) and
graphical (TraceCompass) variants already exist.

CTF should look familiar to C programmers but adds stronger typing.
See CTF - A Flexible, High-performance Binary Trace Format.


c) Writing the on-target serialization code,

See the section below.(Lttng CTF trace emitter vs DPDK specific CTF
trace emitter)

d) Deciding on and writing the I/O transport mechanics,

For performance reasons, it should be backed by a huge-page and write
to file IO.

e) Writing the PC-side deserializer/parser,

Both the babletrace(CLI tool) and Trace Compass(GUI tool) support CTF.

f) Writing tools for filtering and presentation.

See item (e)

Lttng CTF trace emitter vs DPDK specific CTF trace emitter

I have written a performance evaluation application to measure the overhead
of Lttng CTF emitter(The fastpath infrastructure used by
https://lttng.org/ library to emit the trace)


I could improve the performance by 30% by adding the "DPDK"
based plugin for get_clock() and get_cpu(),
Here are the performance numbers after adding the plugin on
x86 and various arm64 board that I have access to,

On high-end x86, it comes around 236 cycles/~100ns @ 2.4GHz (See the
last line in the log(ZERO_ARG))
On arm64, it varies from 312 cycles to 1100 cycles(based on the class of CPU).
In short, Based on the "IPC capabilities", The cost would be around
100ns to 400ns
for single void trace(a trace without any argument)

[lttng-overhead-x86] $ sudo ./calibrate/build/app/calibrate -c 0xc0
make[1]: Entering directory '/export/lttng-overhead-x86/calibrate'
make[1]: Leaving directory '/export/lttng-overhead-x86/calibrate'
EAL: Detected 56 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: Probing VFIO support...
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:01:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
CPU Timer freq is 2600.000000MHz
NOP: cycles=0.194834 ns=0.074936
GET_CLOCK: cycles=47.854658 ns=18.405638
GET_CPU: cycles=30.995892 ns=11.921497
ZERO_ARG: cycles=236.945113 ns=91.132736

We will have only 16.75ns to process 59.2 mpps(40Gbps), So IMO, Lttng
CTF emitter
may not fit the DPDK fast path purpose due to the cost associated with
generic Lttng features.

One option could be to have, native CTF emitter in EAL/DPDK to emit the
trace in a hugepage. I think it would be a handful of cycles if we
limit the features
to the requirements above:

The upside of using Lttng CTF emitter:
a) No need to write a new CTF trace emitter(the item (c))

The downside of Lttng CTF emitter(the item (c))
a) performance issue(See above)
b) Lack of Windows OS support. It looks like, it has basic FreeBSD support.
c) dpdk library dependency to lttng for trace.

So, Probably it good to have native CTF emitter in DPDK and reuse all
open-source trace viewer(babeltrace and  TraceCompass) and format(CTF)
I think, it would be best of both world.

Any thoughts on this subject? Based on the community feedback, I can
work on the patch for v20.05.

  reply index

Thread overview: 24+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-01-13 10:40 Jerin Jacob Kollanukkaran
2020-01-13 11:00 ` Ray Kinsella
2020-01-13 12:04   ` [dpdk-dev] [EXT] " Jerin Jacob Kollanukkaran
2020-01-18 15:14   ` [dpdk-dev] " dave
2020-01-20 16:51     ` Stephen Hemminger
2020-01-13 13:05 ` Bruce Richardson
2020-01-13 14:46   ` Jerin Jacob
2020-01-13 14:58     ` Bruce Richardson
2020-01-13 15:13       ` Jerin Jacob
2020-01-13 16:12         ` Bruce Richardson
2020-01-17  4:41           ` Jerin Jacob [this message]
2020-01-17  8:04             ` David Marchand
2020-01-17  9:52               ` Jerin Jacob
2020-01-17 10:30                 ` Mattias Rönnblom
2020-01-17 10:54                   ` Jerin Jacob
2020-02-15 10:21                     ` Jerin Jacob
2020-02-17  9:35                       ` Mattias Rönnblom
2020-02-17 10:23                         ` Jerin Jacob
2020-01-17 10:43                 ` David Marchand
2020-01-17 11:08                   ` Jerin Jacob
2020-01-27 16:12 ` Aaron Conole
2020-01-27 17:23   ` Jerin Jacob
2020-01-20  4:48 Jerin Jacob Kollanukkaran
2020-01-20 12:08 ` Ray Kinsella

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CALBAE1PcVvGa3e2HSUM4iVgO6pRnhv14rSW-R0RmoDStgoLC-g@mail.gmail.com \
    --to=jerinjacobk@gmail.com \
    --cc=abhinandan.gujjar@intel.com \
    --cc=aconole@redhat.com \
    --cc=adrien.mazarguil@6wind.com \
    --cc=adwivedi@marvell.com \
    --cc=ajit.khaparde@broadcom.com \
    --cc=akhil.goyal@nxp.com \
    --cc=anand.rawat@intel.com \
    --cc=anatoly.burakov@intel.com \
    --cc=anoobj@marvell.com \
    --cc=artem.andreev@oktetlabs.ru \
    --cc=arybchenko@solarflare.com \
    --cc=ashishg@marvell.com \
    --cc=beilei.xing@intel.com \
    --cc=bernard.iremonger@intel.com \
    --cc=bluca@debian.org \
    --cc=bruce.richardson@intel.com \
    --cc=byron.marohn@intel.com \
    --cc=chas3@att.com \
    --cc=cloud.wangxiaoyun@huawei.com \
    --cc=cristian.dumitrescu@intel.com \
    --cc=david.hunt@intel.com \
    --cc=david.marchand@redhat.com \
    --cc=declan.doherty@intel.com \
    --cc=deepak.k.jain@intel.com \
    --cc=dev@dpdk.org \
    --cc=drc@linux.vnet.ibm.com \
    --cc=ed.czeck@atomicrules.com \
    --cc=erik.g.carrillo@intel.com \
    --cc=evgenys@amazon.com \
    --cc=ferruh.yigit@intel.com \
    --cc=fiona.trahe@intel.com \
    --cc=g.singh@nxp.com \
    --cc=gage.eads@intel.com \
    --cc=gavin.hu@arm.com \
    --cc=gtzalik@amazon.com \
    --cc=haiyangz@microsoft.com \
    --cc=harini.ramakrishnan@microsoft.com \
    --cc=harry.van.haaren@intel.com \
    --cc=heinrich.kuhn@netronome.com \
    --cc=hemant.agrawal@nxp.com \
    --cc=honnappa.nagarahalli@arm.com \
    --cc=humin29@huawei.com \
    --cc=hyonkim@cisco.com \
    --cc=igor.russkikh@aquantia.com \
    --cc=igorch@amazon.com \
    --cc=jan.gutter@netronome.com \
    --cc=jasvinder.singh@intel.com \
    --cc=jerinj@marvell.com \
    --cc=jgrajcia@cisco.com \
    --cc=jianjay.zhou@huawei.com \
    --cc=jingjing.wu@intel.com \
    --cc=john.griffin@intel.com \
    --cc=john.mcnamara@intel.com \
    --cc=john.miller@atomicrules.com \
    --cc=johndale@cisco.com \
    --cc=joyce.kong@arm.com \
    --cc=jsrikanth@marvell.com \
    --cc=keith.wiles@intel.com \
    --cc=kevin.laatz@intel.com \
    --cc=kirankumark@marvell.com \
    --cc=kirill.rybalchenko@intel.com \
    --cc=konstantin.ananyev@intel.com \
    --cc=ktraynor@redhat.com \
    --cc=kys@microsoft.com \
    --cc=lee.daly@intel.com \
    --cc=liang.j.ma@intel.com \
    --cc=linville@tuxdriver.com \
    --cc=lironh@marvell.com \
    --cc=maicolgabriel@hotmail.com \
    --cc=maintainers@dpdk.org \
    --cc=marko.kovacevic@intel.com \
    --cc=maryam.tahhan@intel.com \
    --cc=matan@mellanox.com \
    --cc=matt.peters@windriver.com \
    --cc=mattias.ronnblom@ericsson.com \
    --cc=maxime.coquelin@redhat.com \
    --cc=mczekaj@marvell.com \
    --cc=mdr@ashroe.eu \
    --cc=michaelsh@marvell.com \
    --cc=mk@semihalf.com \
    --cc=mtetsuyah@gmail.com \
    --cc=mw@semihalf.com \
    --cc=ndabilpuram@marvell.com \
    --cc=nicolas.chautru@intel.com \
    --cc=nipun.gupta@nxp.com \
    --cc=ocardona@microsoft.com \
    --cc=olivier.matz@6wind.com \
    --cc=orika@mellanox.com \
    --cc=pablo.de.lara.guarch@intel.com \
    --cc=pallavi.kadam@intel.com \
    --cc=pavel.belous@aquantia.com \
    --cc=pbhagavatula@marvell.com \
    --cc=peter.mccarthy@intel.com \
    --cc=phil.yang@arm.com \
    --cc=pkapoor@marvell.com \
    --cc=qi.z.zhang@intel.com \
    --cc=qiming.yang@intel.com \
    --cc=radu.nicolau@intel.com \
    --cc=rahul.lakkireddy@chelsio.com \
    --cc=rasland@mellanox.com \
    --cc=ravi1.kumar@amd.com \
    --cc=remes@netcope.com \
    --cc=reshma.pattan@intel.com \
    --cc=rmody@marvell.com \
    --cc=rnagadheeraj@marvell.com \
    --cc=rosen.xu@intel.com \
    --cc=roy.fan.zhang@intel.com \
    --cc=rsanford@akamai.com \
    --cc=ruifeng.wang@arm.com \
    --cc=sachin.saxena@nxp.com \
    --cc=sameh.gobriel@intel.com \
    --cc=shahafs@mellanox.com \
    --cc=shepard.siegel@atomicrules.com \
    --cc=shshaikh@marvell.com \
    --cc=skori@marvell.com \
    --cc=skoteshwar@marvell.com \
    --cc=somnath.kotur@broadcom.com \
    --cc=srinivasan@marvell.com \
    --cc=ssahu@marvell.com \
    --cc=steven.webster@windriver.com \
    --cc=sthemmin@microsoft.com \
    --cc=sthotton@marvell.com \
    --cc=tdu@semihalf.com \
    --cc=thomas@monjalon.net \
    --cc=tianfei.zhang@intel.com \
    --cc=tiwei.bie@intel.com \
    --cc=tomasz.kantecki@intel.com \
    --cc=vattunuru@marvell.com \
    --cc=viacheslavo@mellanox.com \
    --cc=viktorin@rehivetech.com \
    --cc=vladimir.medvedkin@intel.com \
    --cc=wenzhuo.lu@intel.com \
    --cc=xavier.huwei@huawei.com \
    --cc=xiao.w.wang@intel.com \
    --cc=xiaolong.ye@intel.com \
    --cc=xiaoyun.li@intel.com \
    --cc=xuanziyang2@huawei.com \
    --cc=yipeng1.wang@intel.com \
    --cc=yisen.zhuang@huawei.com \
    --cc=yongwang@vmware.com \
    --cc=zhihong.wang@intel.com \
    --cc=zhouguoyang@huawei.com \
    --cc=zr@semihalf.com \


* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK-dev Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/dpdk-dev/0 dpdk-dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dpdk-dev dpdk-dev/ https://lore.kernel.org/dpdk-dev \
	public-inbox-index dpdk-dev

Example config snippet for mirrors

Newsgroup available over NNTP:

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git