* Re: [SPDK] NVMe command ordering
@ 2018-03-23 16:21 Jonas Pfefferle
0 siblings, 0 replies; 8+ messages in thread
From: Jonas Pfefferle @ 2018-03-23 16:21 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 2113 bytes --]
One other thing I stumbled upon:
7.1 "There are no ordering restrictions for completions to the host."
Does that mean if, for example, command A and B were executed in order A,B
they do not have to complete in this order on the CQ?
Thanks,
Jonas
On Fri, 23 Mar 2018 12:59:54 +0100
"Jonas Pfefferle" <pepperjo(a)japf.ch> wrote:
> Thanks!
>
> On Fri, 23 Mar 2018 11:25:26 +0000
> Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com> wrote:
>> On Fri, Mar 23, 2018, 12:04 Jonas Pfefferle <pepperjo(a)japf.ch>
>>wrote:
>>
>>> Hi @all,
>>>
>>> Can anyone clarify this for me:
>>> NVMe spec 6.3 Command Ordering Requirements
>>> "For all commands which are not part of a fused operation (refer to
>>>section
>>> 4.10), or for which the write size is greater than AWUN, each
>>>command is
>>> processed as an independent entity without reference to other
>>>commands
>>> submitted to the same I/O Submission Queue or to commands submitted
>>>to
>>> other
>>> I/O Submission Queues"
>>>
>>> So writes <= AWUN size are atomic but there is no ordering between
>>>the
>>> commands, e.g. write command A issued before write command B to the
>>>same
>>> LBA
>>> can be executed in order AB or BA, the only thing guaranteed is that
>>>they
>>> are executed atomically? And there is no way to enforce ordering
>>>except on
>>> the host side?
>>>
>>
>> Right, that's exactly what the spec says in that same paragraph you
>>quote
>> above: "If there are ordering requirements between these commands,
>>host
>> software or the associated application is required to enforce that
>>ordering
>> above the level of the controller. " (1.3a, P. 179).
>>
>> Regards,
>> Andrey
>>
>>
>>> Regards,
>>> Jonas
>>>
>>>
>>> _______________________________________________
>>> SPDK mailing list
>>> SPDK(a)lists.01.org
>>> https://lists.01.org/mailman/listinfo/spdk
>>>
>> --
>>
>> Regards,
>> Andrey
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [SPDK] NVMe command ordering
@ 2018-03-27 7:56 Jonas Pfefferle
0 siblings, 0 replies; 8+ messages in thread
From: Jonas Pfefferle @ 2018-03-27 7:56 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 4485 bytes --]
Thanks for the notes/clarifications. Interesting paper.
Jonas
On Fri, 23 Mar 2018 17:08:47 -0700
Paul Von-Stamwitz <PVonStamwitz(a)us.fujitsu.com> wrote:
> This is purely FYI:
>
> Interesting paper from FAST '18.
>
> Barrier-Enabled IO Stack for Flash Storage
> https://www.usenix.org/conference/fast18/presentation/won
>
> -Paul
>
> -----Original Message-----
>From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Marushak,
>Nathan
> Sent: Friday, March 23, 2018 9:29 AM
> To: Storage Performance Development Kit
> Subject: Re: [SPDK] NVMe command ordering
>
> Hey Jonas,
>
> Looks like you got the answer here hopefully, but I wanted to
>provide 1 additional point of clarity. For HDD or SSD (or any other
>type of storage media), it's always been incumbent upon the
>application (most notably the file system) to provide the checks and
>balances to ensure ordering.
>
> Thanks,
> Nate
>
>> -----Original Message-----
>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse,
>>Paul
>> E
>> Sent: Friday, March 23, 2018 9:23 AM
>> To: Storage Performance Development Kit <spdk(a)lists.01.org>; Andrey
>> Kuzmin <andrey.v.kuzmin(a)gmail.com>
>> Subject: Re: [SPDK] NVMe command ordering
>>
>> Yup
>>
>> -----Original Message-----
>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Jonas
>> Pfefferle
>> Sent: Friday, March 23, 2018 9:21 AM
>> To: Storage Performance Development Kit <spdk(a)lists.01.org>; Andrey
>> Kuzmin <andrey.v.kuzmin(a)gmail.com>
>> Subject: Re: [SPDK] NVMe command ordering
>>
>> One other thing I stumbled upon:
>> 7.1 "There are no ordering restrictions for completions to the
>>host."
>>
>> Does that mean if, for example, command A and B were executed in
>>order
>> A,B they do not have to complete in this order on the CQ?
>>
>> Thanks,
>> Jonas
>>
>> On Fri, 23 Mar 2018 12:59:54 +0100
>> "Jonas Pfefferle" <pepperjo(a)japf.ch> wrote:
>> > Thanks!
>> >
>> > On Fri, 23 Mar 2018 11:25:26 +0000
>> > Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com> wrote:
>> >> On Fri, Mar 23, 2018, 12:04 Jonas Pfefferle <pepperjo(a)japf.ch>
>> >>wrote:
>> >>
>> >>> Hi @all,
>> >>>
>> >>> Can anyone clarify this for me:
>> >>> NVMe spec 6.3 Command Ordering Requirements "For all commands
>> >>>which are not part of a fused operation (refer to section 4.10),
>> >>>or for which the write size is greater than AWUN, each command is
>>
>> >>>processed as an independent entity without reference to other
>> >>>commands submitted to the same I/O Submission Queue or to
>>commands
>> >>>submitted to other I/O Submission Queues"
>> >>>
>> >>> So writes <= AWUN size are atomic but there is no ordering
>>between
>> >>>the commands, e.g. write command A issued before write command B
>> >>>to the same LBA can be executed in order AB or BA, the only
>>thing
>> >>>guaranteed is that they are executed atomically? And there is no
>> >>>way to enforce ordering except on the host side?
>> >>>
>> >>
>> >> Right, that's exactly what the spec says in that same paragraph
>>you
>> >>quote
>> >> above: "If there are ordering requirements between these
>>commands,
>> >>host software or the associated application is required to
>>enforce
>> >>that ordering above the level of the controller. " (1.3a, P.
>>179).
>> >>
>> >> Regards,
>> >> Andrey
>> >>
>> >>
>> >>> Regards,
>> >>> Jonas
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> SPDK mailing list
>> >>> SPDK(a)lists.01.org
>> >>> https://lists.01.org/mailman/listinfo/spdk
>> >>>
>> >> --
>> >>
>> >> Regards,
>> >> Andrey
>> >
>> >
>> > _______________________________________________
>> > SPDK mailing list
>> > SPDK(a)lists.01.org
>> > https://lists.01.org/mailman/listinfo/spdk
>>
>>
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [SPDK] NVMe command ordering
@ 2018-03-24 0:08 Paul Von-Stamwitz
0 siblings, 0 replies; 8+ messages in thread
From: Paul Von-Stamwitz @ 2018-03-24 0:08 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 3972 bytes --]
This is purely FYI:
Interesting paper from FAST '18.
Barrier-Enabled IO Stack for Flash Storage
https://www.usenix.org/conference/fast18/presentation/won
-Paul
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Marushak, Nathan
Sent: Friday, March 23, 2018 9:29 AM
To: Storage Performance Development Kit
Subject: Re: [SPDK] NVMe command ordering
Hey Jonas,
Looks like you got the answer here hopefully, but I wanted to provide 1 additional point of clarity. For HDD or SSD (or any other type of storage media), it's always been incumbent upon the application (most notably the file system) to provide the checks and balances to ensure ordering.
Thanks,
Nate
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul
> E
> Sent: Friday, March 23, 2018 9:23 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>; Andrey
> Kuzmin <andrey.v.kuzmin(a)gmail.com>
> Subject: Re: [SPDK] NVMe command ordering
>
> Yup
>
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Jonas
> Pfefferle
> Sent: Friday, March 23, 2018 9:21 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>; Andrey
> Kuzmin <andrey.v.kuzmin(a)gmail.com>
> Subject: Re: [SPDK] NVMe command ordering
>
> One other thing I stumbled upon:
> 7.1 "There are no ordering restrictions for completions to the host."
>
> Does that mean if, for example, command A and B were executed in order
> A,B they do not have to complete in this order on the CQ?
>
> Thanks,
> Jonas
>
> On Fri, 23 Mar 2018 12:59:54 +0100
> "Jonas Pfefferle" <pepperjo(a)japf.ch> wrote:
> > Thanks!
> >
> > On Fri, 23 Mar 2018 11:25:26 +0000
> > Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com> wrote:
> >> On Fri, Mar 23, 2018, 12:04 Jonas Pfefferle <pepperjo(a)japf.ch>
> >>wrote:
> >>
> >>> Hi @all,
> >>>
> >>> Can anyone clarify this for me:
> >>> NVMe spec 6.3 Command Ordering Requirements "For all commands
> >>>which are not part of a fused operation (refer to section 4.10),
> >>>or for which the write size is greater than AWUN, each command is
> >>>processed as an independent entity without reference to other
> >>>commands submitted to the same I/O Submission Queue or to commands
> >>>submitted to other I/O Submission Queues"
> >>>
> >>> So writes <= AWUN size are atomic but there is no ordering between
> >>>the commands, e.g. write command A issued before write command B
> >>>to the same LBA can be executed in order AB or BA, the only thing
> >>>guaranteed is that they are executed atomically? And there is no
> >>>way to enforce ordering except on the host side?
> >>>
> >>
> >> Right, that's exactly what the spec says in that same paragraph you
> >>quote
> >> above: "If there are ordering requirements between these commands,
> >>host software or the associated application is required to enforce
> >>that ordering above the level of the controller. " (1.3a, P. 179).
> >>
> >> Regards,
> >> Andrey
> >>
> >>
> >>> Regards,
> >>> Jonas
> >>>
> >>>
> >>> _______________________________________________
> >>> SPDK mailing list
> >>> SPDK(a)lists.01.org
> >>> https://lists.01.org/mailman/listinfo/spdk
> >>>
> >> --
> >>
> >> Regards,
> >> Andrey
> >
> >
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [SPDK] NVMe command ordering
@ 2018-03-23 16:29 Marushak, Nathan
0 siblings, 0 replies; 8+ messages in thread
From: Marushak, Nathan @ 2018-03-23 16:29 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 3411 bytes --]
Hey Jonas,
Looks like you got the answer here hopefully, but I wanted to provide 1 additional point of clarity. For HDD or SSD (or any other type of storage media), it's always been incumbent upon the application (most notably the file system) to provide the checks and balances to ensure ordering.
Thanks,
Nate
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
> Sent: Friday, March 23, 2018 9:23 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>; Andrey Kuzmin
> <andrey.v.kuzmin(a)gmail.com>
> Subject: Re: [SPDK] NVMe command ordering
>
> Yup
>
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Jonas Pfefferle
> Sent: Friday, March 23, 2018 9:21 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>; Andrey Kuzmin
> <andrey.v.kuzmin(a)gmail.com>
> Subject: Re: [SPDK] NVMe command ordering
>
> One other thing I stumbled upon:
> 7.1 "There are no ordering restrictions for completions to the host."
>
> Does that mean if, for example, command A and B were executed in order A,B
> they do not have to complete in this order on the CQ?
>
> Thanks,
> Jonas
>
> On Fri, 23 Mar 2018 12:59:54 +0100
> "Jonas Pfefferle" <pepperjo(a)japf.ch> wrote:
> > Thanks!
> >
> > On Fri, 23 Mar 2018 11:25:26 +0000
> > Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com> wrote:
> >> On Fri, Mar 23, 2018, 12:04 Jonas Pfefferle <pepperjo(a)japf.ch>
> >>wrote:
> >>
> >>> Hi @all,
> >>>
> >>> Can anyone clarify this for me:
> >>> NVMe spec 6.3 Command Ordering Requirements "For all commands which
> >>>are not part of a fused operation (refer to section 4.10), or for
> >>>which the write size is greater than AWUN, each command is processed
> >>>as an independent entity without reference to other commands
> >>>submitted to the same I/O Submission Queue or to commands submitted
> >>>to other I/O Submission Queues"
> >>>
> >>> So writes <= AWUN size are atomic but there is no ordering between
> >>>the commands, e.g. write command A issued before write command B to
> >>>the same LBA can be executed in order AB or BA, the only thing
> >>>guaranteed is that they are executed atomically? And there is no way
> >>>to enforce ordering except on the host side?
> >>>
> >>
> >> Right, that's exactly what the spec says in that same paragraph you
> >>quote
> >> above: "If there are ordering requirements between these commands,
> >>host software or the associated application is required to enforce
> >>that ordering above the level of the controller. " (1.3a, P. 179).
> >>
> >> Regards,
> >> Andrey
> >>
> >>
> >>> Regards,
> >>> Jonas
> >>>
> >>>
> >>> _______________________________________________
> >>> SPDK mailing list
> >>> SPDK(a)lists.01.org
> >>> https://lists.01.org/mailman/listinfo/spdk
> >>>
> >> --
> >>
> >> Regards,
> >> Andrey
> >
> >
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [SPDK] NVMe command ordering
@ 2018-03-23 16:23 Luse, Paul E
0 siblings, 0 replies; 8+ messages in thread
From: Luse, Paul E @ 2018-03-23 16:23 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 2501 bytes --]
Yup
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Jonas Pfefferle
Sent: Friday, March 23, 2018 9:21 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>; Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com>
Subject: Re: [SPDK] NVMe command ordering
One other thing I stumbled upon:
7.1 "There are no ordering restrictions for completions to the host."
Does that mean if, for example, command A and B were executed in order A,B they do not have to complete in this order on the CQ?
Thanks,
Jonas
On Fri, 23 Mar 2018 12:59:54 +0100
"Jonas Pfefferle" <pepperjo(a)japf.ch> wrote:
> Thanks!
>
> On Fri, 23 Mar 2018 11:25:26 +0000
> Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com> wrote:
>> On Fri, Mar 23, 2018, 12:04 Jonas Pfefferle <pepperjo(a)japf.ch>
>>wrote:
>>
>>> Hi @all,
>>>
>>> Can anyone clarify this for me:
>>> NVMe spec 6.3 Command Ordering Requirements "For all commands which
>>>are not part of a fused operation (refer to section 4.10), or for
>>>which the write size is greater than AWUN, each command is processed
>>>as an independent entity without reference to other commands
>>>submitted to the same I/O Submission Queue or to commands submitted
>>>to other I/O Submission Queues"
>>>
>>> So writes <= AWUN size are atomic but there is no ordering between
>>>the commands, e.g. write command A issued before write command B to
>>>the same LBA can be executed in order AB or BA, the only thing
>>>guaranteed is that they are executed atomically? And there is no way
>>>to enforce ordering except on the host side?
>>>
>>
>> Right, that's exactly what the spec says in that same paragraph you
>>quote
>> above: "If there are ordering requirements between these commands,
>>host software or the associated application is required to enforce
>>that ordering above the level of the controller. " (1.3a, P. 179).
>>
>> Regards,
>> Andrey
>>
>>
>>> Regards,
>>> Jonas
>>>
>>>
>>> _______________________________________________
>>> SPDK mailing list
>>> SPDK(a)lists.01.org
>>> https://lists.01.org/mailman/listinfo/spdk
>>>
>> --
>>
>> Regards,
>> Andrey
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [SPDK] NVMe command ordering
@ 2018-03-23 11:59 Jonas Pfefferle
0 siblings, 0 replies; 8+ messages in thread
From: Jonas Pfefferle @ 2018-03-23 11:59 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 1557 bytes --]
Thanks!
On Fri, 23 Mar 2018 11:25:26 +0000
Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com> wrote:
> On Fri, Mar 23, 2018, 12:04 Jonas Pfefferle <pepperjo(a)japf.ch>
>wrote:
>
>> Hi @all,
>>
>> Can anyone clarify this for me:
>> NVMe spec 6.3 Command Ordering Requirements
>> "For all commands which are not part of a fused operation (refer to
>>section
>> 4.10), or for which the write size is greater than AWUN, each
>>command is
>> processed as an independent entity without reference to other
>>commands
>> submitted to the same I/O Submission Queue or to commands submitted
>>to
>> other
>> I/O Submission Queues"
>>
>> So writes <= AWUN size are atomic but there is no ordering between
>>the
>> commands, e.g. write command A issued before write command B to the
>>same
>> LBA
>> can be executed in order AB or BA, the only thing guaranteed is that
>>they
>> are executed atomically? And there is no way to enforce ordering
>>except on
>> the host side?
>>
>
> Right, that's exactly what the spec says in that same paragraph you
>quote
> above: "If there are ordering requirements between these commands,
>host
> software or the associated application is required to enforce that
>ordering
> above the level of the controller. " (1.3a, P. 179).
>
> Regards,
> Andrey
>
>
>> Regards,
>> Jonas
>>
>>
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>>
> --
>
> Regards,
> Andrey
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [SPDK] NVMe command ordering
@ 2018-03-23 11:25 Andrey Kuzmin
0 siblings, 0 replies; 8+ messages in thread
From: Andrey Kuzmin @ 2018-03-23 11:25 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 1353 bytes --]
On Fri, Mar 23, 2018, 12:04 Jonas Pfefferle <pepperjo(a)japf.ch> wrote:
> Hi @all,
>
> Can anyone clarify this for me:
> NVMe spec 6.3 Command Ordering Requirements
> "For all commands which are not part of a fused operation (refer to section
> 4.10), or for which the write size is greater than AWUN, each command is
> processed as an independent entity without reference to other commands
> submitted to the same I/O Submission Queue or to commands submitted to
> other
> I/O Submission Queues"
>
> So writes <= AWUN size are atomic but there is no ordering between the
> commands, e.g. write command A issued before write command B to the same
> LBA
> can be executed in order AB or BA, the only thing guaranteed is that they
> are executed atomically? And there is no way to enforce ordering except on
> the host side?
>
Right, that's exactly what the spec says in that same paragraph you quote
above: "If there are ordering requirements between these commands, host
software or the associated application is required to enforce that ordering
above the level of the controller. " (1.3a, P. 179).
Regards,
Andrey
> Regards,
> Jonas
>
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
--
Regards,
Andrey
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 2071 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* [SPDK] NVMe command ordering
@ 2018-03-23 9:04 Jonas Pfefferle
0 siblings, 0 replies; 8+ messages in thread
From: Jonas Pfefferle @ 2018-03-23 9:04 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 771 bytes --]
Hi @all,
Can anyone clarify this for me:
NVMe spec 6.3 Command Ordering Requirements
"For all commands which are not part of a fused operation (refer to section
4.10), or for which the write size is greater than AWUN, each command is
processed as an independent entity without reference to other commands
submitted to the same I/O Submission Queue or to commands submitted to other
I/O Submission Queues"
So writes <= AWUN size are atomic but there is no ordering between the
commands, e.g. write command A issued before write command B to the same LBA
can be executed in order AB or BA, the only thing guaranteed is that they
are executed atomically? And there is no way to enforce ordering except on
the host side?
Regards,
Jonas
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2018-03-27 7:56 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-03-23 16:21 [SPDK] NVMe command ordering Jonas Pfefferle
-- strict thread matches above, loose matches on Subject: below --
2018-03-27 7:56 Jonas Pfefferle
2018-03-24 0:08 Paul Von-Stamwitz
2018-03-23 16:29 Marushak, Nathan
2018-03-23 16:23 Luse, Paul E
2018-03-23 11:59 Jonas Pfefferle
2018-03-23 11:25 Andrey Kuzmin
2018-03-23 9:04 Jonas Pfefferle
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.