[RFC] Reliable Multicast on top of RTRS

[RFC] Reliable Multicast on top of RTRS

[Danil Kipnis]

Hi @,

RTRS allows for reliable transmission of sg lists between two hosts
over rdma. It is optimised for block io. One can implement a client
and a server module on top of RTRS which would allow for reliable
transmission to a group of hosts.

In the networking world this is called reliable multicast. I think one
can say that reliable multicast is an equivalent to what is called
"mirror" in the storage world. There is something called XoR network
coding which seems to be an equivalent of raid5. There is also Reed
Solomon network coding.

Having a reliable multicast with coding rdma-based transport layer
would allow for very flexible and scalable designs of distributed
replication solutions based on different in-kernel transport, block
and replication drivers.

What do you think?

Best,
Danil.

Re: [RFC] Reliable Multicast on top of RTRS

[Bart Van Assche]

On 2020-09-02 01:04, Danil Kipnis wrote:
> RTRS allows for reliable transmission of sg lists between two hosts
> over rdma. It is optimised for block io. One can implement a client
> and a server module on top of RTRS which would allow for reliable
> transmission to a group of hosts.
> 
> In the networking world this is called reliable multicast. I think one
> can say that reliable multicast is an equivalent to what is called
> "mirror" in the storage world. There is something called XoR network
> coding which seems to be an equivalent of raid5. There is also Reed
> Solomon network coding.
> 
> Having a reliable multicast with coding rdma-based transport layer
> would allow for very flexible and scalable designs of distributed
> replication solutions based on different in-kernel transport, block
> and replication drivers.
> 
> What do you think?

How will the resulting software differ from DRBD (other than that it
uses RDMA)? How will it be guaranteed that the resulting software does
not suffer from the problems that have been solved by the introduction
of the DRBD activity log
(https://www.linbit.com/drbd-user-guide/users-guide-drbd-8-4/#s-activity-log)?

Thanks,

Bart.

Re: [RFC] Reliable Multicast on top of RTRS

[Sagi Grimberg]

> Hi @,
> 
> RTRS allows for reliable transmission of sg lists between two hosts
> over rdma. It is optimised for block io. One can implement a client
> and a server module on top of RTRS which would allow for reliable
> transmission to a group of hosts.
> 
> In the networking world this is called reliable multicast. I think one
> can say that reliable multicast is an equivalent to what is called
> "mirror" in the storage world.

md-raid1

> There is something called XoR network
> coding which seems to be an equivalent of raid5.

md-raid5

> There is also Reed
> Solomon network coding.
> 
> Having a reliable multicast with coding rdma-based transport layer
> would allow for very flexible and scalable designs of distributed
> replication solutions based on different in-kernel transport, block
> and replication drivers.
> 
> What do you think?

You should probably use the device-mapper stack or modify it to fit your
needs.

Re: [RFC] Reliable Multicast on top of RTRS

[Danil Kipnis]

On Thu, Sep 3, 2020 at 1:07 AM Bart Van Assche <bvanassche@acm.org> wrote:
>
> On 2020-09-02 01:04, Danil Kipnis wrote:
> > RTRS allows for reliable transmission of sg lists between two hosts
> > over rdma. It is optimised for block io. One can implement a client
> > and a server module on top of RTRS which would allow for reliable
> > transmission to a group of hosts.
> >
> > In the networking world this is called reliable multicast. I think one
> > can say that reliable multicast is an equivalent to what is called
> > "mirror" in the storage world. There is something called XoR network
> > coding which seems to be an equivalent of raid5. There is also Reed
> > Solomon network coding.
> >
> > Having a reliable multicast with coding rdma-based transport layer
> > would allow for very flexible and scalable designs of distributed
> > replication solutions based on different in-kernel transport, block
> > and replication drivers.
> >
> > What do you think?
>
> How will the resulting software differ from DRBD (other than that it
> uses RDMA)?

DRBD replicates disks (a local one with a remote one). The idea here
would be to replicate on the level of hosts, i.e. to support
replication inside the transport layer. A reliable mc on top of rtrs
would be more similar to a stack where an md-raid<x> sits on a client
on top of two or more imported srp/nvmeof/rnbd devices, but with an
ability to resync between remote machines directly.

> How will it be guaranteed that the resulting software does
> not suffer from the problems that have been solved by the introduction
> of the DRBD activity log
> (https://www.linbit.com/drbd-user-guide/users-guide-drbd-8-4/#s-activity-log)?

The above would require some kind of activity log also, I'm afraid.

>
> Thanks,
>
> Bart.

Re: [RFC] Reliable Multicast on top of RTRS

[Bart Van Assche]

On 2020-09-04 04:35, Danil Kipnis wrote:
> On Thu, Sep 3, 2020 at 1:07 AM Bart Van Assche <bvanassche@acm.org> wrote:
>> How will it be guaranteed that the resulting software does
>> not suffer from the problems that have been solved by the introduction
>> of the DRBD activity log
>> (https://www.linbit.com/drbd-user-guide/users-guide-drbd-8-4/#s-activity-log)?
> 
> The above would require some kind of activity log also, I'm afraid.

How about collaborating with the DRBD team? My concern is that otherwise
we will end up with two drivers in the kernel that implement block device
replication between servers connected over a network.

Thanks,

Bart.

Re: [RFC] Reliable Multicast on top of RTRS

[Danil Kipnis]

On Fri, Sep 4, 2020 at 5:33 PM Bart Van Assche <bvanassche@acm.org> wrote:
>
> On 2020-09-04 04:35, Danil Kipnis wrote:
> > On Thu, Sep 3, 2020 at 1:07 AM Bart Van Assche <bvanassche@acm.org> wrote:
> >> How will it be guaranteed that the resulting software does
> >> not suffer from the problems that have been solved by the introduction
> >> of the DRBD activity log
> >> (https://www.linbit.com/drbd-user-guide/users-guide-drbd-8-4/#s-activity-log)?
> >
> > The above would require some kind of activity log also, I'm afraid.
>
> How about collaborating with the DRBD team? My concern is that otherwise
> we will end up with two drivers in the kernel that implement block device
> replication between servers connected over a network.

I have two general understanding questions:
- What is the conceptual difference between DRBD and an md-raid1 with
one local leg and one remote (imported over srp/nvmeof/rnbd)?
- Is this possible to setup an md-raid1 on a client sitting on top of
two remote DRBD devices, which are configured in "active-active" mode?

Does anybody know?

>
> Thanks,
>
> Bart.

Re: [RFC] Reliable Multicast on top of RTRS

[Bart Van Assche]

On 9/9/20 4:42 AM, Danil Kipnis wrote:
> I have two general understanding questions:
> - What is the conceptual difference between DRBD and an md-raid1 with
> one local leg and one remote (imported over srp/nvmeof/rnbd)?

Conceptually the difference is probably not that big. But the DRBD activity
log makes a huge difference when recovering from a network disconnect or
ungraceful shutdown.

> - Is this possible to setup an md-raid1 on a client sitting on top of
> two remote DRBD devices, which are configured in "active-active" mode?

Are you perhaps referring to a single DRBD cluster? I'm not sure but I think
that would make DRBD complain about concurrent writes.

Thanks,

Bart.

Re: [RFC] Reliable Multicast on top of RTRS

[Bart Van Assche]

On 2020-09-09 04:42, Danil Kipnis wrote:
> On Fri, Sep 4, 2020 at 5:33 PM Bart Van Assche <bvanassche@acm.org> wrote:
>> On 2020-09-04 04:35, Danil Kipnis wrote:
>>> On Thu, Sep 3, 2020 at 1:07 AM Bart Van Assche <bvanassche@acm.org> wrote:
>>>> How will it be guaranteed that the resulting software does
>>>> not suffer from the problems that have been solved by the introduction
>>>> of the DRBD activity log
>>>> (https://www.linbit.com/drbd-user-guide/users-guide-drbd-8-4/#s-activity-log)?
>>>
>>> The above would require some kind of activity log also, I'm afraid.
>>
>> How about collaborating with the DRBD team? My concern is that otherwise
>> we will end up with two drivers in the kernel that implement block device
>> replication between servers connected over a network.
> 
> I have two general understanding questions:
> - What is the conceptual difference between DRBD and an md-raid1 with
> one local leg and one remote (imported over srp/nvmeof/rnbd)?

I'm not sure there is a conceptual difference. But there will be a big
difference in recovery speed after a temporary network outage (assuming that
the md-raid write intent bitmap has been disabled).

> - Is this possible to setup an md-raid1 on a client sitting on top of
> two remote DRBD devices, which are configured in "active-active" mode?

I don't think that DRBD supports this. From the DRBD source code:
"this code path is to recover from a situation that "should not happen":
concurrent writes in multi-primary setup."

Bart.

Re: [RFC] Reliable Multicast on top of RTRS

[Danil Kipnis]

Hi Bart,

On Sun, Sep 27, 2020 at 2:03 AM Bart Van Assche <bvanassche@acm.org> wrote:
>
> On 2020-09-09 04:42, Danil Kipnis wrote:
> > On Fri, Sep 4, 2020 at 5:33 PM Bart Van Assche <bvanassche@acm.org> wrote:
> >> On 2020-09-04 04:35, Danil Kipnis wrote:
> >>> On Thu, Sep 3, 2020 at 1:07 AM Bart Van Assche <bvanassche@acm.org> wrote:
> >>>> How will it be guaranteed that the resulting software does
> >>>> not suffer from the problems that have been solved by the introduction
> >>>> of the DRBD activity log
> >>>> (https://www.linbit.com/drbd-user-guide/users-guide-drbd-8-4/#s-activity-log)?
> >>>
> >>> The above would require some kind of activity log also, I'm afraid.
> >>
> >> How about collaborating with the DRBD team? My concern is that otherwise
> >> we will end up with two drivers in the kernel that implement block device
> >> replication between servers connected over a network.
> >
> > I have two general understanding questions:
> > - What is the conceptual difference between DRBD and an md-raid1 with
> > one local leg and one remote (imported over srp/nvmeof/rnbd)?
>
> I'm not sure there is a conceptual difference. But there will be a big
> difference in recovery speed after a temporary network outage (assuming that
> the md-raid write intent bitmap has been disabled).

I think RMR is conceptually different to either of the setups
(drbd or md-raid over srp/iser/nvmeof/rnbd devices) in the sense that
the logic required for replication policies (coding) is present inside rdma
subsystem which would allow to potentially offload it to underlying
rdma devices. The user of the rdma enabled devices can then utilize
them for both: block io transport and replication.

Another difference is that one can put a volume manager on top of RMR and
have it work as a distributed one.

>
> > - Is this possible to setup an md-raid1 on a client sitting on top of
> > two remote DRBD devices, which are configured in "active-active" mode?
>
> I don't think that DRBD supports this. From the DRBD source code:
> "this code path is to recover from a situation that "should not happen":
> concurrent writes in multi-primary setup."

This means md-raid on top of two block devices imported over rdma has write
latency twice shorter (while having recovery latency twice as high) as drbd.
RMR would allow for having single hop for both: write IO and resync IO.

Thank you,
Danil.

Re: [RFC] Reliable Multicast on top of RTRS

[Danil Kipnis]

Hi Sagi,

On Fri, Sep 4, 2020 at 12:18 AM Sagi Grimberg <sagi@grimberg.me> wrote:
>
>
> > Hi @,
> >
> > RTRS allows for reliable transmission of sg lists between two hosts
> > over rdma. It is optimised for block io. One can implement a client
> > and a server module on top of RTRS which would allow for reliable
> > transmission to a group of hosts.
> >
> > In the networking world this is called reliable multicast. I think one
> > can say that reliable multicast is an equivalent to what is called
> > "mirror" in the storage world.
>
> md-raid1

Exact

>
> > There is something called XoR network
> > coding which seems to be an equivalent of raid5.
>
> md-raid5

Exact

>
> > There is also Reed
> > Solomon network coding.
> >
> > Having a reliable multicast with coding rdma-based transport layer
> > would allow for very flexible and scalable designs of distributed
> > replication solutions based on different in-kernel transport, block
> > and replication drivers.
> >
> > What do you think?
>
> You should probably use the device-mapper stack or modify it to fit your
> needs.

Will look into the dm framework,
Thank you,
Danil.

Re: [RFC] Reliable Multicast on top of RTRS

[Danil Kipnis]

On Fri, Sep 4, 2020 at 5:33 PM Bart Van Assche <bvanassche@acm.org> wrote:
>
> On 2020-09-04 04:35, Danil Kipnis wrote:
> > On Thu, Sep 3, 2020 at 1:07 AM Bart Van Assche <bvanassche@acm.org> wrote:
> >> How will it be guaranteed that the resulting software does
> >> not suffer from the problems that have been solved by the introduction
> >> of the DRBD activity log
> >> (https://www.linbit.com/drbd-user-guide/users-guide-drbd-8-4/#s-activity-log)?
> >
> > The above would require some kind of activity log also, I'm afraid.
>
> How about collaborating with the DRBD team? My concern is that otherwise
> we will end up with two drivers in the kernel that implement block device
> replication between servers connected over a network.

Will take a closer look at drbd,

Thank you,
Danil.

>
> Thanks,
>
> Bart.

Re: [RFC] Reliable Multicast on top of RTRS

[Haris Iqbal]

Hey there,

We've got a prototype version working in-house. We've also implemented
a block device client and server. Another client and server could
probably be rbd and dm-thin. Will update as soon as we have a github
link.

Some technical details

- Ability to perform sync across storage nodes without the involvement
of the client.

- This helps performing sync, extending/adding legs/members without
the help of the client.

Candidate users

- We've implemented a stand-alone replicating block device. The client
is similar to the rnbd-clt and our own corresponding "store"/server
does linear mapping on the server side.

- rbd could be a client of rmr-clt. The rmr-srv (store) would talk to
lvm. rbd would provide the block device (over multiple objects) on the
client side. Lvm would function as the store on the server side. One
object would be stored on one dm-thin volume. rmr would provide for
the replication in the network.

Setups: RMR vs MD-RAID vs DRBD

- Active-active

- RMR as means of replication over network differs from the md-raid
configuration because sync traffic goes directly between servers. The
difference to the drbd setup is that the IO traffic goes to both legs
in a single hop.

How does RMR solve the activity log issue

- Synchronous replication; much like Protocol C of DRBD.

- RMR tracks all successful queue_depth (max number of IOs that can be
inflight at any moment) worth of last IOs on each storage node.

Best,
Haris

Signed-off: alexv.marov@gmail.com
Reviewed-by: danil.kipnis@posteo.net

On Sun, Nov 22, 2020 at 5:20 PM Danil Kipnis
<danil.kipnis@cloud.ionos.com> wrote:
>
> On Fri, Sep 4, 2020 at 5:33 PM Bart Van Assche <bvanassche@acm.org> wrote:
> >
> > On 2020-09-04 04:35, Danil Kipnis wrote:
> > > On Thu, Sep 3, 2020 at 1:07 AM Bart Van Assche <bvanassche@acm.org> wrote:
> > >> How will it be guaranteed that the resulting software does
> > >> not suffer from the problems that have been solved by the introduction
> > >> of the DRBD activity log
> > >> (https://www.linbit.com/drbd-user-guide/users-guide-drbd-8-4/#s-activity-log)?
> > >
> > > The above would require some kind of activity log also, I'm afraid.
> >
> > How about collaborating with the DRBD team? My concern is that otherwise
> > we will end up with two drivers in the kernel that implement block device
> > replication between servers connected over a network.
>
> Will take a closer look at drbd,
>
> Thank you,
> Danil.
>
> >
> > Thanks,
> >
> > Bart.

Re: [RFC] Reliable Multicast on top of RTRS

[danil.kipnis]

Hi Haris,

Thanks for pushing RMR forward! Lutz Pogrell and Alexei and you did a 
huge amount of work. I'm honestly convinced RMR will be a very useful 
extension for the kernel. It is currently used with a block client and a 
block server. Haris already mentioned rbd and dm-thin as a different 
potential pair of users. I'm still of the opinion that sched and mm 
could be the most interesting users of rmr.

Looking forward to the github link,
Cheers,
Danil.

P.S. BTW, where is my beloved libtbl?!


On 21.11.2022 14:57, Haris Iqbal wrote:
> Hey there,
> 
> We've got a prototype version working in-house. We've also implemented
> a block device client and server. Another client and server could
> probably be rbd and dm-thin. Will update as soon as we have a github
> link.
> 
> Some technical details
> 
> - Ability to perform sync across storage nodes without the involvement
> of the client.
> 
> - This helps performing sync, extending/adding legs/members without
> the help of the client.
> 
> Candidate users
> 
> - We've implemented a stand-alone replicating block device. The client
> is similar to the rnbd-clt and our own corresponding "store"/server
> does linear mapping on the server side.
> 
> - rbd could be a client of rmr-clt. The rmr-srv (store) would talk to
> lvm. rbd would provide the block device (over multiple objects) on the
> client side. Lvm would function as the store on the server side. One
> object would be stored on one dm-thin volume. rmr would provide for
> the replication in the network.
> 
> Setups: RMR vs MD-RAID vs DRBD
> 
> - Active-active
> 
> - RMR as means of replication over network differs from the md-raid
> configuration because sync traffic goes directly between servers. The
> difference to the drbd setup is that the IO traffic goes to both legs
> in a single hop.
> 
> How does RMR solve the activity log issue
> 
> - Synchronous replication; much like Protocol C of DRBD.
> 
> - RMR tracks all successful queue_depth (max number of IOs that can be
> inflight at any moment) worth of last IOs on each storage node.
> 
> Best,
> Haris
> 
> Signed-off: alexv.marov@gmail.com
> Reviewed-by: danil.kipnis@posteo.net
> 
> On Sun, Nov 22, 2020 at 5:20 PM Danil Kipnis
> <danil.kipnis@cloud.ionos.com> wrote:
>> 
>> On Fri, Sep 4, 2020 at 5:33 PM Bart Van Assche <bvanassche@acm.org> 
>> wrote:
>> >
>> > On 2020-09-04 04:35, Danil Kipnis wrote:
>> > > On Thu, Sep 3, 2020 at 1:07 AM Bart Van Assche <bvanassche@acm.org> wrote:
>> > >> How will it be guaranteed that the resulting software does
>> > >> not suffer from the problems that have been solved by the introduction
>> > >> of the DRBD activity log
>> > >> (https://www.linbit.com/drbd-user-guide/users-guide-drbd-8-4/#s-activity-log)?
>> > >
>> > > The above would require some kind of activity log also, I'm afraid.
>> >
>> > How about collaborating with the DRBD team? My concern is that otherwise
>> > we will end up with two drivers in the kernel that implement block device
>> > replication between servers connected over a network.
>> 
>> Will take a closer look at drbd,
>> 
>> Thank you,
>> Danil.
>> 
>> >
>> > Thanks,
>> >
>> > Bart.