[SPDK] vhost target failover?

[SPDK] vhost target failover?

[Huang Frank]

[-- Attachment #1: Type: text/plain, Size: 230 bytes --]

Hi，all

What's the failover mechism of vhost target? If the vhost target goes to crash, how to make sure the guest can process without affectting?

Thanks,
Frank.

________________________________
kinzent(a)hotmail.com

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 721 bytes --]

Re: [SPDK] vhost target failover?

[Ming Lin]

[-- Attachment #1: Type: text/plain, Size: 4388 bytes --]

Hi Frank,

The virtio initiator will automatically re-connect to vhost target
after vhost daemon restarts.

Regards,
Ming

On Mon, Apr 16, 2018 at 6:55 PM, Huang Frank <kinzent(a)hotmail.com> wrote:
> Hi,Ming
>
> The work is awesome!
> Do you mind to share some key points of your solution?
>
> ________________________________
> kinzent(a)hotmail.com
>
>
> From: Ming Lin
> Date: 2018-04-17 07:14
> To: Storage Performance Development Kit
> Subject: Re: [SPDK] vhost target failover?
> On Mon, Apr 16, 2018 at 12:37 PM, Harris, James R
> <james.r.harris(a)intel.com> wrote:
>> I agree with Pawel regarding vhost crashes.  If the SPDK vhost target
>> crashed, I would recommend treating it similarly to the host kernel
>> crashing.  There’s no way to know exactly what may have been inadvertently
>> modified due to the crash.
>>
>>
>>
>> But the idea of “managed failover” could be useful in terms of upgrading
>> the
>> vhost target.  I’m not aware of anyone actively working on this at the
>> moment however.
>
> Hi Jim,
>
> We(Alibaba) have done some work to handle vhost target crash.
> We'll share it(and other modification&improvement) in the coming US SPDK
> summit.
>
> Regards,
> Ming
>
>>
>>
>>
>> Regards,
>>
>>
>>
>> -Jim
>>
>>
>>
>>
>>
>> From: "Wodkowski, PawelX" <pawelx.wodkowski(a)intel.com>
>> Date: Monday, April 16, 2018 at 1:47 AM
>> To: Storage Performance Development Kit <spdk(a)lists.01.org>, James Harris
>> <james.r.harris(a)intel.com>
>> Subject: RE: Re: [SPDK] vhost target failover?
>>
>>
>>
>> We can implement any kind of failover mechanism. Starting some kind of
>> proxy
>> process to pass vhost user messages to child process would not be so hard
>> to
>> implement. In theory.
>>
>> But the question is how to handle crash in the middle of request? In
>> virtio-net situation is much simpler – if TCP network packet is lost it
>> will
>> be retransmitted. If packet is retransmitted after vhost-net target
>> restart
>> - it will be filtered by TCP layer. What to do if in the middle of eg 2MB
>> write vhost crashes and restarts? One solution could be to promote the
>> vhost
>> target restart/recover to the SCSI reset event. It's doable.
>>
>>
>>
>> Another issue is that if vhost crashes it mean there was some serious
>> issue
>> including possibility of trashing guest memory. Personally, I recommend
>> (and
>> I think DPDK recommend this also) to save what you can and restart
>> everything.
>>
>>
>>
>> Adding Jim as he could write something more here.
>>
>>
>>
>> Pawel
>>
>>
>>
>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Huang Frank
>> Sent: Monday, April 16, 2018 10:09 AM
>> To: spdk <spdk(a)lists.01.org>
>> Subject: Re: [SPDK] vhost target failover?
>>
>>
>>
>> hi, Pawel
>>
>>
>>
>> Thanks.
>>
>> Is there any  suggestions for this issue? Or any plans to provide High
>> Availability of vhost target in the future?
>>
>>
>>
>> ________________________________
>>
>> kinzent(a)hotmail.com
>>
>>
>>
>> From: Wodkowski, PawelX
>>
>> Date: 2018-04-16 16:02
>>
>> To: Storage Performance Development Kit
>>
>> Subject: Re: [SPDK] vhost target failover?
>>
>> There is none currently. Device will be seen as dead. All requests will
>> stuck. Anything else related to guest should be unaffected.
>>
>>
>>
>> Pawel
>>
>>
>>
>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Huang Frank
>> Sent: Monday, April 16, 2018 9:42 AM
>> To: spdk <spdk(a)lists.01.org>
>> Subject: [SPDK] vhost target failover?
>>
>>
>>
>> Hi，all
>>
>>
>>
>> What's the failover mechism of vhost target? If the vhost target goes to
>> crash, how to make sure the guest can process without affectting?
>>
>>
>>
>> Thanks,
>>
>> Frank.
>>
>>
>>
>> ________________________________
>>
>> kinzent(a)hotmail.com
>>
>>
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

Re: [SPDK] vhost target failover?

[Huang Frank]

[-- Attachment #1: Type: text/plain, Size: 3868 bytes --]

Hi,Ming

The work is awesome!
Do you mind to share some key points of your solution?

________________________________
kinzent(a)hotmail.com

From: Ming Lin<mailto:minggr(a)gmail.com>
Date: 2018-04-17 07:14
To: Storage Performance Development Kit<mailto:spdk(a)lists.01.org>
Subject: Re: [SPDK] vhost target failover?
On Mon, Apr 16, 2018 at 12:37 PM, Harris, James R
<james.r.harris(a)intel.com> wrote:
> I agree with Pawel regarding vhost crashes.  If the SPDK vhost target
> crashed, I would recommend treating it similarly to the host kernel
> crashing.  There’s no way to know exactly what may have been inadvertently
> modified due to the crash.
>
>
>
> But the idea of “managed failover” could be useful in terms of upgrading the
> vhost target.  I’m not aware of anyone actively working on this at the
> moment however.

Hi Jim,

We(Alibaba) have done some work to handle vhost target crash.
We'll share it(and other modification&improvement) in the coming US SPDK summit.

Regards,
Ming

>
>
>
> Regards,
>
>
>
> -Jim
>
>
>
>
>
> From: "Wodkowski, PawelX" <pawelx.wodkowski(a)intel.com>
> Date: Monday, April 16, 2018 at 1:47 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>, James Harris
> <james.r.harris(a)intel.com>
> Subject: RE: Re: [SPDK] vhost target failover?
>
>
>
> We can implement any kind of failover mechanism. Starting some kind of proxy
> process to pass vhost user messages to child process would not be so hard to
> implement. In theory.
>
> But the question is how to handle crash in the middle of request? In
> virtio-net situation is much simpler – if TCP network packet is lost it will
> be retransmitted. If packet is retransmitted after vhost-net target restart
> - it will be filtered by TCP layer. What to do if in the middle of eg 2MB
> write vhost crashes and restarts? One solution could be to promote the vhost
> target restart/recover to the SCSI reset event. It's doable.
>
>
>
> Another issue is that if vhost crashes it mean there was some serious issue
> including possibility of trashing guest memory. Personally, I recommend (and
> I think DPDK recommend this also) to save what you can and restart
> everything.
>
>
>
> Adding Jim as he could write something more here.
>
>
>
> Pawel
>
>
>
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Huang Frank
> Sent: Monday, April 16, 2018 10:09 AM
> To: spdk <spdk(a)lists.01.org>
> Subject: Re: [SPDK] vhost target failover?
>
>
>
> hi, Pawel
>
>
>
> Thanks.
>
> Is there any  suggestions for this issue? Or any plans to provide High
> Availability of vhost target in the future?
>
>
>
> ________________________________
>
> kinzent(a)hotmail.com
>
>
>
> From: Wodkowski, PawelX
>
> Date: 2018-04-16 16:02
>
> To: Storage Performance Development Kit
>
> Subject: Re: [SPDK] vhost target failover?
>
> There is none currently. Device will be seen as dead. All requests will
> stuck. Anything else related to guest should be unaffected.
>
>
>
> Pawel
>
>
>
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Huang Frank
> Sent: Monday, April 16, 2018 9:42 AM
> To: spdk <spdk(a)lists.01.org>
> Subject: [SPDK] vhost target failover?
>
>
>
> Hi，all
>
>
>
> What's the failover mechism of vhost target? If the vhost target goes to
> crash, how to make sure the guest can process without affectting?
>
>
>
> Thanks,
>
> Frank.
>
>
>
> ________________________________
>
> kinzent(a)hotmail.com
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 7027 bytes --]

Re: [SPDK] vhost target failover?

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 4878 bytes --]

Awesome, thanks Ming!

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Ming Lin
Sent: Monday, April 16, 2018 4:41 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] vhost target failover?

On Mon, Apr 16, 2018 at 4:19 PM, Luse, Paul E <paul.e.luse(a)intel.com> wrote:
> Hi Ming,
>
> Any chance of seeing a patch series before then? That would be a valuable contribution to the community for sure!

Hi Paul,

We are on the process to get internal approve and will send the patches out then.

Regards,
Ming

>
> Thx
> Paul
>
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Ming Lin
> Sent: Monday, April 16, 2018 4:15 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] vhost target failover?
>
> On Mon, Apr 16, 2018 at 12:37 PM, Harris, James R <james.r.harris(a)intel.com> wrote:
>> I agree with Pawel regarding vhost crashes.  If the SPDK vhost target 
>> crashed, I would recommend treating it similarly to the host kernel 
>> crashing.  There’s no way to know exactly what may have been 
>> inadvertently modified due to the crash.
>>
>>
>>
>> But the idea of “managed failover” could be useful in terms of 
>> upgrading the vhost target.  I’m not aware of anyone actively working 
>> on this at the moment however.
>
> Hi Jim,
>
> We(Alibaba) have done some work to handle vhost target crash.
> We'll share it(and other modification&improvement) in the coming US SPDK summit.
>
> Regards,
> Ming
>
>>
>>
>>
>> Regards,
>>
>>
>>
>> -Jim
>>
>>
>>
>>
>>
>> From: "Wodkowski, PawelX" <pawelx.wodkowski(a)intel.com>
>> Date: Monday, April 16, 2018 at 1:47 AM
>> To: Storage Performance Development Kit <spdk(a)lists.01.org>, James 
>> Harris <james.r.harris(a)intel.com>
>> Subject: RE: Re: [SPDK] vhost target failover?
>>
>>
>>
>> We can implement any kind of failover mechanism. Starting some kind 
>> of proxy process to pass vhost user messages to child process would 
>> not be so hard to implement. In theory.
>>
>> But the question is how to handle crash in the middle of request? In 
>> virtio-net situation is much simpler – if TCP network packet is lost 
>> it will be retransmitted. If packet is retransmitted after vhost-net 
>> target restart
>> - it will be filtered by TCP layer. What to do if in the middle of eg 
>> 2MB write vhost crashes and restarts? One solution could be to 
>> promote the vhost target restart/recover to the SCSI reset event. It's doable.
>>
>>
>>
>> Another issue is that if vhost crashes it mean there was some serious 
>> issue including possibility of trashing guest memory. Personally, I 
>> recommend (and I think DPDK recommend this also) to save what you can 
>> and restart everything.
>>
>>
>>
>> Adding Jim as he could write something more here.
>>
>>
>>
>> Pawel
>>
>>
>>
>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Huang 
>> Frank
>> Sent: Monday, April 16, 2018 10:09 AM
>> To: spdk <spdk(a)lists.01.org>
>> Subject: Re: [SPDK] vhost target failover?
>>
>>
>>
>> hi, Pawel
>>
>>
>>
>> Thanks.
>>
>> Is there any  suggestions for this issue? Or any plans to provide 
>> High Availability of vhost target in the future?
>>
>>
>>
>> ________________________________
>>
>> kinzent(a)hotmail.com
>>
>>
>>
>> From: Wodkowski, PawelX
>>
>> Date: 2018-04-16 16:02
>>
>> To: Storage Performance Development Kit
>>
>> Subject: Re: [SPDK] vhost target failover?
>>
>> There is none currently. Device will be seen as dead. All requests 
>> will stuck. Anything else related to guest should be unaffected.
>>
>>
>>
>> Pawel
>>
>>
>>
>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Huang 
>> Frank
>> Sent: Monday, April 16, 2018 9:42 AM
>> To: spdk <spdk(a)lists.01.org>
>> Subject: [SPDK] vhost target failover?
>>
>>
>>
>> Hi，all
>>
>>
>>
>> What's the failover mechism of vhost target? If the vhost target goes 
>> to crash, how to make sure the guest can process without affectting?
>>
>>
>>
>> Thanks,
>>
>> Frank.
>>
>>
>>
>> ________________________________
>>
>> kinzent(a)hotmail.com
>>
>>
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] vhost target failover?

[Ming Lin]

[-- Attachment #1: Type: text/plain, Size: 4446 bytes --]

On Mon, Apr 16, 2018 at 4:19 PM, Luse, Paul E <paul.e.luse(a)intel.com> wrote:
> Hi Ming,
>
> Any chance of seeing a patch series before then? That would be a valuable contribution to the community for sure!

Hi Paul,

We are on the process to get internal approve and will send the
patches out then.

Regards,
Ming

>
> Thx
> Paul
>
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Ming Lin
> Sent: Monday, April 16, 2018 4:15 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] vhost target failover?
>
> On Mon, Apr 16, 2018 at 12:37 PM, Harris, James R <james.r.harris(a)intel.com> wrote:
>> I agree with Pawel regarding vhost crashes.  If the SPDK vhost target
>> crashed, I would recommend treating it similarly to the host kernel
>> crashing.  There’s no way to know exactly what may have been
>> inadvertently modified due to the crash.
>>
>>
>>
>> But the idea of “managed failover” could be useful in terms of
>> upgrading the vhost target.  I’m not aware of anyone actively working
>> on this at the moment however.
>
> Hi Jim,
>
> We(Alibaba) have done some work to handle vhost target crash.
> We'll share it(and other modification&improvement) in the coming US SPDK summit.
>
> Regards,
> Ming
>
>>
>>
>>
>> Regards,
>>
>>
>>
>> -Jim
>>
>>
>>
>>
>>
>> From: "Wodkowski, PawelX" <pawelx.wodkowski(a)intel.com>
>> Date: Monday, April 16, 2018 at 1:47 AM
>> To: Storage Performance Development Kit <spdk(a)lists.01.org>, James
>> Harris <james.r.harris(a)intel.com>
>> Subject: RE: Re: [SPDK] vhost target failover?
>>
>>
>>
>> We can implement any kind of failover mechanism. Starting some kind of
>> proxy process to pass vhost user messages to child process would not
>> be so hard to implement. In theory.
>>
>> But the question is how to handle crash in the middle of request? In
>> virtio-net situation is much simpler – if TCP network packet is lost
>> it will be retransmitted. If packet is retransmitted after vhost-net
>> target restart
>> - it will be filtered by TCP layer. What to do if in the middle of eg
>> 2MB write vhost crashes and restarts? One solution could be to promote
>> the vhost target restart/recover to the SCSI reset event. It's doable.
>>
>>
>>
>> Another issue is that if vhost crashes it mean there was some serious
>> issue including possibility of trashing guest memory. Personally, I
>> recommend (and I think DPDK recommend this also) to save what you can
>> and restart everything.
>>
>>
>>
>> Adding Jim as he could write something more here.
>>
>>
>>
>> Pawel
>>
>>
>>
>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Huang Frank
>> Sent: Monday, April 16, 2018 10:09 AM
>> To: spdk <spdk(a)lists.01.org>
>> Subject: Re: [SPDK] vhost target failover?
>>
>>
>>
>> hi, Pawel
>>
>>
>>
>> Thanks.
>>
>> Is there any  suggestions for this issue? Or any plans to provide High
>> Availability of vhost target in the future?
>>
>>
>>
>> ________________________________
>>
>> kinzent(a)hotmail.com
>>
>>
>>
>> From: Wodkowski, PawelX
>>
>> Date: 2018-04-16 16:02
>>
>> To: Storage Performance Development Kit
>>
>> Subject: Re: [SPDK] vhost target failover?
>>
>> There is none currently. Device will be seen as dead. All requests
>> will stuck. Anything else related to guest should be unaffected.
>>
>>
>>
>> Pawel
>>
>>
>>
>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Huang Frank
>> Sent: Monday, April 16, 2018 9:42 AM
>> To: spdk <spdk(a)lists.01.org>
>> Subject: [SPDK] vhost target failover?
>>
>>
>>
>> Hi，all
>>
>>
>>
>> What's the failover mechism of vhost target? If the vhost target goes
>> to crash, how to make sure the guest can process without affectting?
>>
>>
>>
>> Thanks,
>>
>> Frank.
>>
>>
>>
>> ________________________________
>>
>> kinzent(a)hotmail.com
>>
>>
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] vhost target failover?

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 3947 bytes --]

Hi Ming,

Any chance of seeing a patch series before then? That would be a valuable contribution to the community for sure!

Thx
Paul

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Ming Lin
Sent: Monday, April 16, 2018 4:15 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] vhost target failover?

On Mon, Apr 16, 2018 at 12:37 PM, Harris, James R <james.r.harris(a)intel.com> wrote:
> I agree with Pawel regarding vhost crashes.  If the SPDK vhost target 
> crashed, I would recommend treating it similarly to the host kernel 
> crashing.  There’s no way to know exactly what may have been 
> inadvertently modified due to the crash.
>
>
>
> But the idea of “managed failover” could be useful in terms of 
> upgrading the vhost target.  I’m not aware of anyone actively working 
> on this at the moment however.

Hi Jim,

We(Alibaba) have done some work to handle vhost target crash.
We'll share it(and other modification&improvement) in the coming US SPDK summit.

Regards,
Ming

>
>
>
> Regards,
>
>
>
> -Jim
>
>
>
>
>
> From: "Wodkowski, PawelX" <pawelx.wodkowski(a)intel.com>
> Date: Monday, April 16, 2018 at 1:47 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>, James 
> Harris <james.r.harris(a)intel.com>
> Subject: RE: Re: [SPDK] vhost target failover?
>
>
>
> We can implement any kind of failover mechanism. Starting some kind of 
> proxy process to pass vhost user messages to child process would not 
> be so hard to implement. In theory.
>
> But the question is how to handle crash in the middle of request? In 
> virtio-net situation is much simpler – if TCP network packet is lost 
> it will be retransmitted. If packet is retransmitted after vhost-net 
> target restart
> - it will be filtered by TCP layer. What to do if in the middle of eg 
> 2MB write vhost crashes and restarts? One solution could be to promote 
> the vhost target restart/recover to the SCSI reset event. It's doable.
>
>
>
> Another issue is that if vhost crashes it mean there was some serious 
> issue including possibility of trashing guest memory. Personally, I 
> recommend (and I think DPDK recommend this also) to save what you can 
> and restart everything.
>
>
>
> Adding Jim as he could write something more here.
>
>
>
> Pawel
>
>
>
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Huang Frank
> Sent: Monday, April 16, 2018 10:09 AM
> To: spdk <spdk(a)lists.01.org>
> Subject: Re: [SPDK] vhost target failover?
>
>
>
> hi, Pawel
>
>
>
> Thanks.
>
> Is there any  suggestions for this issue? Or any plans to provide High 
> Availability of vhost target in the future?
>
>
>
> ________________________________
>
> kinzent(a)hotmail.com
>
>
>
> From: Wodkowski, PawelX
>
> Date: 2018-04-16 16:02
>
> To: Storage Performance Development Kit
>
> Subject: Re: [SPDK] vhost target failover?
>
> There is none currently. Device will be seen as dead. All requests 
> will stuck. Anything else related to guest should be unaffected.
>
>
>
> Pawel
>
>
>
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Huang Frank
> Sent: Monday, April 16, 2018 9:42 AM
> To: spdk <spdk(a)lists.01.org>
> Subject: [SPDK] vhost target failover?
>
>
>
> Hi，all
>
>
>
> What's the failover mechism of vhost target? If the vhost target goes 
> to crash, how to make sure the guest can process without affectting?
>
>
>
> Thanks,
>
> Frank.
>
>
>
> ________________________________
>
> kinzent(a)hotmail.com
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] vhost target failover?

[Ming Lin]

[-- Attachment #1: Type: text/plain, Size: 3405 bytes --]

On Mon, Apr 16, 2018 at 12:37 PM, Harris, James R
<james.r.harris(a)intel.com> wrote:
> I agree with Pawel regarding vhost crashes.  If the SPDK vhost target
> crashed, I would recommend treating it similarly to the host kernel
> crashing.  There’s no way to know exactly what may have been inadvertently
> modified due to the crash.
>
>
>
> But the idea of “managed failover” could be useful in terms of upgrading the
> vhost target.  I’m not aware of anyone actively working on this at the
> moment however.

Hi Jim,

We(Alibaba) have done some work to handle vhost target crash.
We'll share it(and other modification&improvement) in the coming US SPDK summit.

Regards,
Ming

>
>
>
> Regards,
>
>
>
> -Jim
>
>
>
>
>
> From: "Wodkowski, PawelX" <pawelx.wodkowski(a)intel.com>
> Date: Monday, April 16, 2018 at 1:47 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>, James Harris
> <james.r.harris(a)intel.com>
> Subject: RE: Re: [SPDK] vhost target failover?
>
>
>
> We can implement any kind of failover mechanism. Starting some kind of proxy
> process to pass vhost user messages to child process would not be so hard to
> implement. In theory.
>
> But the question is how to handle crash in the middle of request? In
> virtio-net situation is much simpler – if TCP network packet is lost it will
> be retransmitted. If packet is retransmitted after vhost-net target restart
> - it will be filtered by TCP layer. What to do if in the middle of eg 2MB
> write vhost crashes and restarts? One solution could be to promote the vhost
> target restart/recover to the SCSI reset event. It's doable.
>
>
>
> Another issue is that if vhost crashes it mean there was some serious issue
> including possibility of trashing guest memory. Personally, I recommend (and
> I think DPDK recommend this also) to save what you can and restart
> everything.
>
>
>
> Adding Jim as he could write something more here.
>
>
>
> Pawel
>
>
>
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Huang Frank
> Sent: Monday, April 16, 2018 10:09 AM
> To: spdk <spdk(a)lists.01.org>
> Subject: Re: [SPDK] vhost target failover?
>
>
>
> hi, Pawel
>
>
>
> Thanks.
>
> Is there any  suggestions for this issue? Or any plans to provide High
> Availability of vhost target in the future?
>
>
>
> ________________________________
>
> kinzent(a)hotmail.com
>
>
>
> From: Wodkowski, PawelX
>
> Date: 2018-04-16 16:02
>
> To: Storage Performance Development Kit
>
> Subject: Re: [SPDK] vhost target failover?
>
> There is none currently. Device will be seen as dead. All requests will
> stuck. Anything else related to guest should be unaffected.
>
>
>
> Pawel
>
>
>
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Huang Frank
> Sent: Monday, April 16, 2018 9:42 AM
> To: spdk <spdk(a)lists.01.org>
> Subject: [SPDK] vhost target failover?
>
>
>
> Hi，all
>
>
>
> What's the failover mechism of vhost target? If the vhost target goes to
> crash, how to make sure the guest can process without affectting?
>
>
>
> Thanks,
>
> Frank.
>
>
>
> ________________________________
>
> kinzent(a)hotmail.com
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

Re: [SPDK] vhost target failover?

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 2845 bytes --]

I agree with Pawel regarding vhost crashes.  If the SPDK vhost target crashed, I would recommend treating it similarly to the host kernel crashing.  There’s no way to know exactly what may have been inadvertently modified due to the crash.

But the idea of “managed failover” could be useful in terms of upgrading the vhost target.  I’m not aware of anyone actively working on this at the moment however.

Regards,

-Jim


From: "Wodkowski, PawelX" <pawelx.wodkowski(a)intel.com>
Date: Monday, April 16, 2018 at 1:47 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>, James Harris <james.r.harris(a)intel.com>
Subject: RE: Re: [SPDK] vhost target failover?

We can implement any kind of failover mechanism. Starting some kind of proxy process to pass vhost user messages to child process would not be so hard to implement. In theory.
But the question is how to handle crash in the middle of request? In virtio-net situation is much simpler – if TCP network packet is lost it will be retransmitted. If packet is retransmitted after vhost-net target restart - it will be filtered by TCP layer. What to do if in the middle of eg 2MB write vhost crashes and restarts? One solution could be to promote the vhost target restart/recover to the SCSI reset event. It's doable.

Another issue is that if vhost crashes it mean there was some serious issue including possibility of trashing guest memory. Personally, I recommend (and I think DPDK recommend this also) to save what you can and restart everything.

Adding Jim as he could write something more here.

Pawel

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Huang Frank
Sent: Monday, April 16, 2018 10:09 AM
To: spdk <spdk(a)lists.01.org>
Subject: Re: [SPDK] vhost target failover?

hi, Pawel

Thanks.
Is there any  suggestions for this issue? Or any plans to provide High Availability of vhost target in the future?

________________________________
kinzent(a)hotmail.com<mailto:kinzent(a)hotmail.com>

From: Wodkowski, PawelX<mailto:pawelx.wodkowski(a)intel.com>
Date: 2018-04-16 16:02
To: Storage Performance Development Kit<mailto:spdk(a)lists.01.org>
Subject: Re: [SPDK] vhost target failover?
There is none currently. Device will be seen as dead. All requests will stuck. Anything else related to guest should be unaffected.

Pawel

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Huang Frank
Sent: Monday, April 16, 2018 9:42 AM
To: spdk <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] vhost target failover?

Hi，all

What's the failover mechism of vhost target? If the vhost target goes to crash, how to make sure the guest can process without affectting?

Thanks,
Frank.

________________________________
kinzent(a)hotmail.com<mailto:kinzent(a)hotmail.com>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 14592 bytes --]

Re: [SPDK] vhost target failover?

[Wodkowski, PawelX]

[-- Attachment #1: Type: text/plain, Size: 2148 bytes --]

We can implement any kind of failover mechanism. Starting some kind of proxy process to pass vhost user messages to child process would not be so hard to implement. In theory.
But the question is how to handle crash in the middle of request? In virtio-net situation is much simpler - if TCP network packet is lost it will be retransmitted. If packet is retransmitted after vhost-net target restart - it will be filtered by TCP layer. What to do if in the middle of eg 2MB write vhost crashes and restarts? One solution could be to promote the vhost target restart/recover to the SCSI reset event. It's doable.

Another issue is that if vhost crashes it mean there was some serious issue including possibility of trashing guest memory. Personally, I recommend (and I think DPDK recommend this also) to save what you can and restart everything.

Adding Jim as he could write something more here.

Pawel

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Huang Frank
Sent: Monday, April 16, 2018 10:09 AM
To: spdk <spdk(a)lists.01.org>
Subject: Re: [SPDK] vhost target failover?

hi, Pawel

Thanks.
Is there any  suggestions for this issue? Or any plans to provide High Availability of vhost target in the future?

________________________________
kinzent(a)hotmail.com<mailto:kinzent(a)hotmail.com>

From: Wodkowski, PawelX<mailto:pawelx.wodkowski(a)intel.com>
Date: 2018-04-16 16:02
To: Storage Performance Development Kit<mailto:spdk(a)lists.01.org>
Subject: Re: [SPDK] vhost target failover?
There is none currently. Device will be seen as dead. All requests will stuck. Anything else related to guest should be unaffected.

Pawel

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Huang Frank
Sent: Monday, April 16, 2018 9:42 AM
To: spdk <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] vhost target failover?

Hi，all

What's the failover mechism of vhost target? If the vhost target goes to crash, how to make sure the guest can process without affectting?

Thanks,
Frank.

________________________________
kinzent(a)hotmail.com<mailto:kinzent(a)hotmail.com>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 13743 bytes --]

Re: [SPDK] vhost target failover?

[Huang Frank]

[-- Attachment #1: Type: text/plain, Size: 988 bytes --]

hi, Pawel

Thanks.
Is there any  suggestions for this issue? Or any plans to provide High Availability of vhost target in the future?

________________________________
kinzent(a)hotmail.com

From: Wodkowski, PawelX<mailto:pawelx.wodkowski(a)intel.com>
Date: 2018-04-16 16:02
To: Storage Performance Development Kit<mailto:spdk(a)lists.01.org>
Subject: Re: [SPDK] vhost target failover?
There is none currently. Device will be seen as dead. All requests will stuck. Anything else related to guest should be unaffected.

Pawel

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Huang Frank
Sent: Monday, April 16, 2018 9:42 AM
To: spdk <spdk(a)lists.01.org>
Subject: [SPDK] vhost target failover?

Hi，all

What's the failover mechism of vhost target? If the vhost target goes to crash, how to make sure the guest can process without affectting?

Thanks,
Frank.

________________________________
kinzent(a)hotmail.com<mailto:kinzent(a)hotmail.com>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 6816 bytes --]

Re: [SPDK] vhost target failover?

[Wodkowski, PawelX]

[-- Attachment #1: Type: text/plain, Size: 590 bytes --]

There is none currently. Device will be seen as dead. All requests will stuck. Anything else related to guest should be unaffected.

Pawel

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Huang Frank
Sent: Monday, April 16, 2018 9:42 AM
To: spdk <spdk(a)lists.01.org>
Subject: [SPDK] vhost target failover?

Hi，all

What's the failover mechism of vhost target? If the vhost target goes to crash, how to make sure the guest can process without affectting?

Thanks,
Frank.

________________________________
kinzent(a)hotmail.com<mailto:kinzent(a)hotmail.com>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 5533 bytes --]