QEMU-Devel Archive on lore.kernel.org
 help / color / Atom feed
* Network connection with COLO VM
@ 2019-11-27  4:20 Daniel Cho
  2019-11-27 10:51 ` Dr. David Alan Gilbert
  0 siblings, 1 reply; 11+ messages in thread
From: Daniel Cho @ 2019-11-27  4:20 UTC (permalink / raw)
  To: qemu-devel

[-- Attachment #1: Type: text/plain, Size: 298 bytes --]

Hello everyone,

Could we ssh to colo VM (means PVM & SVM are starting)?

SSH will connect to colo VM for a while, but it will disconnect with error
*client_loop: send disconnect: Broken pipe*

It seems to colo VM could not keep network session.

Does it be a known issue?

Best Regard,
Daniel Cho

[-- Attachment #2: Type: text/html, Size: 493 bytes --]

<div dir="ltr">Hello everyone, <div><br></div><div>Could we ssh to colo VM (means PVM &amp; SVM are starting)?</div><div><br></div><div>SSH will connect to colo VM for a while, but it will disconnect with error</div><div><b>client_loop: send disconnect: Broken pipe</b><br></div><div><b><br></b></div><div>It seems to colo VM could not keep network session.</div><div><b><br></b></div><div>Does it be a known issue?</div><div><br></div><div>Best Regard, </div><div>Daniel Cho </div></div>

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Network connection with COLO VM
  2019-11-27  4:20 Network connection with COLO VM Daniel Cho
@ 2019-11-27 10:51 ` Dr. David Alan Gilbert
  2019-11-28  1:26   ` Zhang, Chen
  0 siblings, 1 reply; 11+ messages in thread
From: Dr. David Alan Gilbert @ 2019-11-27 10:51 UTC (permalink / raw)
  To: Daniel Cho, chen.zhang, lukasstraub2; +Cc: qemu-devel

* Daniel Cho (danielcho@qnap.com) wrote:
> Hello everyone,
> 
> Could we ssh to colo VM (means PVM & SVM are starting)?
> 

Lets cc in Zhang Chen and Lukas Straub.

> SSH will connect to colo VM for a while, but it will disconnect with error
> *client_loop: send disconnect: Broken pipe*
> 
> It seems to colo VM could not keep network session.
> 
> Does it be a known issue?

That sounds like the COLO proxy is getting upset; it's supposed
to compare packets sent by the primary and secondary and only
send one to the outside - you shouldn't be talking directly to
the guest, but always via the proxy.  See docs/colo-proxy.txt

Dave

> Best Regard,
> Daniel Cho
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK



^ permalink raw reply	[flat|nested] 11+ messages in thread

* RE: Network connection with COLO VM
  2019-11-27 10:51 ` Dr. David Alan Gilbert
@ 2019-11-28  1:26   ` Zhang, Chen
  2019-11-29  2:42     ` Daniel Cho
  0 siblings, 1 reply; 11+ messages in thread
From: Zhang, Chen @ 2019-11-28  1:26 UTC (permalink / raw)
  To: Dr. David Alan Gilbert, Daniel Cho, lukasstraub2; +Cc: qemu-devel



> -----Original Message-----
> From: Dr. David Alan Gilbert <dgilbert@redhat.com>
> Sent: Wednesday, November 27, 2019 6:51 PM
> To: Daniel Cho <danielcho@qnap.com>; Zhang, Chen
> <chen.zhang@intel.com>; lukasstraub2@web.de
> Cc: qemu-devel@nongnu.org
> Subject: Re: Network connection with COLO VM
> 
> * Daniel Cho (danielcho@qnap.com) wrote:
> > Hello everyone,
> >
> > Could we ssh to colo VM (means PVM & SVM are starting)?
> >
> 
> Lets cc in Zhang Chen and Lukas Straub.

Thanks Dave.

> 
> > SSH will connect to colo VM for a while, but it will disconnect with
> > error
> > *client_loop: send disconnect: Broken pipe*
> >
> > It seems to colo VM could not keep network session.
> >
> > Does it be a known issue?
> 
> That sounds like the COLO proxy is getting upset; it's supposed to compare
> packets sent by the primary and secondary and only send one to the outside
> - you shouldn't be talking directly to the guest, but always via the proxy.  See
> docs/colo-proxy.txt
> 

Hi Daniel,

I have try ssh to COLO guest with 8 hours, not occurred this issue.
Please check your network/qemu configuration.
But I found another problem maybe related this issue, if no network communication for a period of time(maybe 10min), the first message send to guest have a chance with delay(maybe 1-5 sec), I will try to fix it when I have time.

Thanks
Zhang Chen

> Dave
> 
> > Best Regard,
> > Daniel Cho
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK



^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Network connection with COLO VM
  2019-11-28  1:26   ` Zhang, Chen
@ 2019-11-29  2:42     ` Daniel Cho
  2019-11-29 18:04       ` Zhang, Chen
  0 siblings, 1 reply; 11+ messages in thread
From: Daniel Cho @ 2019-11-29  2:42 UTC (permalink / raw)
  To: Zhang, Chen; +Cc: lukasstraub2, Dr. David Alan Gilbert, qemu-devel

[-- Attachment #1: Type: text/plain, Size: 2408 bytes --]

Hi David,  Zhang,

Thanks for replying my question.
We know why will occur this issue.
As you said, the COLO VM's network needs
colo-proxy to control packets, so the guest's
interface should set the filter to solve the problem.

But we found another question, when we set the
fault-tolerance feature to guest (primary VM is running,
secondary VM is pausing), the guest's network would not
responds any request for a while (in our environment
about 20~30 secs) after secondary VM runs.

Does it be a normal situation, or a known issue?

Our test is creating primary VM for a while, then creating
secondary VM to make it with COLO feature.

Best Regard,
Daniel Cho

Zhang, Chen <chen.zhang@intel.com> 於 2019年11月28日 週四 上午9:26寫道:

>
>
> > -----Original Message-----
> > From: Dr. David Alan Gilbert <dgilbert@redhat.com>
> > Sent: Wednesday, November 27, 2019 6:51 PM
> > To: Daniel Cho <danielcho@qnap.com>; Zhang, Chen
> > <chen.zhang@intel.com>; lukasstraub2@web.de
> > Cc: qemu-devel@nongnu.org
> > Subject: Re: Network connection with COLO VM
> >
> > * Daniel Cho (danielcho@qnap.com) wrote:
> > > Hello everyone,
> > >
> > > Could we ssh to colo VM (means PVM & SVM are starting)?
> > >
> >
> > Lets cc in Zhang Chen and Lukas Straub.
>
> Thanks Dave.
>
> >
> > > SSH will connect to colo VM for a while, but it will disconnect with
> > > error
> > > *client_loop: send disconnect: Broken pipe*
> > >
> > > It seems to colo VM could not keep network session.
> > >
> > > Does it be a known issue?
> >
> > That sounds like the COLO proxy is getting upset; it's supposed to
> compare
> > packets sent by the primary and secondary and only send one to the
> outside
> > - you shouldn't be talking directly to the guest, but always via the
> proxy.  See
> > docs/colo-proxy.txt
> >
>
> Hi Daniel,
>
> I have try ssh to COLO guest with 8 hours, not occurred this issue.
> Please check your network/qemu configuration.
> But I found another problem maybe related this issue, if no network
> communication for a period of time(maybe 10min), the first message send to
> guest have a chance with delay(maybe 1-5 sec), I will try to fix it when I
> have time.
>
> Thanks
> Zhang Chen
>
> > Dave
> >
> > > Best Regard,
> > > Daniel Cho
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
>

[-- Attachment #2: Type: text/html, Size: 3627 bytes --]

<div dir="ltr"><div>Hi David,  Zhang, </div><div><br></div><div>Thanks for replying my question. </div><div>We know why will occur this issue. </div><div>As you said, the COLO VM&#39;s network needs </div><div>colo-proxy to control packets, so the guest&#39;s </div><div>interface should set the filter to solve the problem.</div><div><br></div><div>But we found another question, when we set the </div><div>fault-tolerance feature to guest (primary VM is running, </div><div>secondary VM is pausing), the guest&#39;s network would not </div><div>responds any request for a while (in our environment </div><div>about 20~30 secs) after secondary VM runs.</div><div><br></div><div>Does it be a normal situation, or a known issue?  <br></div><div><br></div><div>Our test is creating primary VM for a while, then creating <br></div><div>secondary VM to make it with COLO feature.</div><div><br></div><div>Best Regard, </div><div>Daniel Cho </div><div><br></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Zhang, Chen &lt;<a href="mailto:chen.zhang@intel.com">chen.zhang@intel.com</a>&gt; 於 2019年11月28日 週四 上午9:26寫道:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
<br>
&gt; -----Original Message-----<br>
&gt; From: Dr. David Alan Gilbert &lt;<a href="mailto:dgilbert@redhat.com" target="_blank">dgilbert@redhat.com</a>&gt;<br>
&gt; Sent: Wednesday, November 27, 2019 6:51 PM<br>
&gt; To: Daniel Cho &lt;<a href="mailto:danielcho@qnap.com" target="_blank">danielcho@qnap.com</a>&gt;; Zhang, Chen<br>
&gt; &lt;<a href="mailto:chen.zhang@intel.com" target="_blank">chen.zhang@intel.com</a>&gt;; <a href="mailto:lukasstraub2@web.de" target="_blank">lukasstraub2@web.de</a><br>
&gt; Cc: <a href="mailto:qemu-devel@nongnu.org" target="_blank">qemu-devel@nongnu.org</a><br>
&gt; Subject: Re: Network connection with COLO VM<br>
&gt; <br>
&gt; * Daniel Cho (<a href="mailto:danielcho@qnap.com" target="_blank">danielcho@qnap.com</a>) wrote:<br>
&gt; &gt; Hello everyone,<br>
&gt; &gt;<br>
&gt; &gt; Could we ssh to colo VM (means PVM &amp; SVM are starting)?<br>
&gt; &gt;<br>
&gt; <br>
&gt; Lets cc in Zhang Chen and Lukas Straub.<br>
<br>
Thanks Dave.<br>
<br>
&gt; <br>
&gt; &gt; SSH will connect to colo VM for a while, but it will disconnect with<br>
&gt; &gt; error<br>
&gt; &gt; *client_loop: send disconnect: Broken pipe*<br>
&gt; &gt;<br>
&gt; &gt; It seems to colo VM could not keep network session.<br>
&gt; &gt;<br>
&gt; &gt; Does it be a known issue?<br>
&gt; <br>
&gt; That sounds like the COLO proxy is getting upset; it&#39;s supposed to compare<br>
&gt; packets sent by the primary and secondary and only send one to the outside<br>
&gt; - you shouldn&#39;t be talking directly to the guest, but always via the proxy.  See<br>
&gt; docs/colo-proxy.txt<br>
&gt; <br>
<br>
Hi Daniel,<br>
<br>
I have try ssh to COLO guest with 8 hours, not occurred this issue.<br>
Please check your network/qemu configuration.<br>
But I found another problem maybe related this issue, if no network communication for a period of time(maybe 10min), the first message send to guest have a chance with delay(maybe 1-5 sec), I will try to fix it when I have time.<br>
<br>
Thanks<br>
Zhang Chen<br>
<br>
&gt; Dave<br>
&gt; <br>
&gt; &gt; Best Regard,<br>
&gt; &gt; Daniel Cho<br>
&gt; --<br>
&gt; Dr. David Alan Gilbert / <a href="mailto:dgilbert@redhat.com" target="_blank">dgilbert@redhat.com</a> / Manchester, UK<br>
<br>
</blockquote></div></div>

^ permalink raw reply	[flat|nested] 11+ messages in thread

* RE: Network connection with COLO VM
  2019-11-29  2:42     ` Daniel Cho
@ 2019-11-29 18:04       ` Zhang, Chen
  2019-12-02  3:55         ` Daniel Cho
  0 siblings, 1 reply; 11+ messages in thread
From: Zhang, Chen @ 2019-11-29 18:04 UTC (permalink / raw)
  To: Daniel Cho; +Cc: lukasstraub2, Dr. David Alan Gilbert, qemu-devel

[-- Attachment #1: Type: text/plain, Size: 3141 bytes --]



From: Daniel Cho <danielcho@qnap.com>
Sent: Friday, November 29, 2019 10:43 AM
To: Zhang, Chen <chen.zhang@intel.com>
Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>; lukasstraub2@web.de; qemu-devel@nongnu.org
Subject: Re: Network connection with COLO VM

Hi David,  Zhang,

Thanks for replying my question.
We know why will occur this issue.
As you said, the COLO VM's network needs
colo-proxy to control packets, so the guest's
interface should set the filter to solve the problem.

But we found another question, when we set the
fault-tolerance feature to guest (primary VM is running,
secondary VM is pausing), the guest's network would not
responds any request for a while (in our environment
about 20~30 secs) after secondary VM runs.

Does it be a normal situation, or a known issue?

Our test is creating primary VM for a while, then creating
secondary VM to make it with COLO feature.

Hi Daniel,

Happy to hear you have solved ssh disconnection issue.

Do you use Lukas’s patch on this case?
I think we need use block mirror to sync the disk to secondary node first, then stop the primary VM and build COLO system.
In the stop moment, you need add some netfilter and chardev socket node for COLO, maybe you need re-check this part.

Best Regard,
Daniel Cho

Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>> 於 2019年11月28日 週四 上午9:26寫道:


> -----Original Message-----
> From: Dr. David Alan Gilbert <dgilbert@redhat.com<mailto:dgilbert@redhat.com>>
> Sent: Wednesday, November 27, 2019 6:51 PM
> To: Daniel Cho <danielcho@qnap.com<mailto:danielcho@qnap.com>>; Zhang, Chen
> <chen.zhang@intel.com<mailto:chen.zhang@intel.com>>; lukasstraub2@web.de<mailto:lukasstraub2@web.de>
> Cc: qemu-devel@nongnu.org<mailto:qemu-devel@nongnu.org>
> Subject: Re: Network connection with COLO VM
>
> * Daniel Cho (danielcho@qnap.com<mailto:danielcho@qnap.com>) wrote:
> > Hello everyone,
> >
> > Could we ssh to colo VM (means PVM & SVM are starting)?
> >
>
> Lets cc in Zhang Chen and Lukas Straub.

Thanks Dave.

>
> > SSH will connect to colo VM for a while, but it will disconnect with
> > error
> > *client_loop: send disconnect: Broken pipe*
> >
> > It seems to colo VM could not keep network session.
> >
> > Does it be a known issue?
>
> That sounds like the COLO proxy is getting upset; it's supposed to compare
> packets sent by the primary and secondary and only send one to the outside
> - you shouldn't be talking directly to the guest, but always via the proxy.  See
> docs/colo-proxy.txt
>

Hi Daniel,

I have try ssh to COLO guest with 8 hours, not occurred this issue.
Please check your network/qemu configuration.
But I found another problem maybe related this issue, if no network communication for a period of time(maybe 10min), the first message send to guest have a chance with delay(maybe 1-5 sec), I will try to fix it when I have time.

Thanks
Zhang Chen

> Dave
>
> > Best Regard,
> > Daniel Cho
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com<mailto:dgilbert@redhat.com> / Manchester, UK

[-- Attachment #2: Type: text/html, Size: 8459 bytes --]

<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:DengXian;
	panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
	{font-family:"\@DengXian";
	panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:purple;
	text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
	{mso-style-name:msonormal;
	mso-margin-top-alt:auto;
	margin-right:0in;
	mso-margin-bottom-alt:auto;
	margin-left:0in;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;}
span.EmailStyle18
	{mso-style-type:personal-reply;
	font-family:"Calibri",sans-serif;
	color:windowtext;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-family:"Calibri",sans-serif;}
@page WordSection1
	{size:8.5in 11.0in;
	margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
	{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><a name="_____replyseparator"></a><b>From:</b> Daniel Cho &lt;danielcho@qnap.com&gt;
<br>
<b>Sent:</b> Friday, November 29, 2019 10:43 AM<br>
<b>To:</b> Zhang, Chen &lt;chen.zhang@intel.com&gt;<br>
<b>Cc:</b> Dr. David Alan Gilbert &lt;dgilbert@redhat.com&gt;; lukasstraub2@web.de; qemu-devel@nongnu.org<br>
<b>Subject:</b> Re: Network connection with COLO VM<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<div>
<div>
<p class="MsoNormal">Hi David,&nbsp; Zhang,&nbsp;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks for replying my question.&nbsp;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">We know why will occur this issue.&nbsp;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">As you said, the COLO VM's network needs&nbsp;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">colo-proxy to control packets, so the guest's&nbsp;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">interface should set the filter to solve&nbsp;the problem.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class="MsoNormal">But we found another question, when we set the&nbsp;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">fault-tolerance feature to guest (primary VM is running,&nbsp;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">secondary VM is pausing), the guest's network would not&nbsp;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">responds any request for a while (in our environment&nbsp;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">about 20~30 secs) after secondary VM runs.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class="MsoNormal">Does it be a normal situation, or a known issue?&nbsp;&nbsp;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class="MsoNormal">Our test is creating primary VM for a while, then creating&nbsp;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">secondary VM to make it with COLO feature.<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">Hi Daniel,<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">Happy to hear you have solved ssh disconnection issue.<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">Do you use Lukas’s patch on this case?<o:p></o:p></p>
<p class="MsoNormal">I think we need use block mirror to sync the disk to secondary node first, then stop the primary VM and build COLO system.<o:p></o:p></p>
<p class="MsoNormal">In the stop moment, you need add some netfilter and chardev socket node for COLO, maybe you need re-check this part.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class="MsoNormal">Best Regard,&nbsp;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Daniel Cho&nbsp;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">Zhang, Chen &lt;<a href="mailto:chen.zhang@intel.com">chen.zhang@intel.com</a>&gt;
<span lang="ZH-CN" style="font-family:DengXian">於</span> 2019<span lang="ZH-CN" style="font-family:DengXian">年</span>11<span lang="ZH-CN" style="font-family:DengXian">月</span>28<span lang="ZH-CN" style="font-family:DengXian">日</span><span lang="ZH-CN">
</span><span lang="ZH-CN" style="font-family:DengXian">週四</span><span lang="ZH-CN">
</span><span lang="ZH-CN" style="font-family:DengXian">上午</span>9:26<span lang="ZH-CN" style="font-family:DengXian">寫道:</span><o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br>
&gt; -----Original Message-----<br>
&gt; From: Dr. David Alan Gilbert &lt;<a href="mailto:dgilbert@redhat.com" target="_blank">dgilbert@redhat.com</a>&gt;<br>
&gt; Sent: Wednesday, November 27, 2019 6:51 PM<br>
&gt; To: Daniel Cho &lt;<a href="mailto:danielcho@qnap.com" target="_blank">danielcho@qnap.com</a>&gt;; Zhang, Chen<br>
&gt; &lt;<a href="mailto:chen.zhang@intel.com" target="_blank">chen.zhang@intel.com</a>&gt;;
<a href="mailto:lukasstraub2@web.de" target="_blank">lukasstraub2@web.de</a><br>
&gt; Cc: <a href="mailto:qemu-devel@nongnu.org" target="_blank">qemu-devel@nongnu.org</a><br>
&gt; Subject: Re: Network connection with COLO VM<br>
&gt; <br>
&gt; * Daniel Cho (<a href="mailto:danielcho@qnap.com" target="_blank">danielcho@qnap.com</a>) wrote:<br>
&gt; &gt; Hello everyone,<br>
&gt; &gt;<br>
&gt; &gt; Could we ssh to colo VM (means PVM &amp; SVM are starting)?<br>
&gt; &gt;<br>
&gt; <br>
&gt; Lets cc in Zhang Chen and Lukas Straub.<br>
<br>
Thanks Dave.<br>
<br>
&gt; <br>
&gt; &gt; SSH will connect to colo VM for a while, but it will disconnect with<br>
&gt; &gt; error<br>
&gt; &gt; *client_loop: send disconnect: Broken pipe*<br>
&gt; &gt;<br>
&gt; &gt; It seems to colo VM could not keep network session.<br>
&gt; &gt;<br>
&gt; &gt; Does it be a known issue?<br>
&gt; <br>
&gt; That sounds like the COLO proxy is getting upset; it's supposed to compare<br>
&gt; packets sent by the primary and secondary and only send one to the outside<br>
&gt; - you shouldn't be talking directly to the guest, but always via the proxy.&nbsp; See<br>
&gt; docs/colo-proxy.txt<br>
&gt; <br>
<br>
Hi Daniel,<br>
<br>
I have try ssh to COLO guest with 8 hours, not occurred this issue.<br>
Please check your network/qemu configuration.<br>
But I found another problem maybe related this issue, if no network communication for a period of time(maybe 10min), the first message send to guest have a chance with delay(maybe 1-5 sec), I will try to fix it when I have time.<br>
<br>
Thanks<br>
Zhang Chen<br>
<br>
&gt; Dave<br>
&gt; <br>
&gt; &gt; Best Regard,<br>
&gt; &gt; Daniel Cho<br>
&gt; --<br>
&gt; Dr. David Alan Gilbert / <a href="mailto:dgilbert@redhat.com" target="_blank">dgilbert@redhat.com</a> / Manchester, UK<o:p></o:p></p>
</blockquote>
</div>
</div>
</div>
</div>
</body>
</html>

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Network connection with COLO VM
  2019-11-29 18:04       ` Zhang, Chen
@ 2019-12-02  3:55         ` Daniel Cho
  2019-12-02  9:58           ` Dr. David Alan Gilbert
  0 siblings, 1 reply; 11+ messages in thread
From: Daniel Cho @ 2019-12-02  3:55 UTC (permalink / raw)
  To: Zhang, Chen; +Cc: lukasstraub2, Dr. David Alan Gilbert, qemu-devel

[-- Attachment #1: Type: text/plain, Size: 4776 bytes --]

Hi Zhang,

We use qemu-4.1.0 release on this case.

I think we need use block mirror to sync the disk to secondary node first,
then stop the primary VM and build COLO system.

In the stop moment, you need add some netfilter and chardev socket node for
COLO, maybe you need re-check this part.


Our test was already follow those step. Maybe I could describe the detail
of the test flow and issues.


Step 1:

Create primary VM without any netfilter and chardev for COLO, and using
other host ping primary VM continually.


Step 2:

Create secondary VM (the same device/drive with primary VM), and do block
mirror sync ( ping to primary VM normally )


Step 3:

After block mirror sync finish, add those netfilter and chardev to primary
VM and secondary VM for COLO ( *Can't* ping to primary VM but those packets
will be received later )


Step 4:

Start migrate primary VM to secondary VM, and primary VM & secondary VM are
running ( ping to primary VM works and receive those packets on step 3
status )




Between Step 3 to Step 4, it will take 10~20 seconds in our environment.

I could image this issue (delay reply packets) is because of setting COLO
proxy for temporary status,

but we thought 10~20 seconds might a little long. (If primary VM is already
doing some jobs, it might lose the data.)


Could we reduce those time? or those delay is depends on different VM?


Best Regard,

Daniel Cho.



Zhang, Chen <chen.zhang@intel.com> 於 2019年11月30日 週六 上午2:04寫道:

>
>
>
>
> *From:* Daniel Cho <danielcho@qnap.com>
> *Sent:* Friday, November 29, 2019 10:43 AM
> *To:* Zhang, Chen <chen.zhang@intel.com>
> *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com>; lukasstraub2@web.de;
> qemu-devel@nongnu.org
> *Subject:* Re: Network connection with COLO VM
>
>
>
> Hi David,  Zhang,
>
>
>
> Thanks for replying my question.
>
> We know why will occur this issue.
>
> As you said, the COLO VM's network needs
>
> colo-proxy to control packets, so the guest's
>
> interface should set the filter to solve the problem.
>
>
>
> But we found another question, when we set the
>
> fault-tolerance feature to guest (primary VM is running,
>
> secondary VM is pausing), the guest's network would not
>
> responds any request for a while (in our environment
>
> about 20~30 secs) after secondary VM runs.
>
>
>
> Does it be a normal situation, or a known issue?
>
>
>
> Our test is creating primary VM for a while, then creating
>
> secondary VM to make it with COLO feature.
>
>
>
> Hi Daniel,
>
>
>
> Happy to hear you have solved ssh disconnection issue.
>
>
>
> Do you use Lukas’s patch on this case?
>
> I think we need use block mirror to sync the disk to secondary node first,
> then stop the primary VM and build COLO system.
>
> In the stop moment, you need add some netfilter and chardev socket node
> for COLO, maybe you need re-check this part.
>
>
>
> Best Regard,
>
> Daniel Cho
>
>
>
> Zhang, Chen <chen.zhang@intel.com> 於 2019年11月28日 週四 上午9:26寫道:
>
>
>
> > -----Original Message-----
> > From: Dr. David Alan Gilbert <dgilbert@redhat.com>
> > Sent: Wednesday, November 27, 2019 6:51 PM
> > To: Daniel Cho <danielcho@qnap.com>; Zhang, Chen
> > <chen.zhang@intel.com>; lukasstraub2@web.de
> > Cc: qemu-devel@nongnu.org
> > Subject: Re: Network connection with COLO VM
> >
> > * Daniel Cho (danielcho@qnap.com) wrote:
> > > Hello everyone,
> > >
> > > Could we ssh to colo VM (means PVM & SVM are starting)?
> > >
> >
> > Lets cc in Zhang Chen and Lukas Straub.
>
> Thanks Dave.
>
> >
> > > SSH will connect to colo VM for a while, but it will disconnect with
> > > error
> > > *client_loop: send disconnect: Broken pipe*
> > >
> > > It seems to colo VM could not keep network session.
> > >
> > > Does it be a known issue?
> >
> > That sounds like the COLO proxy is getting upset; it's supposed to
> compare
> > packets sent by the primary and secondary and only send one to the
> outside
> > - you shouldn't be talking directly to the guest, but always via the
> proxy.  See
> > docs/colo-proxy.txt
> >
>
> Hi Daniel,
>
> I have try ssh to COLO guest with 8 hours, not occurred this issue.
> Please check your network/qemu configuration.
> But I found another problem maybe related this issue, if no network
> communication for a period of time(maybe 10min), the first message send to
> guest have a chance with delay(maybe 1-5 sec), I will try to fix it when I
> have time.
>
> Thanks
> Zhang Chen
>
> > Dave
> >
> > > Best Regard,
> > > Daniel Cho
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
>

[-- Attachment #2: Type: text/html, Size: 10091 bytes --]

<div dir="ltr"><div>Hi Zhang, </div><div><br></div><div>We use qemu-4.1.0 release on this case.</div><div><br></div><div><p class="MsoNormal"><font color="#9900ff">I think we need use block mirror to sync the disk to secondary node first, then stop the primary VM and build COLO system.<u></u><u></u></font></p>
<p class="MsoNormal"><font color="#9900ff">In the stop moment, you need add some netfilter and chardev socket node for COLO, maybe you need re-check this part.</font></p><p class="MsoNormal"><font color="#9900ff"><br></font></p><p class="MsoNormal"><font color="#000000">Our test was already follow those step. Maybe I could describe the detail of the test flow and issues.</font></p><p class="MsoNormal"><font color="#000000"><br></font></p><p class="MsoNormal"><font color="#000000">Step 1: </font></p><p class="MsoNormal">Create primary VM without any netfilter and chardev for COLO, and<font color="#ff0000"> using other host ping primary VM continually</font>. </p><p class="MsoNormal"><br></p><p class="MsoNormal">Step 2: </p><p class="MsoNormal">Create secondary VM (the same device/drive with primary VM), and do block mirror sync (

<font color="#ff0000">ping to primary VM normally </font>)</p><p class="MsoNormal"><br></p><p class="MsoNormal">Step 3:</p><p class="MsoNormal">After block mirror sync finish, add those netfilter and chardev to primary VM and secondary VM for COLO ( <font size="4"><font color="#ff0000"><b>Can&#39;t</b></font> </font><font color="#ff0000">ping to primary VM but those packets will be received later </font>)</p><p class="MsoNormal"><br></p><p class="MsoNormal">Step 4: </p><p class="MsoNormal">Start migrate primary VM to secondary VM, and primary VM &amp; secondary VM are running (

<font color="#ff0000">ping to primary VM works and receive those packets on step 3 status </font>)</p><p class="MsoNormal"><br></p><p class="MsoNormal"> <br></p><p class="MsoNormal">Between Step 3 to Step 4, it will take <font color="#ff0000">10~20 seconds</font> in our environment. </p><p class="MsoNormal">I could image this issue (delay reply packets) is because of setting COLO proxy for temporary status,</p><p class="MsoNormal">but we thought <font color="#ff0000">10~20 seconds </font><font color="#000000">might a little long. (If primary VM is already doing some jobs, it might lose the data.)</font></p><p class="MsoNormal"><font color="#000000"><br></font></p><p class="MsoNormal"><font color="#000000">Could we reduce those time? or those delay is depends on different VM?</font></p><p class="MsoNormal"><br></p><p class="MsoNormal">Best Regard,</p><p class="MsoNormal">Daniel Cho.</p><p class="MsoNormal"><br></p></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Zhang, Chen &lt;<a href="mailto:chen.zhang@intel.com">chen.zhang@intel.com</a>&gt; 於 2019年11月30日 週六 上午2:04寫道:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">





<div lang="EN-US">
<div class="gmail-m_-7246858957356585846WordSection1">
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border-top:none;border-right:none;border-bottom:none;border-left:1.5pt solid blue;padding:0in 0in 0in 4pt">
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in">
<p class="MsoNormal"><a name="m_-7246858957356585846______replyseparator"></a><b>From:</b> Daniel Cho &lt;<a href="mailto:danielcho@qnap.com" target="_blank">danielcho@qnap.com</a>&gt;
<br>
<b>Sent:</b> Friday, November 29, 2019 10:43 AM<br>
<b>To:</b> Zhang, Chen &lt;<a href="mailto:chen.zhang@intel.com" target="_blank">chen.zhang@intel.com</a>&gt;<br>
<b>Cc:</b> Dr. David Alan Gilbert &lt;<a href="mailto:dgilbert@redhat.com" target="_blank">dgilbert@redhat.com</a>&gt;; <a href="mailto:lukasstraub2@web.de" target="_blank">lukasstraub2@web.de</a>; <a href="mailto:qemu-devel@nongnu.org" target="_blank">qemu-devel@nongnu.org</a><br>
<b>Subject:</b> Re: Network connection with COLO VM<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">Hi David,  Zhang, <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks for replying my question. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">We know why will occur this issue. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">As you said, the COLO VM&#39;s network needs <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">colo-proxy to control packets, so the guest&#39;s <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">interface should set the filter to solve the problem.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">But we found another question, when we set the <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">fault-tolerance feature to guest (primary VM is running, <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">secondary VM is pausing), the guest&#39;s network would not <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">responds any request for a while (in our environment <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">about 20~30 secs) after secondary VM runs.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Does it be a normal situation, or a known issue?  <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Our test is creating primary VM for a while, then creating <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">secondary VM to make it with COLO feature.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Hi Daniel,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Happy to hear you have solved ssh disconnection issue.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Do you use Lukas’s patch on this case?<u></u><u></u></p>
<p class="MsoNormal">I think we need use block mirror to sync the disk to secondary node first, then stop the primary VM and build COLO system.<u></u><u></u></p>
<p class="MsoNormal">In the stop moment, you need add some netfilter and chardev socket node for COLO, maybe you need re-check this part.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Best Regard, <u></u></p>
</div>
<div>
<p class="MsoNormal">Daniel Cho <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">Zhang, Chen &lt;<a href="mailto:chen.zhang@intel.com" target="_blank">chen.zhang@intel.com</a>&gt;
<span lang="ZH-CN" style="font-family:DengXian">於</span> 2019<span lang="ZH-CN" style="font-family:DengXian">年</span>11<span lang="ZH-CN" style="font-family:DengXian">月</span>28<span lang="ZH-CN" style="font-family:DengXian">日</span><span lang="ZH-CN">
</span><span lang="ZH-CN" style="font-family:DengXian">週四</span><span lang="ZH-CN">
</span><span lang="ZH-CN" style="font-family:DengXian">上午</span>9:26<span lang="ZH-CN" style="font-family:DengXian">寫道:</span><u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal" style="margin-bottom:12pt"><br>
<br>
&gt; -----Original Message-----<br>
&gt; From: Dr. David Alan Gilbert &lt;<a href="mailto:dgilbert@redhat.com" target="_blank">dgilbert@redhat.com</a>&gt;<br>
&gt; Sent: Wednesday, November 27, 2019 6:51 PM<br>
&gt; To: Daniel Cho &lt;<a href="mailto:danielcho@qnap.com" target="_blank">danielcho@qnap.com</a>&gt;; Zhang, Chen<br>
&gt; &lt;<a href="mailto:chen.zhang@intel.com" target="_blank">chen.zhang@intel.com</a>&gt;;
<a href="mailto:lukasstraub2@web.de" target="_blank">lukasstraub2@web.de</a><br>
&gt; Cc: <a href="mailto:qemu-devel@nongnu.org" target="_blank">qemu-devel@nongnu.org</a><br>
&gt; Subject: Re: Network connection with COLO VM<br>
&gt; <br>
&gt; * Daniel Cho (<a href="mailto:danielcho@qnap.com" target="_blank">danielcho@qnap.com</a>) wrote:<br>
&gt; &gt; Hello everyone,<br>
&gt; &gt;<br>
&gt; &gt; Could we ssh to colo VM (means PVM &amp; SVM are starting)?<br>
&gt; &gt;<br>
&gt; <br>
&gt; Lets cc in Zhang Chen and Lukas Straub.<br>
<br>
Thanks Dave.<br>
<br>
&gt; <br>
&gt; &gt; SSH will connect to colo VM for a while, but it will disconnect with<br>
&gt; &gt; error<br>
&gt; &gt; *client_loop: send disconnect: Broken pipe*<br>
&gt; &gt;<br>
&gt; &gt; It seems to colo VM could not keep network session.<br>
&gt; &gt;<br>
&gt; &gt; Does it be a known issue?<br>
&gt; <br>
&gt; That sounds like the COLO proxy is getting upset; it&#39;s supposed to compare<br>
&gt; packets sent by the primary and secondary and only send one to the outside<br>
&gt; - you shouldn&#39;t be talking directly to the guest, but always via the proxy.  See<br>
&gt; docs/colo-proxy.txt<br>
&gt; <br>
<br>
Hi Daniel,<br>
<br>
I have try ssh to COLO guest with 8 hours, not occurred this issue.<br>
Please check your network/qemu configuration.<br>
But I found another problem maybe related this issue, if no network communication for a period of time(maybe 10min), the first message send to guest have a chance with delay(maybe 1-5 sec), I will try to fix it when I have time.<br>
<br>
Thanks<br>
Zhang Chen<br>
<br>
&gt; Dave<br>
&gt; <br>
&gt; &gt; Best Regard,<br>
&gt; &gt; Daniel Cho<br>
&gt; --<br>
&gt; Dr. David Alan Gilbert / <a href="mailto:dgilbert@redhat.com" target="_blank">dgilbert@redhat.com</a> / Manchester, UK<u></u><u></u></p>
</blockquote>
</div>
</div>
</div>
</div>
</div>

</blockquote></div></div>

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Network connection with COLO VM
  2019-12-02  3:55         ` Daniel Cho
@ 2019-12-02  9:58           ` Dr. David Alan Gilbert
  2019-12-03  9:08             ` Daniel Cho
  0 siblings, 1 reply; 11+ messages in thread
From: Dr. David Alan Gilbert @ 2019-12-02  9:58 UTC (permalink / raw)
  To: Daniel Cho; +Cc: Zhang, Chen, lukasstraub2, qemu-devel

* Daniel Cho (danielcho@qnap.com) wrote:
> Hi Zhang,
> 
> We use qemu-4.1.0 release on this case.
> 
> I think we need use block mirror to sync the disk to secondary node first,
> then stop the primary VM and build COLO system.
> 
> In the stop moment, you need add some netfilter and chardev socket node for
> COLO, maybe you need re-check this part.
> 
> 
> Our test was already follow those step. Maybe I could describe the detail
> of the test flow and issues.
> 
> 
> Step 1:
> 
> Create primary VM without any netfilter and chardev for COLO, and using
> other host ping primary VM continually.
> 
> 
> Step 2:
> 
> Create secondary VM (the same device/drive with primary VM), and do block
> mirror sync ( ping to primary VM normally )
> 
> 
> Step 3:
> 
> After block mirror sync finish, add those netfilter and chardev to primary
> VM and secondary VM for COLO ( *Can't* ping to primary VM but those packets
> will be received later )
> 
> 
> Step 4:
> 
> Start migrate primary VM to secondary VM, and primary VM & secondary VM are
> running ( ping to primary VM works and receive those packets on step 3
> status )
> 
> 
> 
> 
> Between Step 3 to Step 4, it will take 10~20 seconds in our environment.
> 
> I could image this issue (delay reply packets) is because of setting COLO
> proxy for temporary status,
> 
> but we thought 10~20 seconds might a little long. (If primary VM is already
> doing some jobs, it might lose the data.)
> 
> 
> Could we reduce those time? or those delay is depends on different VM?

I think you need to set up the netfilter and chardev on the primary at
the start;  the filter contains the state of the TCP connections working
with the VM, so adding it later can't gain that state for existing
connections.

Dave

> 
> Best Regard,
> 
> Daniel Cho.
> 
> 
> 
> Zhang, Chen <chen.zhang@intel.com> 於 2019年11月30日 週六 上午2:04寫道:
> 
> >
> >
> >
> >
> > *From:* Daniel Cho <danielcho@qnap.com>
> > *Sent:* Friday, November 29, 2019 10:43 AM
> > *To:* Zhang, Chen <chen.zhang@intel.com>
> > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com>; lukasstraub2@web.de;
> > qemu-devel@nongnu.org
> > *Subject:* Re: Network connection with COLO VM
> >
> >
> >
> > Hi David,  Zhang,
> >
> >
> >
> > Thanks for replying my question.
> >
> > We know why will occur this issue.
> >
> > As you said, the COLO VM's network needs
> >
> > colo-proxy to control packets, so the guest's
> >
> > interface should set the filter to solve the problem.
> >
> >
> >
> > But we found another question, when we set the
> >
> > fault-tolerance feature to guest (primary VM is running,
> >
> > secondary VM is pausing), the guest's network would not
> >
> > responds any request for a while (in our environment
> >
> > about 20~30 secs) after secondary VM runs.
> >
> >
> >
> > Does it be a normal situation, or a known issue?
> >
> >
> >
> > Our test is creating primary VM for a while, then creating
> >
> > secondary VM to make it with COLO feature.
> >
> >
> >
> > Hi Daniel,
> >
> >
> >
> > Happy to hear you have solved ssh disconnection issue.
> >
> >
> >
> > Do you use Lukas’s patch on this case?
> >
> > I think we need use block mirror to sync the disk to secondary node first,
> > then stop the primary VM and build COLO system.
> >
> > In the stop moment, you need add some netfilter and chardev socket node
> > for COLO, maybe you need re-check this part.
> >
> >
> >
> > Best Regard,
> >
> > Daniel Cho
> >
> >
> >
> > Zhang, Chen <chen.zhang@intel.com> 於 2019年11月28日 週四 上午9:26寫道:
> >
> >
> >
> > > -----Original Message-----
> > > From: Dr. David Alan Gilbert <dgilbert@redhat.com>
> > > Sent: Wednesday, November 27, 2019 6:51 PM
> > > To: Daniel Cho <danielcho@qnap.com>; Zhang, Chen
> > > <chen.zhang@intel.com>; lukasstraub2@web.de
> > > Cc: qemu-devel@nongnu.org
> > > Subject: Re: Network connection with COLO VM
> > >
> > > * Daniel Cho (danielcho@qnap.com) wrote:
> > > > Hello everyone,
> > > >
> > > > Could we ssh to colo VM (means PVM & SVM are starting)?
> > > >
> > >
> > > Lets cc in Zhang Chen and Lukas Straub.
> >
> > Thanks Dave.
> >
> > >
> > > > SSH will connect to colo VM for a while, but it will disconnect with
> > > > error
> > > > *client_loop: send disconnect: Broken pipe*
> > > >
> > > > It seems to colo VM could not keep network session.
> > > >
> > > > Does it be a known issue?
> > >
> > > That sounds like the COLO proxy is getting upset; it's supposed to
> > compare
> > > packets sent by the primary and secondary and only send one to the
> > outside
> > > - you shouldn't be talking directly to the guest, but always via the
> > proxy.  See
> > > docs/colo-proxy.txt
> > >
> >
> > Hi Daniel,
> >
> > I have try ssh to COLO guest with 8 hours, not occurred this issue.
> > Please check your network/qemu configuration.
> > But I found another problem maybe related this issue, if no network
> > communication for a period of time(maybe 10min), the first message send to
> > guest have a chance with delay(maybe 1-5 sec), I will try to fix it when I
> > have time.
> >
> > Thanks
> > Zhang Chen
> >
> > > Dave
> > >
> > > > Best Regard,
> > > > Daniel Cho
> > > --
> > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >
> >
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK



^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Network connection with COLO VM
  2019-12-02  9:58           ` Dr. David Alan Gilbert
@ 2019-12-03  9:08             ` Daniel Cho
  2019-12-03 13:25               ` Dr. David Alan Gilbert
  0 siblings, 1 reply; 11+ messages in thread
From: Daniel Cho @ 2019-12-03  9:08 UTC (permalink / raw)
  To: Dr. David Alan Gilbert; +Cc: Zhang, Chen, lukasstraub2, qemu-devel

[-- Attachment #1: Type: text/plain, Size: 6897 bytes --]

Hi Dave,

We could use the exist interface to add netfilter and chardev, it might not
have the problem you said.

However, the netfilter and chardev on the primary at the start, that means
we could not dynamic set COLO
feature to VM?

We try to change this chardev to prevent primary VM will stuck to wait
secondary VM.

-chardev socket,id=compare1,host=127.0.0.1,port=9004,server,wait \

to

-chardev socket,id=compare1,host=127.0.0.1,port=9004,server,nowait \

But it will make primary VM's network not works. (Can't get ip), until
starting connect with secondary VM.


Otherwise, the primary VM with netfileter / chardev and without netfilter /
chardev , they takes very different
booting time.
Without  netfilter / chardev : about 1 mins
With   netfilter / chardev : about 5 mins
Is this an issue?

Best regards,
Daniel Cho


Dr. David Alan Gilbert <dgilbert@redhat.com> 於 2019年12月2日 週一 下午5:58寫道:

> * Daniel Cho (danielcho@qnap.com) wrote:
> > Hi Zhang,
> >
> > We use qemu-4.1.0 release on this case.
> >
> > I think we need use block mirror to sync the disk to secondary node
> first,
> > then stop the primary VM and build COLO system.
> >
> > In the stop moment, you need add some netfilter and chardev socket node
> for
> > COLO, maybe you need re-check this part.
> >
> >
> > Our test was already follow those step. Maybe I could describe the detail
> > of the test flow and issues.
> >
> >
> > Step 1:
> >
> > Create primary VM without any netfilter and chardev for COLO, and using
> > other host ping primary VM continually.
> >
> >
> > Step 2:
> >
> > Create secondary VM (the same device/drive with primary VM), and do block
> > mirror sync ( ping to primary VM normally )
> >
> >
> > Step 3:
> >
> > After block mirror sync finish, add those netfilter and chardev to
> primary
> > VM and secondary VM for COLO ( *Can't* ping to primary VM but those
> packets
> > will be received later )
> >
> >
> > Step 4:
> >
> > Start migrate primary VM to secondary VM, and primary VM & secondary VM
> are
> > running ( ping to primary VM works and receive those packets on step 3
> > status )
> >
> >
> >
> >
> > Between Step 3 to Step 4, it will take 10~20 seconds in our environment.
> >
> > I could image this issue (delay reply packets) is because of setting COLO
> > proxy for temporary status,
> >
> > but we thought 10~20 seconds might a little long. (If primary VM is
> already
> > doing some jobs, it might lose the data.)
> >
> >
> > Could we reduce those time? or those delay is depends on different VM?
>
> I think you need to set up the netfilter and chardev on the primary at
> the start;  the filter contains the state of the TCP connections working
> with the VM, so adding it later can't gain that state for existing
> connections.
>
> Dave
>
> >
> > Best Regard,
> >
> > Daniel Cho.
> >
> >
> >
> > Zhang, Chen <chen.zhang@intel.com> 於 2019年11月30日 週六 上午2:04寫道:
> >
> > >
> > >
> > >
> > >
> > > *From:* Daniel Cho <danielcho@qnap.com>
> > > *Sent:* Friday, November 29, 2019 10:43 AM
> > > *To:* Zhang, Chen <chen.zhang@intel.com>
> > > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com>;
> lukasstraub2@web.de;
> > > qemu-devel@nongnu.org
> > > *Subject:* Re: Network connection with COLO VM
> > >
> > >
> > >
> > > Hi David,  Zhang,
> > >
> > >
> > >
> > > Thanks for replying my question.
> > >
> > > We know why will occur this issue.
> > >
> > > As you said, the COLO VM's network needs
> > >
> > > colo-proxy to control packets, so the guest's
> > >
> > > interface should set the filter to solve the problem.
> > >
> > >
> > >
> > > But we found another question, when we set the
> > >
> > > fault-tolerance feature to guest (primary VM is running,
> > >
> > > secondary VM is pausing), the guest's network would not
> > >
> > > responds any request for a while (in our environment
> > >
> > > about 20~30 secs) after secondary VM runs.
> > >
> > >
> > >
> > > Does it be a normal situation, or a known issue?
> > >
> > >
> > >
> > > Our test is creating primary VM for a while, then creating
> > >
> > > secondary VM to make it with COLO feature.
> > >
> > >
> > >
> > > Hi Daniel,
> > >
> > >
> > >
> > > Happy to hear you have solved ssh disconnection issue.
> > >
> > >
> > >
> > > Do you use Lukas’s patch on this case?
> > >
> > > I think we need use block mirror to sync the disk to secondary node
> first,
> > > then stop the primary VM and build COLO system.
> > >
> > > In the stop moment, you need add some netfilter and chardev socket node
> > > for COLO, maybe you need re-check this part.
> > >
> > >
> > >
> > > Best Regard,
> > >
> > > Daniel Cho
> > >
> > >
> > >
> > > Zhang, Chen <chen.zhang@intel.com> 於 2019年11月28日 週四 上午9:26寫道:
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Dr. David Alan Gilbert <dgilbert@redhat.com>
> > > > Sent: Wednesday, November 27, 2019 6:51 PM
> > > > To: Daniel Cho <danielcho@qnap.com>; Zhang, Chen
> > > > <chen.zhang@intel.com>; lukasstraub2@web.de
> > > > Cc: qemu-devel@nongnu.org
> > > > Subject: Re: Network connection with COLO VM
> > > >
> > > > * Daniel Cho (danielcho@qnap.com) wrote:
> > > > > Hello everyone,
> > > > >
> > > > > Could we ssh to colo VM (means PVM & SVM are starting)?
> > > > >
> > > >
> > > > Lets cc in Zhang Chen and Lukas Straub.
> > >
> > > Thanks Dave.
> > >
> > > >
> > > > > SSH will connect to colo VM for a while, but it will disconnect
> with
> > > > > error
> > > > > *client_loop: send disconnect: Broken pipe*
> > > > >
> > > > > It seems to colo VM could not keep network session.
> > > > >
> > > > > Does it be a known issue?
> > > >
> > > > That sounds like the COLO proxy is getting upset; it's supposed to
> > > compare
> > > > packets sent by the primary and secondary and only send one to the
> > > outside
> > > > - you shouldn't be talking directly to the guest, but always via the
> > > proxy.  See
> > > > docs/colo-proxy.txt
> > > >
> > >
> > > Hi Daniel,
> > >
> > > I have try ssh to COLO guest with 8 hours, not occurred this issue.
> > > Please check your network/qemu configuration.
> > > But I found another problem maybe related this issue, if no network
> > > communication for a period of time(maybe 10min), the first message
> send to
> > > guest have a chance with delay(maybe 1-5 sec), I will try to fix it
> when I
> > > have time.
> > >
> > > Thanks
> > > Zhang Chen
> > >
> > > > Dave
> > > >
> > > > > Best Regard,
> > > > > Daniel Cho
> > > > --
> > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> > >
> > >
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
>

[-- Attachment #2: Type: text/html, Size: 10589 bytes --]

<div dir="ltr"><div>Hi Dave, </div><div><br></div><div>We could use the exist interface to add netfilter and chardev, it might not have the problem you said.</div><div><br></div><div>However, <font color="#ff0000">the netfilter and chardev on the primary at the start</font>, that means we could not dynamic set COLO</div><div>feature to VM?</div><div><br></div><div>We try to change this chardev to prevent primary VM will stuck to wait secondary VM.</div><div><pre style="font-family:monospace,Courier;padding:1em;border:1px solid rgb(221,221,221);background-color:rgb(249,249,249);line-height:1.3em;font-size:14px"><span style="color:black">-chardev socket,id=compare1,host=127.0.0.1,port=9004,server,</span><font color="#ff0000">wait</font><font color="#000000"> \</font></pre></div><div>to</div><div><pre style="font-family:monospace,Courier;padding:1em;border:1px solid rgb(221,221,221);background-color:rgb(249,249,249);line-height:1.3em;font-size:14px"><span style="color:black">-chardev socket,id=compare1,host=127.0.0.1,port=9004,server,</span><font color="#ff0000">nowait</font><font color="#000000"> \</font></pre></div><div>But it will make primary VM&#39;s network not works. (Can&#39;t get ip), until starting connect with secondary VM.</div><div><br></div><div><br></div><div>Otherwise, the primary VM with netfileter / chardev and without netfilter / chardev , they takes very different </div><div>booting time.</div><div><font color="#ff0000">Without</font> 

netfilter / chardev : about 1 mins</div><div><font color="#ff0000">With</font>   netfilter / chardev : about 5 mins  <br></div><div>Is this an issue?</div><div><br></div><div>Best regards,</div><div>Daniel Cho</div><div><br></div><div><br></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Dr. David Alan Gilbert &lt;<a href="mailto:dgilbert@redhat.com">dgilbert@redhat.com</a>&gt; 於 2019年12月2日 週一 下午5:58寫道:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">* Daniel Cho (<a href="mailto:danielcho@qnap.com" target="_blank">danielcho@qnap.com</a>) wrote:<br>
&gt; Hi Zhang,<br>
&gt; <br>
&gt; We use qemu-4.1.0 release on this case.<br>
&gt; <br>
&gt; I think we need use block mirror to sync the disk to secondary node first,<br>
&gt; then stop the primary VM and build COLO system.<br>
&gt; <br>
&gt; In the stop moment, you need add some netfilter and chardev socket node for<br>
&gt; COLO, maybe you need re-check this part.<br>
&gt; <br>
&gt; <br>
&gt; Our test was already follow those step. Maybe I could describe the detail<br>
&gt; of the test flow and issues.<br>
&gt; <br>
&gt; <br>
&gt; Step 1:<br>
&gt; <br>
&gt; Create primary VM without any netfilter and chardev for COLO, and using<br>
&gt; other host ping primary VM continually.<br>
&gt; <br>
&gt; <br>
&gt; Step 2:<br>
&gt; <br>
&gt; Create secondary VM (the same device/drive with primary VM), and do block<br>
&gt; mirror sync ( ping to primary VM normally )<br>
&gt; <br>
&gt; <br>
&gt; Step 3:<br>
&gt; <br>
&gt; After block mirror sync finish, add those netfilter and chardev to primary<br>
&gt; VM and secondary VM for COLO ( *Can&#39;t* ping to primary VM but those packets<br>
&gt; will be received later )<br>
&gt; <br>
&gt; <br>
&gt; Step 4:<br>
&gt; <br>
&gt; Start migrate primary VM to secondary VM, and primary VM &amp; secondary VM are<br>
&gt; running ( ping to primary VM works and receive those packets on step 3<br>
&gt; status )<br>
&gt; <br>
&gt; <br>
&gt; <br>
&gt; <br>
&gt; Between Step 3 to Step 4, it will take 10~20 seconds in our environment.<br>
&gt; <br>
&gt; I could image this issue (delay reply packets) is because of setting COLO<br>
&gt; proxy for temporary status,<br>
&gt; <br>
&gt; but we thought 10~20 seconds might a little long. (If primary VM is already<br>
&gt; doing some jobs, it might lose the data.)<br>
&gt; <br>
&gt; <br>
&gt; Could we reduce those time? or those delay is depends on different VM?<br>
<br>
I think you need to set up the netfilter and chardev on the primary at<br>
the start;  the filter contains the state of the TCP connections working<br>
with the VM, so adding it later can&#39;t gain that state for existing<br>
connections.<br>
<br>
Dave<br>
<br>
&gt; <br>
&gt; Best Regard,<br>
&gt; <br>
&gt; Daniel Cho.<br>
&gt; <br>
&gt; <br>
&gt; <br>
&gt; Zhang, Chen &lt;<a href="mailto:chen.zhang@intel.com" target="_blank">chen.zhang@intel.com</a>&gt; 於 2019年11月30日 週六 上午2:04寫道:<br>
&gt; <br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt; *From:* Daniel Cho &lt;<a href="mailto:danielcho@qnap.com" target="_blank">danielcho@qnap.com</a>&gt;<br>
&gt; &gt; *Sent:* Friday, November 29, 2019 10:43 AM<br>
&gt; &gt; *To:* Zhang, Chen &lt;<a href="mailto:chen.zhang@intel.com" target="_blank">chen.zhang@intel.com</a>&gt;<br>
&gt; &gt; *Cc:* Dr. David Alan Gilbert &lt;<a href="mailto:dgilbert@redhat.com" target="_blank">dgilbert@redhat.com</a>&gt;; <a href="mailto:lukasstraub2@web.de" target="_blank">lukasstraub2@web.de</a>;<br>
&gt; &gt; <a href="mailto:qemu-devel@nongnu.org" target="_blank">qemu-devel@nongnu.org</a><br>
&gt; &gt; *Subject:* Re: Network connection with COLO VM<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt; Hi David,  Zhang,<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt; Thanks for replying my question.<br>
&gt; &gt;<br>
&gt; &gt; We know why will occur this issue.<br>
&gt; &gt;<br>
&gt; &gt; As you said, the COLO VM&#39;s network needs<br>
&gt; &gt;<br>
&gt; &gt; colo-proxy to control packets, so the guest&#39;s<br>
&gt; &gt;<br>
&gt; &gt; interface should set the filter to solve the problem.<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt; But we found another question, when we set the<br>
&gt; &gt;<br>
&gt; &gt; fault-tolerance feature to guest (primary VM is running,<br>
&gt; &gt;<br>
&gt; &gt; secondary VM is pausing), the guest&#39;s network would not<br>
&gt; &gt;<br>
&gt; &gt; responds any request for a while (in our environment<br>
&gt; &gt;<br>
&gt; &gt; about 20~30 secs) after secondary VM runs.<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt; Does it be a normal situation, or a known issue?<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt; Our test is creating primary VM for a while, then creating<br>
&gt; &gt;<br>
&gt; &gt; secondary VM to make it with COLO feature.<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt; Hi Daniel,<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt; Happy to hear you have solved ssh disconnection issue.<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt; Do you use Lukas’s patch on this case?<br>
&gt; &gt;<br>
&gt; &gt; I think we need use block mirror to sync the disk to secondary node first,<br>
&gt; &gt; then stop the primary VM and build COLO system.<br>
&gt; &gt;<br>
&gt; &gt; In the stop moment, you need add some netfilter and chardev socket node<br>
&gt; &gt; for COLO, maybe you need re-check this part.<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt; Best Regard,<br>
&gt; &gt;<br>
&gt; &gt; Daniel Cho<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt; Zhang, Chen &lt;<a href="mailto:chen.zhang@intel.com" target="_blank">chen.zhang@intel.com</a>&gt; 於 2019年11月28日 週四 上午9:26寫道:<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt; &gt; -----Original Message-----<br>
&gt; &gt; &gt; From: Dr. David Alan Gilbert &lt;<a href="mailto:dgilbert@redhat.com" target="_blank">dgilbert@redhat.com</a>&gt;<br>
&gt; &gt; &gt; Sent: Wednesday, November 27, 2019 6:51 PM<br>
&gt; &gt; &gt; To: Daniel Cho &lt;<a href="mailto:danielcho@qnap.com" target="_blank">danielcho@qnap.com</a>&gt;; Zhang, Chen<br>
&gt; &gt; &gt; &lt;<a href="mailto:chen.zhang@intel.com" target="_blank">chen.zhang@intel.com</a>&gt;; <a href="mailto:lukasstraub2@web.de" target="_blank">lukasstraub2@web.de</a><br>
&gt; &gt; &gt; Cc: <a href="mailto:qemu-devel@nongnu.org" target="_blank">qemu-devel@nongnu.org</a><br>
&gt; &gt; &gt; Subject: Re: Network connection with COLO VM<br>
&gt; &gt; &gt;<br>
&gt; &gt; &gt; * Daniel Cho (<a href="mailto:danielcho@qnap.com" target="_blank">danielcho@qnap.com</a>) wrote:<br>
&gt; &gt; &gt; &gt; Hello everyone,<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; &gt; Could we ssh to colo VM (means PVM &amp; SVM are starting)?<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt;<br>
&gt; &gt; &gt; Lets cc in Zhang Chen and Lukas Straub.<br>
&gt; &gt;<br>
&gt; &gt; Thanks Dave.<br>
&gt; &gt;<br>
&gt; &gt; &gt;<br>
&gt; &gt; &gt; &gt; SSH will connect to colo VM for a while, but it will disconnect with<br>
&gt; &gt; &gt; &gt; error<br>
&gt; &gt; &gt; &gt; *client_loop: send disconnect: Broken pipe*<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; &gt; It seems to colo VM could not keep network session.<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; &gt; Does it be a known issue?<br>
&gt; &gt; &gt;<br>
&gt; &gt; &gt; That sounds like the COLO proxy is getting upset; it&#39;s supposed to<br>
&gt; &gt; compare<br>
&gt; &gt; &gt; packets sent by the primary and secondary and only send one to the<br>
&gt; &gt; outside<br>
&gt; &gt; &gt; - you shouldn&#39;t be talking directly to the guest, but always via the<br>
&gt; &gt; proxy.  See<br>
&gt; &gt; &gt; docs/colo-proxy.txt<br>
&gt; &gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt; Hi Daniel,<br>
&gt; &gt;<br>
&gt; &gt; I have try ssh to COLO guest with 8 hours, not occurred this issue.<br>
&gt; &gt; Please check your network/qemu configuration.<br>
&gt; &gt; But I found another problem maybe related this issue, if no network<br>
&gt; &gt; communication for a period of time(maybe 10min), the first message send to<br>
&gt; &gt; guest have a chance with delay(maybe 1-5 sec), I will try to fix it when I<br>
&gt; &gt; have time.<br>
&gt; &gt;<br>
&gt; &gt; Thanks<br>
&gt; &gt; Zhang Chen<br>
&gt; &gt;<br>
&gt; &gt; &gt; Dave<br>
&gt; &gt; &gt;<br>
&gt; &gt; &gt; &gt; Best Regard,<br>
&gt; &gt; &gt; &gt; Daniel Cho<br>
&gt; &gt; &gt; --<br>
&gt; &gt; &gt; Dr. David Alan Gilbert / <a href="mailto:dgilbert@redhat.com" target="_blank">dgilbert@redhat.com</a> / Manchester, UK<br>
&gt; &gt;<br>
&gt; &gt;<br>
--<br>
Dr. David Alan Gilbert / <a href="mailto:dgilbert@redhat.com" target="_blank">dgilbert@redhat.com</a> / Manchester, UK<br>
<br>
</blockquote></div></div>

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Network connection with COLO VM
  2019-12-03  9:08             ` Daniel Cho
@ 2019-12-03 13:25               ` Dr. David Alan Gilbert
  2019-12-04  8:32                 ` Zhang, Chen
  0 siblings, 1 reply; 11+ messages in thread
From: Dr. David Alan Gilbert @ 2019-12-03 13:25 UTC (permalink / raw)
  To: Daniel Cho; +Cc: Zhang, Chen, lukasstraub2, qemu-devel

* Daniel Cho (danielcho@qnap.com) wrote:
> Hi Dave,
> 
> We could use the exist interface to add netfilter and chardev, it might not
> have the problem you said.
> 
> However, the netfilter and chardev on the primary at the start, that means
> we could not dynamic set COLO
> feature to VM?

I wasn't expecting that to be possible - I'd expect you to be able
to start in a state that looks the same as a primary+failed secondary;
but I'm not sure.

> We try to change this chardev to prevent primary VM will stuck to wait
> secondary VM.
> 
> -chardev socket,id=compare1,host=127.0.0.1,port=9004,server,wait \
> 
> to
> 
> -chardev socket,id=compare1,host=127.0.0.1,port=9004,server,nowait \
> 
> But it will make primary VM's network not works. (Can't get ip), until
> starting connect with secondary VM.

I'm not sure of the answer to this; I've not tried doing it - I'm not
sure anyone has!
But, the colo components do track the state of tcp connections, so I'm
expecting that they have to already exist to have the state of those
connections available for when you start the secondary.


> Otherwise, the primary VM with netfileter / chardev and without netfilter /
> chardev , they takes very different
> booting time.
> Without  netfilter / chardev : about 1 mins
> With   netfilter / chardev : about 5 mins
> Is this an issue?

that sounds like it needs investigating.

Dave

> Best regards,
> Daniel Cho
> 
> 
> Dr. David Alan Gilbert <dgilbert@redhat.com> 於 2019年12月2日 週一 下午5:58寫道:
> 
> > * Daniel Cho (danielcho@qnap.com) wrote:
> > > Hi Zhang,
> > >
> > > We use qemu-4.1.0 release on this case.
> > >
> > > I think we need use block mirror to sync the disk to secondary node
> > first,
> > > then stop the primary VM and build COLO system.
> > >
> > > In the stop moment, you need add some netfilter and chardev socket node
> > for
> > > COLO, maybe you need re-check this part.
> > >
> > >
> > > Our test was already follow those step. Maybe I could describe the detail
> > > of the test flow and issues.
> > >
> > >
> > > Step 1:
> > >
> > > Create primary VM without any netfilter and chardev for COLO, and using
> > > other host ping primary VM continually.
> > >
> > >
> > > Step 2:
> > >
> > > Create secondary VM (the same device/drive with primary VM), and do block
> > > mirror sync ( ping to primary VM normally )
> > >
> > >
> > > Step 3:
> > >
> > > After block mirror sync finish, add those netfilter and chardev to
> > primary
> > > VM and secondary VM for COLO ( *Can't* ping to primary VM but those
> > packets
> > > will be received later )
> > >
> > >
> > > Step 4:
> > >
> > > Start migrate primary VM to secondary VM, and primary VM & secondary VM
> > are
> > > running ( ping to primary VM works and receive those packets on step 3
> > > status )
> > >
> > >
> > >
> > >
> > > Between Step 3 to Step 4, it will take 10~20 seconds in our environment.
> > >
> > > I could image this issue (delay reply packets) is because of setting COLO
> > > proxy for temporary status,
> > >
> > > but we thought 10~20 seconds might a little long. (If primary VM is
> > already
> > > doing some jobs, it might lose the data.)
> > >
> > >
> > > Could we reduce those time? or those delay is depends on different VM?
> >
> > I think you need to set up the netfilter and chardev on the primary at
> > the start;  the filter contains the state of the TCP connections working
> > with the VM, so adding it later can't gain that state for existing
> > connections.
> >
> > Dave
> >
> > >
> > > Best Regard,
> > >
> > > Daniel Cho.
> > >
> > >
> > >
> > > Zhang, Chen <chen.zhang@intel.com> 於 2019年11月30日 週六 上午2:04寫道:
> > >
> > > >
> > > >
> > > >
> > > >
> > > > *From:* Daniel Cho <danielcho@qnap.com>
> > > > *Sent:* Friday, November 29, 2019 10:43 AM
> > > > *To:* Zhang, Chen <chen.zhang@intel.com>
> > > > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com>;
> > lukasstraub2@web.de;
> > > > qemu-devel@nongnu.org
> > > > *Subject:* Re: Network connection with COLO VM
> > > >
> > > >
> > > >
> > > > Hi David,  Zhang,
> > > >
> > > >
> > > >
> > > > Thanks for replying my question.
> > > >
> > > > We know why will occur this issue.
> > > >
> > > > As you said, the COLO VM's network needs
> > > >
> > > > colo-proxy to control packets, so the guest's
> > > >
> > > > interface should set the filter to solve the problem.
> > > >
> > > >
> > > >
> > > > But we found another question, when we set the
> > > >
> > > > fault-tolerance feature to guest (primary VM is running,
> > > >
> > > > secondary VM is pausing), the guest's network would not
> > > >
> > > > responds any request for a while (in our environment
> > > >
> > > > about 20~30 secs) after secondary VM runs.
> > > >
> > > >
> > > >
> > > > Does it be a normal situation, or a known issue?
> > > >
> > > >
> > > >
> > > > Our test is creating primary VM for a while, then creating
> > > >
> > > > secondary VM to make it with COLO feature.
> > > >
> > > >
> > > >
> > > > Hi Daniel,
> > > >
> > > >
> > > >
> > > > Happy to hear you have solved ssh disconnection issue.
> > > >
> > > >
> > > >
> > > > Do you use Lukas’s patch on this case?
> > > >
> > > > I think we need use block mirror to sync the disk to secondary node
> > first,
> > > > then stop the primary VM and build COLO system.
> > > >
> > > > In the stop moment, you need add some netfilter and chardev socket node
> > > > for COLO, maybe you need re-check this part.
> > > >
> > > >
> > > >
> > > > Best Regard,
> > > >
> > > > Daniel Cho
> > > >
> > > >
> > > >
> > > > Zhang, Chen <chen.zhang@intel.com> 於 2019年11月28日 週四 上午9:26寫道:
> > > >
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Dr. David Alan Gilbert <dgilbert@redhat.com>
> > > > > Sent: Wednesday, November 27, 2019 6:51 PM
> > > > > To: Daniel Cho <danielcho@qnap.com>; Zhang, Chen
> > > > > <chen.zhang@intel.com>; lukasstraub2@web.de
> > > > > Cc: qemu-devel@nongnu.org
> > > > > Subject: Re: Network connection with COLO VM
> > > > >
> > > > > * Daniel Cho (danielcho@qnap.com) wrote:
> > > > > > Hello everyone,
> > > > > >
> > > > > > Could we ssh to colo VM (means PVM & SVM are starting)?
> > > > > >
> > > > >
> > > > > Lets cc in Zhang Chen and Lukas Straub.
> > > >
> > > > Thanks Dave.
> > > >
> > > > >
> > > > > > SSH will connect to colo VM for a while, but it will disconnect
> > with
> > > > > > error
> > > > > > *client_loop: send disconnect: Broken pipe*
> > > > > >
> > > > > > It seems to colo VM could not keep network session.
> > > > > >
> > > > > > Does it be a known issue?
> > > > >
> > > > > That sounds like the COLO proxy is getting upset; it's supposed to
> > > > compare
> > > > > packets sent by the primary and secondary and only send one to the
> > > > outside
> > > > > - you shouldn't be talking directly to the guest, but always via the
> > > > proxy.  See
> > > > > docs/colo-proxy.txt
> > > > >
> > > >
> > > > Hi Daniel,
> > > >
> > > > I have try ssh to COLO guest with 8 hours, not occurred this issue.
> > > > Please check your network/qemu configuration.
> > > > But I found another problem maybe related this issue, if no network
> > > > communication for a period of time(maybe 10min), the first message
> > send to
> > > > guest have a chance with delay(maybe 1-5 sec), I will try to fix it
> > when I
> > > > have time.
> > > >
> > > > Thanks
> > > > Zhang Chen
> > > >
> > > > > Dave
> > > > >
> > > > > > Best Regard,
> > > > > > Daniel Cho
> > > > > --
> > > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> > > >
> > > >
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >
> >
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK



^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Network connection with COLO VM
  2019-12-03 13:25               ` Dr. David Alan Gilbert
@ 2019-12-04  8:32                 ` Zhang, Chen
  2019-12-06  6:31                   ` Daniel Cho
  0 siblings, 1 reply; 11+ messages in thread
From: Zhang, Chen @ 2019-12-04  8:32 UTC (permalink / raw)
  To: Dr. David Alan Gilbert, Daniel Cho; +Cc: lukasstraub2, qemu-devel


On 12/3/2019 9:25 PM, Dr. David Alan Gilbert wrote:
> * Daniel Cho (danielcho@qnap.com) wrote:
>> Hi Dave,
>>
>> We could use the exist interface to add netfilter and chardev, it might not
>> have the problem you said.
>>
>> However, the netfilter and chardev on the primary at the start, that means
>> we could not dynamic set COLO
>> feature to VM?
> I wasn't expecting that to be possible - I'd expect you to be able
> to start in a state that looks the same as a primary+failed secondary;
> but I'm not sure.

Current COLO (with Lukas's patch) can support dynamic set COLO system.

This status is same like the secondary has triggered failover, the 
primary node need to find new secondary

node to combine new COLO system.


>> We try to change this chardev to prevent primary VM will stuck to wait
>> secondary VM.
>>
>> -chardev socket,id=compare1,host=127.0.0.1,port=9004,server,wait \
>>
>> to
>>
>> -chardev socket,id=compare1,host=127.0.0.1,port=9004,server,nowait \
>>
>> But it will make primary VM's network not works. (Can't get ip), until
>> starting connect with secondary VM.

I think you need to check the port 9004 if already connect to the pair node.

> I'm not sure of the answer to this; I've not tried doing it - I'm not
> sure anyone has!
> But, the colo components do track the state of tcp connections, so I'm
> expecting that they have to already exist to have the state of those
> connections available for when you start the secondary.

Yes, you are right.

For this status, we don't need to sync the state of tcp connections, 
because after failover

(or just solo COLO primary node), we have empty all the tcp connections 
state in COLO module.

In the processing of build new COLO pair, we will sync all the VM state 
to secondary node and re-build

new track things in COLO module.


>
>
>> Otherwise, the primary VM with netfileter / chardev and without netfilter /
>> chardev , they takes very different
>> booting time.
>> Without  netfilter / chardev : about 1 mins
>> With   netfilter / chardev : about 5 mins
>> Is this an issue?
> that sounds like it needs investigating.
>
> Dave

Yes, In previous COLO use cases, we need make primary node and secondary 
node boot in the same time.

I did't expect such a big difference on netfilter/chardev.

I think you can try without netfilter/chardev, after VM boot, re-build 
the netfilter/chardev related work with chardev server nowait.


Thanks

Zhang Chen

>
>> Best regards,
>> Daniel Cho
>>
>>
>> Dr. David Alan Gilbert <dgilbert@redhat.com> 於 2019年12月2日 週一 下午5:58寫道:
>>
>>> * Daniel Cho (danielcho@qnap.com) wrote:
>>>> Hi Zhang,
>>>>
>>>> We use qemu-4.1.0 release on this case.
>>>>
>>>> I think we need use block mirror to sync the disk to secondary node
>>> first,
>>>> then stop the primary VM and build COLO system.
>>>>
>>>> In the stop moment, you need add some netfilter and chardev socket node
>>> for
>>>> COLO, maybe you need re-check this part.
>>>>
>>>>
>>>> Our test was already follow those step. Maybe I could describe the detail
>>>> of the test flow and issues.
>>>>
>>>>
>>>> Step 1:
>>>>
>>>> Create primary VM without any netfilter and chardev for COLO, and using
>>>> other host ping primary VM continually.
>>>>
>>>>
>>>> Step 2:
>>>>
>>>> Create secondary VM (the same device/drive with primary VM), and do block
>>>> mirror sync ( ping to primary VM normally )
>>>>
>>>>
>>>> Step 3:
>>>>
>>>> After block mirror sync finish, add those netfilter and chardev to
>>> primary
>>>> VM and secondary VM for COLO ( *Can't* ping to primary VM but those
>>> packets
>>>> will be received later )
>>>>
>>>>
>>>> Step 4:
>>>>
>>>> Start migrate primary VM to secondary VM, and primary VM & secondary VM
>>> are
>>>> running ( ping to primary VM works and receive those packets on step 3
>>>> status )
>>>>
>>>>
>>>>
>>>>
>>>> Between Step 3 to Step 4, it will take 10~20 seconds in our environment.
>>>>
>>>> I could image this issue (delay reply packets) is because of setting COLO
>>>> proxy for temporary status,
>>>>
>>>> but we thought 10~20 seconds might a little long. (If primary VM is
>>> already
>>>> doing some jobs, it might lose the data.)
>>>>
>>>>
>>>> Could we reduce those time? or those delay is depends on different VM?
>>> I think you need to set up the netfilter and chardev on the primary at
>>> the start;  the filter contains the state of the TCP connections working
>>> with the VM, so adding it later can't gain that state for existing
>>> connections.
>>>
>>> Dave
>>>
>>>> Best Regard,
>>>>
>>>> Daniel Cho.
>>>>
>>>>
>>>>
>>>> Zhang, Chen <chen.zhang@intel.com> 於 2019年11月30日 週六 上午2:04寫道:
>>>>
>>>>>
>>>>>
>>>>>
>>>>> *From:* Daniel Cho <danielcho@qnap.com>
>>>>> *Sent:* Friday, November 29, 2019 10:43 AM
>>>>> *To:* Zhang, Chen <chen.zhang@intel.com>
>>>>> *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com>;
>>> lukasstraub2@web.de;
>>>>> qemu-devel@nongnu.org
>>>>> *Subject:* Re: Network connection with COLO VM
>>>>>
>>>>>
>>>>>
>>>>> Hi David,  Zhang,
>>>>>
>>>>>
>>>>>
>>>>> Thanks for replying my question.
>>>>>
>>>>> We know why will occur this issue.
>>>>>
>>>>> As you said, the COLO VM's network needs
>>>>>
>>>>> colo-proxy to control packets, so the guest's
>>>>>
>>>>> interface should set the filter to solve the problem.
>>>>>
>>>>>
>>>>>
>>>>> But we found another question, when we set the
>>>>>
>>>>> fault-tolerance feature to guest (primary VM is running,
>>>>>
>>>>> secondary VM is pausing), the guest's network would not
>>>>>
>>>>> responds any request for a while (in our environment
>>>>>
>>>>> about 20~30 secs) after secondary VM runs.
>>>>>
>>>>>
>>>>>
>>>>> Does it be a normal situation, or a known issue?
>>>>>
>>>>>
>>>>>
>>>>> Our test is creating primary VM for a while, then creating
>>>>>
>>>>> secondary VM to make it with COLO feature.
>>>>>
>>>>>
>>>>>
>>>>> Hi Daniel,
>>>>>
>>>>>
>>>>>
>>>>> Happy to hear you have solved ssh disconnection issue.
>>>>>
>>>>>
>>>>>
>>>>> Do you use Lukas’s patch on this case?
>>>>>
>>>>> I think we need use block mirror to sync the disk to secondary node
>>> first,
>>>>> then stop the primary VM and build COLO system.
>>>>>
>>>>> In the stop moment, you need add some netfilter and chardev socket node
>>>>> for COLO, maybe you need re-check this part.
>>>>>
>>>>>
>>>>>
>>>>> Best Regard,
>>>>>
>>>>> Daniel Cho
>>>>>
>>>>>
>>>>>
>>>>> Zhang, Chen <chen.zhang@intel.com> 於 2019年11月28日 週四 上午9:26寫道:
>>>>>
>>>>>
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Dr. David Alan Gilbert <dgilbert@redhat.com>
>>>>>> Sent: Wednesday, November 27, 2019 6:51 PM
>>>>>> To: Daniel Cho <danielcho@qnap.com>; Zhang, Chen
>>>>>> <chen.zhang@intel.com>; lukasstraub2@web.de
>>>>>> Cc: qemu-devel@nongnu.org
>>>>>> Subject: Re: Network connection with COLO VM
>>>>>>
>>>>>> * Daniel Cho (danielcho@qnap.com) wrote:
>>>>>>> Hello everyone,
>>>>>>>
>>>>>>> Could we ssh to colo VM (means PVM & SVM are starting)?
>>>>>>>
>>>>>> Lets cc in Zhang Chen and Lukas Straub.
>>>>> Thanks Dave.
>>>>>
>>>>>>> SSH will connect to colo VM for a while, but it will disconnect
>>> with
>>>>>>> error
>>>>>>> *client_loop: send disconnect: Broken pipe*
>>>>>>>
>>>>>>> It seems to colo VM could not keep network session.
>>>>>>>
>>>>>>> Does it be a known issue?
>>>>>> That sounds like the COLO proxy is getting upset; it's supposed to
>>>>> compare
>>>>>> packets sent by the primary and secondary and only send one to the
>>>>> outside
>>>>>> - you shouldn't be talking directly to the guest, but always via the
>>>>> proxy.  See
>>>>>> docs/colo-proxy.txt
>>>>>>
>>>>> Hi Daniel,
>>>>>
>>>>> I have try ssh to COLO guest with 8 hours, not occurred this issue.
>>>>> Please check your network/qemu configuration.
>>>>> But I found another problem maybe related this issue, if no network
>>>>> communication for a period of time(maybe 10min), the first message
>>> send to
>>>>> guest have a chance with delay(maybe 1-5 sec), I will try to fix it
>>> when I
>>>>> have time.
>>>>>
>>>>> Thanks
>>>>> Zhang Chen
>>>>>
>>>>>> Dave
>>>>>>
>>>>>>> Best Regard,
>>>>>>> Daniel Cho
>>>>>> --
>>>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>>>
>>> --
>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>
>>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Network connection with COLO VM
  2019-12-04  8:32                 ` Zhang, Chen
@ 2019-12-06  6:31                   ` Daniel Cho
  0 siblings, 0 replies; 11+ messages in thread
From: Daniel Cho @ 2019-12-06  6:31 UTC (permalink / raw)
  To: Zhang, Chen; +Cc: lukasstraub2, Dr. David Alan Gilbert, qemu-devel

[-- Attachment #1: Type: text/plain, Size: 9496 bytes --]

Hi Dave,  Zhang,

Thanks for your help. I will try your recommendations.

Best Regard,
Daniel Cho

Zhang, Chen <chen.zhang@intel.com> 於 2019年12月4日 週三 下午4:32寫道:

>
> On 12/3/2019 9:25 PM, Dr. David Alan Gilbert wrote:
> > * Daniel Cho (danielcho@qnap.com) wrote:
> >> Hi Dave,
> >>
> >> We could use the exist interface to add netfilter and chardev, it might
> not
> >> have the problem you said.
> >>
> >> However, the netfilter and chardev on the primary at the start, that
> means
> >> we could not dynamic set COLO
> >> feature to VM?
> > I wasn't expecting that to be possible - I'd expect you to be able
> > to start in a state that looks the same as a primary+failed secondary;
> > but I'm not sure.
>
> Current COLO (with Lukas's patch) can support dynamic set COLO system.
>
> This status is same like the secondary has triggered failover, the
> primary node need to find new secondary
>
> node to combine new COLO system.
>
>
> >> We try to change this chardev to prevent primary VM will stuck to wait
> >> secondary VM.
> >>
> >> -chardev socket,id=compare1,host=127.0.0.1,port=9004,server,wait \
> >>
> >> to
> >>
> >> -chardev socket,id=compare1,host=127.0.0.1,port=9004,server,nowait \
> >>
> >> But it will make primary VM's network not works. (Can't get ip), until
> >> starting connect with secondary VM.
>
> I think you need to check the port 9004 if already connect to the pair
> node.
>
> > I'm not sure of the answer to this; I've not tried doing it - I'm not
> > sure anyone has!
> > But, the colo components do track the state of tcp connections, so I'm
> > expecting that they have to already exist to have the state of those
> > connections available for when you start the secondary.
>
> Yes, you are right.
>
> For this status, we don't need to sync the state of tcp connections,
> because after failover
>
> (or just solo COLO primary node), we have empty all the tcp connections
> state in COLO module.
>
> In the processing of build new COLO pair, we will sync all the VM state
> to secondary node and re-build
>
> new track things in COLO module.
>
>
> >
> >
> >> Otherwise, the primary VM with netfileter / chardev and without
> netfilter /
> >> chardev , they takes very different
> >> booting time.
> >> Without  netfilter / chardev : about 1 mins
> >> With   netfilter / chardev : about 5 mins
> >> Is this an issue?
> > that sounds like it needs investigating.
> >
> > Dave
>
> Yes, In previous COLO use cases, we need make primary node and secondary
> node boot in the same time.
>
> I did't expect such a big difference on netfilter/chardev.
>
> I think you can try without netfilter/chardev, after VM boot, re-build
> the netfilter/chardev related work with chardev server nowait.
>
>
> Thanks
>
> Zhang Chen
>
> >
> >> Best regards,
> >> Daniel Cho
> >>
> >>
> >> Dr. David Alan Gilbert <dgilbert@redhat.com> 於 2019年12月2日 週一 下午5:58寫道:
> >>
> >>> * Daniel Cho (danielcho@qnap.com) wrote:
> >>>> Hi Zhang,
> >>>>
> >>>> We use qemu-4.1.0 release on this case.
> >>>>
> >>>> I think we need use block mirror to sync the disk to secondary node
> >>> first,
> >>>> then stop the primary VM and build COLO system.
> >>>>
> >>>> In the stop moment, you need add some netfilter and chardev socket
> node
> >>> for
> >>>> COLO, maybe you need re-check this part.
> >>>>
> >>>>
> >>>> Our test was already follow those step. Maybe I could describe the
> detail
> >>>> of the test flow and issues.
> >>>>
> >>>>
> >>>> Step 1:
> >>>>
> >>>> Create primary VM without any netfilter and chardev for COLO, and
> using
> >>>> other host ping primary VM continually.
> >>>>
> >>>>
> >>>> Step 2:
> >>>>
> >>>> Create secondary VM (the same device/drive with primary VM), and do
> block
> >>>> mirror sync ( ping to primary VM normally )
> >>>>
> >>>>
> >>>> Step 3:
> >>>>
> >>>> After block mirror sync finish, add those netfilter and chardev to
> >>> primary
> >>>> VM and secondary VM for COLO ( *Can't* ping to primary VM but those
> >>> packets
> >>>> will be received later )
> >>>>
> >>>>
> >>>> Step 4:
> >>>>
> >>>> Start migrate primary VM to secondary VM, and primary VM & secondary
> VM
> >>> are
> >>>> running ( ping to primary VM works and receive those packets on step 3
> >>>> status )
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Between Step 3 to Step 4, it will take 10~20 seconds in our
> environment.
> >>>>
> >>>> I could image this issue (delay reply packets) is because of setting
> COLO
> >>>> proxy for temporary status,
> >>>>
> >>>> but we thought 10~20 seconds might a little long. (If primary VM is
> >>> already
> >>>> doing some jobs, it might lose the data.)
> >>>>
> >>>>
> >>>> Could we reduce those time? or those delay is depends on different VM?
> >>> I think you need to set up the netfilter and chardev on the primary at
> >>> the start;  the filter contains the state of the TCP connections
> working
> >>> with the VM, so adding it later can't gain that state for existing
> >>> connections.
> >>>
> >>> Dave
> >>>
> >>>> Best Regard,
> >>>>
> >>>> Daniel Cho.
> >>>>
> >>>>
> >>>>
> >>>> Zhang, Chen <chen.zhang@intel.com> 於 2019年11月30日 週六 上午2:04寫道:
> >>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> *From:* Daniel Cho <danielcho@qnap.com>
> >>>>> *Sent:* Friday, November 29, 2019 10:43 AM
> >>>>> *To:* Zhang, Chen <chen.zhang@intel.com>
> >>>>> *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com>;
> >>> lukasstraub2@web.de;
> >>>>> qemu-devel@nongnu.org
> >>>>> *Subject:* Re: Network connection with COLO VM
> >>>>>
> >>>>>
> >>>>>
> >>>>> Hi David,  Zhang,
> >>>>>
> >>>>>
> >>>>>
> >>>>> Thanks for replying my question.
> >>>>>
> >>>>> We know why will occur this issue.
> >>>>>
> >>>>> As you said, the COLO VM's network needs
> >>>>>
> >>>>> colo-proxy to control packets, so the guest's
> >>>>>
> >>>>> interface should set the filter to solve the problem.
> >>>>>
> >>>>>
> >>>>>
> >>>>> But we found another question, when we set the
> >>>>>
> >>>>> fault-tolerance feature to guest (primary VM is running,
> >>>>>
> >>>>> secondary VM is pausing), the guest's network would not
> >>>>>
> >>>>> responds any request for a while (in our environment
> >>>>>
> >>>>> about 20~30 secs) after secondary VM runs.
> >>>>>
> >>>>>
> >>>>>
> >>>>> Does it be a normal situation, or a known issue?
> >>>>>
> >>>>>
> >>>>>
> >>>>> Our test is creating primary VM for a while, then creating
> >>>>>
> >>>>> secondary VM to make it with COLO feature.
> >>>>>
> >>>>>
> >>>>>
> >>>>> Hi Daniel,
> >>>>>
> >>>>>
> >>>>>
> >>>>> Happy to hear you have solved ssh disconnection issue.
> >>>>>
> >>>>>
> >>>>>
> >>>>> Do you use Lukas’s patch on this case?
> >>>>>
> >>>>> I think we need use block mirror to sync the disk to secondary node
> >>> first,
> >>>>> then stop the primary VM and build COLO system.
> >>>>>
> >>>>> In the stop moment, you need add some netfilter and chardev socket
> node
> >>>>> for COLO, maybe you need re-check this part.
> >>>>>
> >>>>>
> >>>>>
> >>>>> Best Regard,
> >>>>>
> >>>>> Daniel Cho
> >>>>>
> >>>>>
> >>>>>
> >>>>> Zhang, Chen <chen.zhang@intel.com> 於 2019年11月28日 週四 上午9:26寫道:
> >>>>>
> >>>>>
> >>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Dr. David Alan Gilbert <dgilbert@redhat.com>
> >>>>>> Sent: Wednesday, November 27, 2019 6:51 PM
> >>>>>> To: Daniel Cho <danielcho@qnap.com>; Zhang, Chen
> >>>>>> <chen.zhang@intel.com>; lukasstraub2@web.de
> >>>>>> Cc: qemu-devel@nongnu.org
> >>>>>> Subject: Re: Network connection with COLO VM
> >>>>>>
> >>>>>> * Daniel Cho (danielcho@qnap.com) wrote:
> >>>>>>> Hello everyone,
> >>>>>>>
> >>>>>>> Could we ssh to colo VM (means PVM & SVM are starting)?
> >>>>>>>
> >>>>>> Lets cc in Zhang Chen and Lukas Straub.
> >>>>> Thanks Dave.
> >>>>>
> >>>>>>> SSH will connect to colo VM for a while, but it will disconnect
> >>> with
> >>>>>>> error
> >>>>>>> *client_loop: send disconnect: Broken pipe*
> >>>>>>>
> >>>>>>> It seems to colo VM could not keep network session.
> >>>>>>>
> >>>>>>> Does it be a known issue?
> >>>>>> That sounds like the COLO proxy is getting upset; it's supposed to
> >>>>> compare
> >>>>>> packets sent by the primary and secondary and only send one to the
> >>>>> outside
> >>>>>> - you shouldn't be talking directly to the guest, but always via the
> >>>>> proxy.  See
> >>>>>> docs/colo-proxy.txt
> >>>>>>
> >>>>> Hi Daniel,
> >>>>>
> >>>>> I have try ssh to COLO guest with 8 hours, not occurred this issue.
> >>>>> Please check your network/qemu configuration.
> >>>>> But I found another problem maybe related this issue, if no network
> >>>>> communication for a period of time(maybe 10min), the first message
> >>> send to
> >>>>> guest have a chance with delay(maybe 1-5 sec), I will try to fix it
> >>> when I
> >>>>> have time.
> >>>>>
> >>>>> Thanks
> >>>>> Zhang Chen
> >>>>>
> >>>>>> Dave
> >>>>>>
> >>>>>>> Best Regard,
> >>>>>>> Daniel Cho
> >>>>>> --
> >>>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >>>>>
> >>> --
> >>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >>>
> >>>
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >
>

[-- Attachment #2: Type: text/html, Size: 14888 bytes --]

<div dir="ltr">Hi Dave,  Zhang,<div><br></div><div>Thanks for your help. I will try your recommendations. </div><div><br></div><div>Best Regard, </div><div>Daniel Cho</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Zhang, Chen &lt;<a href="mailto:chen.zhang@intel.com">chen.zhang@intel.com</a>&gt; 於 2019年12月4日 週三 下午4:32寫道:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
On 12/3/2019 9:25 PM, Dr. David Alan Gilbert wrote:<br>
&gt; * Daniel Cho (<a href="mailto:danielcho@qnap.com" target="_blank">danielcho@qnap.com</a>) wrote:<br>
&gt;&gt; Hi Dave,<br>
&gt;&gt;<br>
&gt;&gt; We could use the exist interface to add netfilter and chardev, it might not<br>
&gt;&gt; have the problem you said.<br>
&gt;&gt;<br>
&gt;&gt; However, the netfilter and chardev on the primary at the start, that means<br>
&gt;&gt; we could not dynamic set COLO<br>
&gt;&gt; feature to VM?<br>
&gt; I wasn&#39;t expecting that to be possible - I&#39;d expect you to be able<br>
&gt; to start in a state that looks the same as a primary+failed secondary;<br>
&gt; but I&#39;m not sure.<br>
<br>
Current COLO (with Lukas&#39;s patch) can support dynamic set COLO system.<br>
<br>
This status is same like the secondary has triggered failover, the <br>
primary node need to find new secondary<br>
<br>
node to combine new COLO system.<br>
<br>
<br>
&gt;&gt; We try to change this chardev to prevent primary VM will stuck to wait<br>
&gt;&gt; secondary VM.<br>
&gt;&gt;<br>
&gt;&gt; -chardev socket,id=compare1,host=127.0.0.1,port=9004,server,wait \<br>
&gt;&gt;<br>
&gt;&gt; to<br>
&gt;&gt;<br>
&gt;&gt; -chardev socket,id=compare1,host=127.0.0.1,port=9004,server,nowait \<br>
&gt;&gt;<br>
&gt;&gt; But it will make primary VM&#39;s network not works. (Can&#39;t get ip), until<br>
&gt;&gt; starting connect with secondary VM.<br>
<br>
I think you need to check the port 9004 if already connect to the pair node.<br>
<br>
&gt; I&#39;m not sure of the answer to this; I&#39;ve not tried doing it - I&#39;m not<br>
&gt; sure anyone has!<br>
&gt; But, the colo components do track the state of tcp connections, so I&#39;m<br>
&gt; expecting that they have to already exist to have the state of those<br>
&gt; connections available for when you start the secondary.<br>
<br>
Yes, you are right.<br>
<br>
For this status, we don&#39;t need to sync the state of tcp connections, <br>
because after failover<br>
<br>
(or just solo COLO primary node), we have empty all the tcp connections <br>
state in COLO module.<br>
<br>
In the processing of build new COLO pair, we will sync all the VM state <br>
to secondary node and re-build<br>
<br>
new track things in COLO module.<br>
<br>
<br>
&gt;<br>
&gt;<br>
&gt;&gt; Otherwise, the primary VM with netfileter / chardev and without netfilter /<br>
&gt;&gt; chardev , they takes very different<br>
&gt;&gt; booting time.<br>
&gt;&gt; Without  netfilter / chardev : about 1 mins<br>
&gt;&gt; With   netfilter / chardev : about 5 mins<br>
&gt;&gt; Is this an issue?<br>
&gt; that sounds like it needs investigating.<br>
&gt;<br>
&gt; Dave<br>
<br>
Yes, In previous COLO use cases, we need make primary node and secondary <br>
node boot in the same time.<br>
<br>
I did&#39;t expect such a big difference on netfilter/chardev.<br>
<br>
I think you can try without netfilter/chardev, after VM boot, re-build <br>
the netfilter/chardev related work with chardev server nowait.<br>
<br>
<br>
Thanks<br>
<br>
Zhang Chen<br>
<br>
&gt;<br>
&gt;&gt; Best regards,<br>
&gt;&gt; Daniel Cho<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt; Dr. David Alan Gilbert &lt;<a href="mailto:dgilbert@redhat.com" target="_blank">dgilbert@redhat.com</a>&gt; 於 2019年12月2日 週一 下午5:58寫道:<br>
&gt;&gt;<br>
&gt;&gt;&gt; * Daniel Cho (<a href="mailto:danielcho@qnap.com" target="_blank">danielcho@qnap.com</a>) wrote:<br>
&gt;&gt;&gt;&gt; Hi Zhang,<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt; We use qemu-4.1.0 release on this case.<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt; I think we need use block mirror to sync the disk to secondary node<br>
&gt;&gt;&gt; first,<br>
&gt;&gt;&gt;&gt; then stop the primary VM and build COLO system.<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt; In the stop moment, you need add some netfilter and chardev socket node<br>
&gt;&gt;&gt; for<br>
&gt;&gt;&gt;&gt; COLO, maybe you need re-check this part.<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt; Our test was already follow those step. Maybe I could describe the detail<br>
&gt;&gt;&gt;&gt; of the test flow and issues.<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt; Step 1:<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt; Create primary VM without any netfilter and chardev for COLO, and using<br>
&gt;&gt;&gt;&gt; other host ping primary VM continually.<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt; Step 2:<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt; Create secondary VM (the same device/drive with primary VM), and do block<br>
&gt;&gt;&gt;&gt; mirror sync ( ping to primary VM normally )<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt; Step 3:<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt; After block mirror sync finish, add those netfilter and chardev to<br>
&gt;&gt;&gt; primary<br>
&gt;&gt;&gt;&gt; VM and secondary VM for COLO ( *Can&#39;t* ping to primary VM but those<br>
&gt;&gt;&gt; packets<br>
&gt;&gt;&gt;&gt; will be received later )<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt; Step 4:<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt; Start migrate primary VM to secondary VM, and primary VM &amp; secondary VM<br>
&gt;&gt;&gt; are<br>
&gt;&gt;&gt;&gt; running ( ping to primary VM works and receive those packets on step 3<br>
&gt;&gt;&gt;&gt; status )<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt; Between Step 3 to Step 4, it will take 10~20 seconds in our environment.<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt; I could image this issue (delay reply packets) is because of setting COLO<br>
&gt;&gt;&gt;&gt; proxy for temporary status,<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt; but we thought 10~20 seconds might a little long. (If primary VM is<br>
&gt;&gt;&gt; already<br>
&gt;&gt;&gt;&gt; doing some jobs, it might lose the data.)<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt; Could we reduce those time? or those delay is depends on different VM?<br>
&gt;&gt;&gt; I think you need to set up the netfilter and chardev on the primary at<br>
&gt;&gt;&gt; the start;  the filter contains the state of the TCP connections working<br>
&gt;&gt;&gt; with the VM, so adding it later can&#39;t gain that state for existing<br>
&gt;&gt;&gt; connections.<br>
&gt;&gt;&gt;<br>
&gt;&gt;&gt; Dave<br>
&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt; Best Regard,<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt; Daniel Cho.<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt; Zhang, Chen &lt;<a href="mailto:chen.zhang@intel.com" target="_blank">chen.zhang@intel.com</a>&gt; 於 2019年11月30日 週六 上午2:04寫道:<br>
&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt; *From:* Daniel Cho &lt;<a href="mailto:danielcho@qnap.com" target="_blank">danielcho@qnap.com</a>&gt;<br>
&gt;&gt;&gt;&gt;&gt; *Sent:* Friday, November 29, 2019 10:43 AM<br>
&gt;&gt;&gt;&gt;&gt; *To:* Zhang, Chen &lt;<a href="mailto:chen.zhang@intel.com" target="_blank">chen.zhang@intel.com</a>&gt;<br>
&gt;&gt;&gt;&gt;&gt; *Cc:* Dr. David Alan Gilbert &lt;<a href="mailto:dgilbert@redhat.com" target="_blank">dgilbert@redhat.com</a>&gt;;<br>
&gt;&gt;&gt; <a href="mailto:lukasstraub2@web.de" target="_blank">lukasstraub2@web.de</a>;<br>
&gt;&gt;&gt;&gt;&gt; <a href="mailto:qemu-devel@nongnu.org" target="_blank">qemu-devel@nongnu.org</a><br>
&gt;&gt;&gt;&gt;&gt; *Subject:* Re: Network connection with COLO VM<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt; Hi David,  Zhang,<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt; Thanks for replying my question.<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt; We know why will occur this issue.<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt; As you said, the COLO VM&#39;s network needs<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt; colo-proxy to control packets, so the guest&#39;s<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt; interface should set the filter to solve the problem.<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt; But we found another question, when we set the<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt; fault-tolerance feature to guest (primary VM is running,<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt; secondary VM is pausing), the guest&#39;s network would not<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt; responds any request for a while (in our environment<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt; about 20~30 secs) after secondary VM runs.<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt; Does it be a normal situation, or a known issue?<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt; Our test is creating primary VM for a while, then creating<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt; secondary VM to make it with COLO feature.<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt; Hi Daniel,<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt; Happy to hear you have solved ssh disconnection issue.<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt; Do you use Lukas’s patch on this case?<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt; I think we need use block mirror to sync the disk to secondary node<br>
&gt;&gt;&gt; first,<br>
&gt;&gt;&gt;&gt;&gt; then stop the primary VM and build COLO system.<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt; In the stop moment, you need add some netfilter and chardev socket node<br>
&gt;&gt;&gt;&gt;&gt; for COLO, maybe you need re-check this part.<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt; Best Regard,<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt; Daniel Cho<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt; Zhang, Chen &lt;<a href="mailto:chen.zhang@intel.com" target="_blank">chen.zhang@intel.com</a>&gt; 於 2019年11月28日 週四 上午9:26寫道:<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;&gt; -----Original Message-----<br>
&gt;&gt;&gt;&gt;&gt;&gt; From: Dr. David Alan Gilbert &lt;<a href="mailto:dgilbert@redhat.com" target="_blank">dgilbert@redhat.com</a>&gt;<br>
&gt;&gt;&gt;&gt;&gt;&gt; Sent: Wednesday, November 27, 2019 6:51 PM<br>
&gt;&gt;&gt;&gt;&gt;&gt; To: Daniel Cho &lt;<a href="mailto:danielcho@qnap.com" target="_blank">danielcho@qnap.com</a>&gt;; Zhang, Chen<br>
&gt;&gt;&gt;&gt;&gt;&gt; &lt;<a href="mailto:chen.zhang@intel.com" target="_blank">chen.zhang@intel.com</a>&gt;; <a href="mailto:lukasstraub2@web.de" target="_blank">lukasstraub2@web.de</a><br>
&gt;&gt;&gt;&gt;&gt;&gt; Cc: <a href="mailto:qemu-devel@nongnu.org" target="_blank">qemu-devel@nongnu.org</a><br>
&gt;&gt;&gt;&gt;&gt;&gt; Subject: Re: Network connection with COLO VM<br>
&gt;&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;&gt; * Daniel Cho (<a href="mailto:danielcho@qnap.com" target="_blank">danielcho@qnap.com</a>) wrote:<br>
&gt;&gt;&gt;&gt;&gt;&gt;&gt; Hello everyone,<br>
&gt;&gt;&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;&gt;&gt; Could we ssh to colo VM (means PVM &amp; SVM are starting)?<br>
&gt;&gt;&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;&gt; Lets cc in Zhang Chen and Lukas Straub.<br>
&gt;&gt;&gt;&gt;&gt; Thanks Dave.<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;&gt;&gt; SSH will connect to colo VM for a while, but it will disconnect<br>
&gt;&gt;&gt; with<br>
&gt;&gt;&gt;&gt;&gt;&gt;&gt; error<br>
&gt;&gt;&gt;&gt;&gt;&gt;&gt; *client_loop: send disconnect: Broken pipe*<br>
&gt;&gt;&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;&gt;&gt; It seems to colo VM could not keep network session.<br>
&gt;&gt;&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;&gt;&gt; Does it be a known issue?<br>
&gt;&gt;&gt;&gt;&gt;&gt; That sounds like the COLO proxy is getting upset; it&#39;s supposed to<br>
&gt;&gt;&gt;&gt;&gt; compare<br>
&gt;&gt;&gt;&gt;&gt;&gt; packets sent by the primary and secondary and only send one to the<br>
&gt;&gt;&gt;&gt;&gt; outside<br>
&gt;&gt;&gt;&gt;&gt;&gt; - you shouldn&#39;t be talking directly to the guest, but always via the<br>
&gt;&gt;&gt;&gt;&gt; proxy.  See<br>
&gt;&gt;&gt;&gt;&gt;&gt; docs/colo-proxy.txt<br>
&gt;&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt; Hi Daniel,<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt; I have try ssh to COLO guest with 8 hours, not occurred this issue.<br>
&gt;&gt;&gt;&gt;&gt; Please check your network/qemu configuration.<br>
&gt;&gt;&gt;&gt;&gt; But I found another problem maybe related this issue, if no network<br>
&gt;&gt;&gt;&gt;&gt; communication for a period of time(maybe 10min), the first message<br>
&gt;&gt;&gt; send to<br>
&gt;&gt;&gt;&gt;&gt; guest have a chance with delay(maybe 1-5 sec), I will try to fix it<br>
&gt;&gt;&gt; when I<br>
&gt;&gt;&gt;&gt;&gt; have time.<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt; Thanks<br>
&gt;&gt;&gt;&gt;&gt; Zhang Chen<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;&gt; Dave<br>
&gt;&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt;&gt;&gt;&gt;&gt; Best Regard,<br>
&gt;&gt;&gt;&gt;&gt;&gt;&gt; Daniel Cho<br>
&gt;&gt;&gt;&gt;&gt;&gt; --<br>
&gt;&gt;&gt;&gt;&gt;&gt; Dr. David Alan Gilbert / <a href="mailto:dgilbert@redhat.com" target="_blank">dgilbert@redhat.com</a> / Manchester, UK<br>
&gt;&gt;&gt;&gt;&gt;<br>
&gt;&gt;&gt; --<br>
&gt;&gt;&gt; Dr. David Alan Gilbert / <a href="mailto:dgilbert@redhat.com" target="_blank">dgilbert@redhat.com</a> / Manchester, UK<br>
&gt;&gt;&gt;<br>
&gt;&gt;&gt;<br>
&gt; --<br>
&gt; Dr. David Alan Gilbert / <a href="mailto:dgilbert@redhat.com" target="_blank">dgilbert@redhat.com</a> / Manchester, UK<br>
&gt;<br>
</blockquote></div>

^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, back to index

Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-11-27  4:20 Network connection with COLO VM Daniel Cho
2019-11-27 10:51 ` Dr. David Alan Gilbert
2019-11-28  1:26   ` Zhang, Chen
2019-11-29  2:42     ` Daniel Cho
2019-11-29 18:04       ` Zhang, Chen
2019-12-02  3:55         ` Daniel Cho
2019-12-02  9:58           ` Dr. David Alan Gilbert
2019-12-03  9:08             ` Daniel Cho
2019-12-03 13:25               ` Dr. David Alan Gilbert
2019-12-04  8:32                 ` Zhang, Chen
2019-12-06  6:31                   ` Daniel Cho

QEMU-Devel Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/qemu-devel/0 qemu-devel/git/0.git
	git clone --mirror https://lore.kernel.org/qemu-devel/1 qemu-devel/git/1.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 qemu-devel qemu-devel/ https://lore.kernel.org/qemu-devel \
		qemu-devel@nongnu.org
	public-inbox-index qemu-devel

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.nongnu.qemu-devel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git