From mboxrd@z Thu Jan 1 00:00:00 1970 From: Yuxin Ren Subject: RCU consistency guarantees Date: Thu, 5 Dec 2019 20:17:50 -0500 Message-ID: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============4716324073142384785==" Return-path: Received: from mail-lj1-x232.google.com (mail-lj1-x232.google.com [IPv6:2a00:1450:4864:20::232]) by lists.lttng.org (Postfix) with ESMTPS id 47TZTV0gTWz117P for ; Thu, 5 Dec 2019 20:18:05 -0500 (EST) Received: by mail-lj1-x232.google.com with SMTP id a13so5751119ljm.10 for ; Thu, 05 Dec 2019 17:18:05 -0800 (PST) List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" To: lttng-dev List-Id: lttng-dev@lists.lttng.org --===============4716324073142384785== Content-Type: multipart/alternative; boundary="0000000000004d338e0598fed15a" --0000000000004d338e0598fed15a Content-Type: text/plain; charset="UTF-8" Hi, I am a student, and learning RCU now, but still know very little about it. Are there any documents/papers/materials which (in)formally define and explain RCU consistency guarantees? I know there are some consistency models in the database area (such as PRAM, Read Uncommitted, etc) from https://jepsen.io/consistency and [1]. How does RCU related to those consistency models? I also found some comments online (One key distinction is that both MVCC and RLU provide much stronger consistency guarantees to readers than does RCU ...) (https://lwn.net/Articles/777036/). I do not understand how we reason/dresibe/compare the consistency guarantees. ( I even do not know what consistency guarantees provided by RCU formally) Could someone explain this to me? [1] Bailis, P., Davidson, A., Fekete, A., Ghodsi, A., Hellerstein, J. M., & Stoica, I. (2013). Highly available transactions: Virtues and limitations. Proceedings of the VLDB Endowment, 7(3), 181-192. Thanks Yuxin --0000000000004d338e0598fed15a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

I am a student, and learning RCU no= w, but still know very little about it.
Are there any documents/p= apers/materials which (in)formally define and explain RCU consistency=C2=A0= guarantees?

I know there are some consistency mode= ls in the database area (such as PRAM,=C2=A0Read Uncommitted, etc) from=C2= =A0https://jeps= en.io/consistency=C2=A0and [1].
How does RCU related to those= consistency models?

I also found some comments on= line (One key distinction is that both MVCC and RLU provide much stronger c= onsistency guarantees to readers than does RCU ...) (https://lwn.net/Articles/777036/).
--0000000000004d338e0598fed15a-- --===============4716324073142384785== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev --===============4716324073142384785==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mathieu Desnoyers Subject: Re: RCU consistency guarantees Date: Fri, 6 Dec 2019 05:49:09 -0500 (EST) Message-ID: <194534011.751.1575629349181.JavaMail.zimbra__43626.0490723484$1575629380$gmane$org@efficios.com> References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============7951139539516759308==" Return-path: Received: from mail.efficios.com (mail.efficios.com [167.114.142.138]) by lists.lttng.org (Postfix) with ESMTPS id 47Tq8X4jFWz11bd for ; Fri, 6 Dec 2019 05:49:16 -0500 (EST) Received: from localhost (ip6-localhost [IPv6:::1]) by mail.efficios.com (Postfix) with ESMTP id E619D35553F for ; Fri, 6 Dec 2019 05:49:09 -0500 (EST) In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" To: Yuxin Ren Cc: lttng-dev List-Id: lttng-dev@lists.lttng.org --===============7951139539516759308== Content-Type: multipart/alternative; boundary="=_6922dbbd-f2bd-4f89-981c-96d88c3c974d" --=_6922dbbd-f2bd-4f89-981c-96d88c3c974d Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable ----- On Dec 5, 2019, at 8:17 PM, Yuxin Ren wrote:=20 > Hi, > I am a student, and learning RCU now, but still know very little about it= . > Are there any documents/papers/materials which (in)formally define and ex= plain > RCU consistency guarantees? You may want to have a look at=20 User-Level Implementations of Read-Copy Update=20 Article in IEEE Transactions on Parallel and Distributed Systems 23(2):375 = - 382 =C2=B7 March 2012=20 as a starting point.=20 Thanks,=20 Mathieu=20 > I know there are some consistency models in the database area (such as PR= AM, > Read Uncommitted, etc) from [ https://jepsen.io/consistency | > https://jepsen.io/consistency ] and [1]. > How does RCU related to those consistency models? > I also found some comments online (One key distinction is that both MVCC = and RLU > provide much stronger consistency guarantees to readers than does RCU ...= ) ( [ > https://lwn.net/Articles/777036/ | https://lwn.net/Articles/777036/ ] ). > I do not understand how we reason/dresibe/compare the consistency guarant= ees. ( > I even do not know what consistency guarantees provided by RCU formally) > Could someone explain this to me? > [1] Bailis, P., Davidson, A., Fekete, A., Ghodsi, A., Hellerstein, J. M.,= & > Stoica, I. (2013). Highly available transactions: Virtues and limitations= . > Proceedings of the VLDB Endowment, 7(3), 181-192. > Thanks > Yuxin > _______________________________________________ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev --=20 Mathieu Desnoyers=20 EfficiOS Inc.=20 http://www.efficios.com=20 --=_6922dbbd-f2bd-4f89-981c-96d88c3c974d Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable

----- On Dec 5, 2019, at 8:17 PM, Y= uxin Ren <ryx@gwmail.gwu.edu> wrote:
Hi,
I am a student, and learning RCU now, but= still know very little about it.
Are there any documents/papers/= materials which (in)formally define and explain RCU consistency guaran= tees?

You may want to have= a look at

User-Level Implementations of Read-= Copy Update
Article=E2=80=82in=E2=80=82IEEE Transactions on Paral= lel and Distributed Systems 23(2):375 - 382 =C2=B7 March 2012
as a starting point.

Thanks,

Mathieu


How does RCU= related to those consistency models?

I also found some comme= nts online (One key distinction is that both MVCC and RLU provide much stro= nger consistency guarantees to readers than does RCU ...) (https://lwn.net/Articles/777036/).
I do not understa= nd how we reason/dresibe/compare the consistency guarantees. ( I even = do not know what consistency guarantees provided by RCU formally)
Could someone explain this to me?


[1] Bailis, P., Davidson= , A., Fekete, A., Ghodsi, A., Hellerstein, J. M., & Stoica, I. (2013). = Highly available transactions: Virtues and limitations. Proceedings of the = VLDB Endowment, 7(3), 181-192.

Thanks
Yuxin

_______________________________________________
lttng-dev mailing li= st
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/= listinfo/lttng-dev

--
Mathieu Desnoyers
EfficiOS Inc.
http= ://www.efficios.com
--=_6922dbbd-f2bd-4f89-981c-96d88c3c974d-- --===============7951139539516759308== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev --===============7951139539516759308==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Yuxin Ren Subject: Re: RCU consistency guarantees Date: Fri, 6 Dec 2019 09:51:32 -0500 Message-ID: References: <194534011.751.1575629349181.JavaMail.zimbra@efficios.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============7949053342543689592==" Return-path: Received: from mail-lj1-x236.google.com (mail-lj1-x236.google.com [IPv6:2a00:1450:4864:20::236]) by lists.lttng.org (Postfix) with ESMTPS id 47TwXN6pc2z11fr for ; Fri, 6 Dec 2019 09:51:48 -0500 (EST) Received: by mail-lj1-x236.google.com with SMTP id c19so7894618lji.11 for ; Fri, 06 Dec 2019 06:51:48 -0800 (PST) In-Reply-To: <194534011.751.1575629349181.JavaMail.zimbra@efficios.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" To: Mathieu Desnoyers Cc: lttng-dev List-Id: lttng-dev@lists.lttng.org --===============7949053342543689592== Content-Type: multipart/alternative; boundary="00000000000059ff2805990a2f47" --00000000000059ff2805990a2f47 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, Dec 6, 2019 at 5:49 AM Mathieu Desnoyers < mathieu.desnoyers@efficios.com> wrote: > > ----- On Dec 5, 2019, at 8:17 PM, Yuxin Ren wrote: > > Hi, > I am a student, and learning RCU now, but still know very little about it= . > Are there any documents/papers/materials which (in)formally define and > explain RCU consistency guarantees? > > > You may want to have a look at > > User-Level Implementations of Read-Copy Update > Article in IEEE Transactions on Parallel and Distributed Systems 23(2):37= 5 > - 382 =C2=B7 March 2012 > Thanks for your info. However, I do not think URCU talks about any consistency model formally. >From previous communication with Paul, he said RCU is not designed for linearizability, and it is totally acceptable that RCU is not linearizable. However, I am curious how to accurately/formally Characterize RCU consistency model/guarantees > > as a starting point. > > Thanks, > > Mathieu > > > I know there are some consistency models in the database area (such as > PRAM, Read Uncommitted, etc) from https://jepsen.io/consistency and [1]. > How does RCU related to those consistency models? > > I also found some comments online (One key distinction is that both MVCC > and RLU provide much stronger consistency guarantees to readers than does > RCU ...) (https://lwn.net/Articles/777036/). > I do not understand how we reason/dresibe/compare the > consistency guarantees. ( I even do not know what consistency guarantees > provided by RCU formally) > Could someone explain this to me? > > > > [1] Bailis, P., Davidson, A., Fekete, A., Ghodsi, A., Hellerstein, J. M., > & Stoica, I. (2013). Highly available transactions: Virtues and > limitations. Proceedings of the VLDB Endowment, 7(3), 181-192. > > Thanks > Yuxin > > _______________________________________________ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > > > -- > Mathieu Desnoyers > EfficiOS Inc. > http://www.efficios.com > --00000000000059ff2805990a2f47 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Fri, Dec 6, 2019 at 5:49 AM Mathie= u Desnoyers <mathieu.d= esnoyers@efficios.com> wrote:

----- On Dec 5, 2019, at 8:17 PM, Yuxin Ren <= ;ryx@gwmail.gwu.edu= > wrote:
Hi,
I am a studen= t, and learning RCU now, but still know very little about it.
Are= there any documents/papers/materials which (in)formally define and explain= RCU consistency=C2=A0guarantees?

You may want to have a look at

User-Le= vel Implementations of Read-Copy Update
Article=E2=80=82in=E2=80= =82IEEE Transactions on Parallel and Distributed Systems 23(2):375 - 382 = =C2=B7 March 2012

=
Thanks for your info.
However, I do not think URCU talks abo= ut any consistency=C2=A0model formally.=C2=A0

From= previous=C2=A0communication with=C2=A0Paul, he said RCU is not designed=C2= =A0for=C2=A0linearizability, and it is totally acceptable=C2=A0that RCU is = not=C2=A0linearizable.
However, I am curious how to accurately/fo= rmally=C2=A0Characterize RCU consistency=C2=A0model/guarantees
<= br>
as a starting point.

Thanks,

Mathieu


I know there are some consistency models in the database area (su= ch as PRAM,=C2=A0Read Uncommitted, etc) from=C2=A0htt= ps://jepsen.io/consistency=C2=A0and [1].
How does RCU related= to those consistency models?

I also found some comments onli= ne (One key distinction is that both MVCC and RLU provide much stronger con= sistency guarantees to readers than does RCU ...) (https://lwn.net/Articles/777036/).
I do not understand how w= e reason/dresibe/compare the consistency=C2=A0guarantees. ( I even do not k= now what consistency guarantees provided by RCU formally)
Could s= omeone explain this=C2=A0to me?


[1]=C2=A0Bailis, P., Davidson, A., Fekete, A., Ghodsi, A.= , Hellerstein, J. M., & Stoica, I. (2013). Highly available transaction= s: Virtues and limitations. Proceedings of the VLDB Endowment, 7(3), 181-19= 2.

Thanks
Yuxin

_______________________________________________
lttng-dev mailing li= st
lttng-= dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/ma= ilman/listinfo/lttng-dev

-- <= br>
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
--00000000000059ff2805990a2f47-- --===============7949053342543689592== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev --===============7949053342543689592==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Yuxin Ren Subject: RCU consistency guarantees Date: Thu, 5 Dec 2019 20:15:10 -0500 Message-ID: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0022435181231759913==" Return-path: Received: from mail-lf1-x130.google.com (mail-lf1-x130.google.com [IPv6:2a00:1450:4864:20::130]) by lists.lttng.org (Postfix) with ESMTPS id 47TZQQ17R8z10nJ for ; Thu, 5 Dec 2019 20:15:25 -0500 (EST) Received: by mail-lf1-x130.google.com with SMTP id b15so3966598lfc.4 for ; Thu, 05 Dec 2019 17:15:25 -0800 (PST) List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" To: lttng-dev List-Id: lttng-dev@lists.lttng.org --===============0022435181231759913== Content-Type: multipart/alternative; boundary="000000000000c1cb7a0598fec713" --000000000000c1cb7a0598fec713 Content-Type: text/plain; charset="UTF-8" Hi, I am a student, and learning RCU now, but still know very little about it. Are there any documents/papers/materials which (in)formally define and explain RCU consistency guarantees? I know there are some consistency models in the database area (such as PRAM, Read Uncommitted, etc) from https://jepsen.io/consistency and [1]. How does RCU related to those consistency models? I also found some comments online (One key distinction is that both MVCC and RLU provide much stronger consistency guarantees to readers than does RCU ...) (https://lwn.net/Articles/777036/). I do not understand how we reason/dresibe/compare the consistency guarantees. ( I even do not know what consistency guarantees provided by RCU formally) Could someone explain this to me? [1] Bailis, P., Davidson, A., Fekete, A., Ghodsi, A., Hellerstein, J. M., & Stoica, I. (2013). Highly available transactions: Virtues and limitations. Proceedings of the VLDB Endowment, 7(3), 181-192. Thanks Yuxin --000000000000c1cb7a0598fec713 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

I am a student, and learning RCU no= w, but still know very little about it.
Are there any documents/p= apers/materials which (in)formally define and explain RCU consistency=C2=A0= guarantees?

I know there are some consistency mode= ls in the database area (such as PRAM,=C2=A0Read Uncommitted, etc) from=C2= =A0https://jepsen.io/consistency<= /a> and [1].
How does RCU related to those consistency models?

I do not understand how we= reason/dresibe/compare the consistency=C2=A0guarantees. ( I even do not kn= ow what consistency guarantees provided by RCU formally)
Could so= meone explain this=C2=A0to me?

[1]=C2=A0Bailis, P.= , Davidson, A., Fekete, A., Ghodsi, A., Hellerstein, J. M., & Stoica, I= . (2013). Highly available transactions: Virtues and limitations. Proceedin= gs of the VLDB Endowment, 7(3), 181-192.

Thanks
Yuxin
--000000000000c1cb7a0598fec713-- --===============0022435181231759913== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev --===============0022435181231759913==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mathieu Desnoyers Subject: Re: RCU consistency guarantees Date: Fri, 6 Dec 2019 10:59:05 -0500 (EST) Message-ID: <512711764.1078.1575647945136.JavaMail.zimbra__23192.7767208376$1575647964$gmane$org@efficios.com> References: <194534011.751.1575629349181.JavaMail.zimbra@efficios.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============7098162895546590200==" Return-path: Received: from mail.efficios.com (mail.efficios.com [167.114.142.138]) by lists.lttng.org (Postfix) with ESMTPS id 47Ty225Ndbz11tS for ; Fri, 6 Dec 2019 10:59:06 -0500 (EST) Received: from localhost (ip6-localhost [IPv6:::1]) by mail.efficios.com (Postfix) with ESMTP id 027D826F8E1 for ; Fri, 6 Dec 2019 10:59:06 -0500 (EST) In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" To: Yuxin Ren Cc: lttng-dev , paulmck List-Id: lttng-dev@lists.lttng.org --===============7098162895546590200== Content-Type: multipart/alternative; boundary="=_a750b78c-95fb-445c-8c13-4664ffae9078" --=_a750b78c-95fb-445c-8c13-4664ffae9078 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable ----- On Dec 6, 2019, at 3:51 PM, Yuxin Ren wrote:=20 > On Fri, Dec 6, 2019 at 5:49 AM Mathieu Desnoyers < [ > mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] = > > wrote: >> ----- On Dec 5, 2019, at 8:17 PM, Yuxin Ren < [ mailto:ryx@gwmail.gwu.ed= u | >> ryx@gwmail.gwu.edu ] > wrote: >>> Hi, >>> I am a student, and learning RCU now, but still know very little about = it. >>> Are there any documents/papers/materials which (in)formally define and = explain >>> RCU consistency guarantees? >> You may want to have a look at >> User-Level Implementations of Read-Copy Update >> Article in IEEE Transactions on Parallel and Distributed Systems 23(2):3= 75 - 382 >> =C2=B7 March 2012 > Thanks for your info. > However, I do not think URCU talks about any consistency model formally. > From previous communication with Paul, he said RCU is not designed for > linearizability, and it is totally acceptable that RCU is not linearizabl= e. > However, I am curious how to accurately/formally Characterize RCU consist= ency > model/guarantees Adding Paul E. McKenney in CC.=20 I am referring to the section "Overview of RCU semantics" in the paper. Not= sure it has the level of=20 formality you are looking for though. Paul, do you have pointers to additio= nal material ?=20 Thanks,=20 Mathieu=20 >> as a starting point. >> Thanks, >> Mathieu >>> I know there are some consistency models in the database area (such as = PRAM, >>> Read Uncommitted, etc) from [ https://jepsen.io/consistency | >>> https://jepsen.io/consistency ] and [1]. >>> How does RCU related to those consistency models? >>> I also found some comments online (One key distinction is that both MVC= C and RLU >>> provide much stronger consistency guarantees to readers than does RCU .= ..) ( [ >>> https://lwn.net/Articles/777036/ | https://lwn.net/Articles/777036/ ] )= . >>> I do not understand how we reason/dresibe/compare the consistency guara= ntees. ( >>> I even do not know what consistency guarantees provided by RCU formally= ) >>> Could someone explain this to me? >>> [1] Bailis, P., Davidson, A., Fekete, A., Ghodsi, A., Hellerstein, J. M= ., & >>> Stoica, I. (2013). Highly available transactions: Virtues and limitatio= ns. >>> Proceedings of the VLDB Endowment, 7(3), 181-192. >>> Thanks >>> Yuxin >>> _______________________________________________ >>> lttng-dev mailing list >>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org ] >>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev | >>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ] >> -- >> Mathieu Desnoyers >> EfficiOS Inc. >> [ http://www.efficios.com/ | http://www.efficios.com ] --=20 Mathieu Desnoyers=20 EfficiOS Inc.=20 http://www.efficios.com=20 --=_a750b78c-95fb-445c-8c13-4664ffae9078 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable

----- On Dec 6, 2019, at 3:51 PM, Y= uxin Ren <ryx@gwmail.gwu.edu> wrote:


On Fri, Dec 6, 2019 at 5:49 AM Mathieu Desnoyers <= mathieu.desnoyers@efficios.com> wrote:=

----- On Dec 5, 2019, at 8:17 PM, Yuxin Ren <ryx@gwma= il.gwu.edu> wrote:
Hi,
I am a student, and learning RCU now, but still know very = little about it.
Are there any documents/papers/materials which (= in)formally define and explain RCU consistency guarantees?
=

You may want to have a look at

User-Level Implementations of Read-Copy Update
Article=E2=80=82i= n=E2=80=82IEEE Transactions on Parallel and Distributed Systems 23(2):375 -= 382 =C2=B7 March 2012

T= hanks for your info.
However, I do not think URCU talks about any= consistency model formally. 

From previous co= mmunication with Paul, he said RCU is not designed for linea= rizability, and it is totally acceptable that RCU is not lineariz= able.
However, I am curious how to accurately/formally Chara= cterize RCU consistency model/guarantees
Adding Paul E. McKenney in CC.
I am referring to the section "Overview of= RCU semantics" in the paper. Not sure it has the level of
formality you are looking for though. Paul, do you have = pointers to additional material ?

Thanks,
Mathieu



as a starting point.
Thanks,

Mathieu


I know there are some consist= ency models in the database area (such as PRAM, Read Uncommitted, etc)= from https://jepse= n.io/consistency and [1].
How does RCU related to those = consistency models?

I also found some comments online (One ke= y distinction is that both MVCC and RLU provide much stronger consistency g= uarantees to readers than does RCU ...) (https://lwn.net/Articles/777036/).
I do n= ot understand how we reason/dresibe/compare the consistency guarantees= . ( I even do not know what consistency guarantees provided by RCU formally= )
Could someone explain this to me?
=

Thanks, = > = > Mathieu = > = > >> as a starting point. > = > >> Thanks, > = > >> Mathieu > = > >>> I know there are some consistency models in the database area (such a= s PRAM, > >>> Read Uncommitted, etc) from [ https://jepsen.io/consistency | > >>> https://jepsen.io/consistency ] and [1]. > >>> How does RCU related to those consistency models? > = > >>> I also found some comments online (One key distinction is that both M= VCC and RLU > >>> provide much stronger consistency guarantees to readers than does RCU= ...) ( [ > >>> https://lwn.net/Articles/777036/ | https://lwn.net/Articles/777036/ ]= ). > >>> I do not understand how we reason/dresibe/compare the consistency gua= rantees. ( > >>> I even do not know what consistency guarantees provided by RCU formal= ly) > >>> Could someone explain this to me? > = > >>> [1] Bailis, P., Davidson, A., Fekete, A., Ghodsi, A., Hellerstein, J.= M., & > >>> Stoica, I. (2013). Highly available transactions: Virtues and limitat= ions. > >>> Proceedings of the VLDB Endowment, 7(3), 181-192. > = > >>> Thanks > >>> Yuxin > = > >>> _______________________________________________ > >>> lttng-dev mailing list > >>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org ] > >>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev | > >>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ] > = > >> -- > >> Mathieu Desnoyers > >> EfficiOS Inc. > >> [ http://www.efficios.com/ | http://www.efficios.com ] > = > -- = > Mathieu Desnoyers = > EfficiOS Inc. = > http://www.efficios.com = From mboxrd@z Thu Jan 1 00:00:00 1970 From: Yuxin Ren Subject: Re: RCU consistency guarantees Date: Fri, 6 Dec 2019 19:00:13 -0500 Message-ID: References: <194534011.751.1575629349181.JavaMail.zimbra@efficios.com> <512711764.1078.1575647945136.JavaMail.zimbra@efficios.com> <20191206163052.GG2889@paulmck-ThinkPad-P72> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2512842037105845090==" Return-path: Received: from mail-lj1-x242.google.com (mail-lj1-x242.google.com [IPv6:2a00:1450:4864:20::242]) by lists.lttng.org (Postfix) with ESMTPS id 47V8jV52zVz12L0 for ; Fri, 6 Dec 2019 19:00:29 -0500 (EST) Received: by mail-lj1-x242.google.com with SMTP id s22so9451871ljs.7 for ; Fri, 06 Dec 2019 16:00:29 -0800 (PST) In-Reply-To: <20191206163052.GG2889@paulmck-ThinkPad-P72> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" To: paulmck@kernel.org Cc: lttng-dev List-Id: lttng-dev@lists.lttng.org --===============2512842037105845090== Content-Type: multipart/alternative; boundary="000000000000995117059911d988" --000000000000995117059911d988 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks so much for your great help. I definitely will look at those resources and papers! One more thing that I am confused As I mentioned earlier, someone said One key distinction is that both MVCC and RLU provide much stronger consistency guarantees to readers than does RCU ...) (https://lwn.net/Articles/777036/). I am not sure if the above statement is correct or not. But in general, How can we compare RCU consistency guarantees to other techniques (such as RLU)? How to reason about which one has stronger or weaker guarantees? Thanks Yuxin On Fri, Dec 6, 2019 at 11:30 AM Paul E. McKenney wrote= : > On Fri, Dec 06, 2019 at 10:59:05AM -0500, Mathieu Desnoyers wrote: > > ----- On Dec 6, 2019, at 3:51 PM, Yuxin Ren wrote: > > > > > On Fri, Dec 6, 2019 at 5:49 AM Mathieu Desnoyers < [ > > > mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.co= m > ] > > > > wrote: > > > > >> ----- On Dec 5, 2019, at 8:17 PM, Yuxin Ren < [ mailto: > ryx@gwmail.gwu.edu | > > >> ryx@gwmail.gwu.edu ] > wrote: > > > > >>> Hi, > > >>> I am a student, and learning RCU now, but still know very little > about it. > > >>> Are there any documents/papers/materials which (in)formally define > and explain > > >>> RCU consistency guarantees? > > > > >> You may want to have a look at > > > > >> User-Level Implementations of Read-Copy Update > > >> Article in IEEE Transactions on Parallel and Distributed Systems > 23(2):375 - 382 > > >> =C2=B7 March 2012 > > > > > Thanks for your info. > > > However, I do not think URCU talks about any consistency model > formally. > > > > > From previous communication with Paul, he said RCU is not designed fo= r > > > linearizability, and it is totally acceptable that RCU is not > linearizable. > > > However, I am curious how to accurately/formally Characterize RCU > consistency > > > model/guarantees > > > > Adding Paul E. McKenney in CC. > > > > I am referring to the section "Overview of RCU semantics" in the paper. > Not sure it has the level of > > formality you are looking for though. Paul, do you have pointers to > additional material ? > > Indeed I do! The Linux kernel memory model (LKMM) includes RCU. It is > in tools/memory-model in recent kernel source trees, which includes > documentation. This is an executable model, which means that you > can create litmus tests and have the model formally and automatically > evaluate them. > > There are also a number of publications covering LKMM: > > o A formal kernel memory-ordering model > https://lwn.net/Articles/718628/ > https://lwn.net/Articles/720550/ > > These cover the release stores and dependency ordering that > provide RCU's publish-subscribe guarantees. > > Backup material here: > > > https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinuxM= M/ > > With these two likely being of particular interest: > > > https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinuxM= M/RCUguarantees.html > > https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinuxM= M/srcu.html > > o Frightening Small Children and Disconcerting Grown-ups: > Concurrency in the Linux Kernel > https://dl.acm.org/citation.cfm?id=3D3177156 > http://www0.cs.ucl.ac.uk/staff/j.alglave/papers/asplos18.pdf > > Backup material: > > http://diy.inria.fr/linux/ > > o Who's afraid of a big bad optimizing compiler? > https://lwn.net/Articles/793253/ > > o Calibrating your fear of big bad optimizing compilers > https://lwn.net/Articles/799218/ > > These last two justify use of normal C-language assignment > statements to initialize and access data referenced by > RCU-protected pointers. > > There is a large body of litmus tests (thousands of them) here: > > https://github.com/paulmckrcu/litmus > > Many of these litmus tests involve RCU, and these can be located by > search for files containing rcu_read_lock(), rcu_read_unlock(), > synchronize_rcu(), and so on. > > Or were you looking for something else? > > Thanx, Paul > > > Thanks, > > > > Mathieu > > > > >> as a starting point. > > > > >> Thanks, > > > > >> Mathieu > > > > >>> I know there are some consistency models in the database area (such > as PRAM, > > >>> Read Uncommitted, etc) from [ https://jepsen.io/consistency | > > >>> https://jepsen.io/consistency ] and [1]. > > >>> How does RCU related to those consistency models? > > > > >>> I also found some comments online (One key distinction is that both > MVCC and RLU > > >>> provide much stronger consistency guarantees to readers than does > RCU ...) ( [ > > >>> https://lwn.net/Articles/777036/ | https://lwn.net/Articles/777036/ > ] ). > > >>> I do not understand how we reason/dresibe/compare the consistency > guarantees. ( > > >>> I even do not know what consistency guarantees provided by RCU > formally) > > >>> Could someone explain this to me? > > > > >>> [1] Bailis, P., Davidson, A., Fekete, A., Ghodsi, A., Hellerstein, > J. M., & > > >>> Stoica, I. (2013). Highly available transactions: Virtues and > limitations. > > >>> Proceedings of the VLDB Endowment, 7(3), 181-192. > > > > >>> Thanks > > >>> Yuxin > > > > >>> _______________________________________________ > > >>> lttng-dev mailing list > > >>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org ] > > >>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev | > > >>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ] > > > > >> -- > > >> Mathieu Desnoyers > > >> EfficiOS Inc. > > >> [ http://www.efficios.com/ | http://www.efficios.com ] > > > > -- > > Mathieu Desnoyers > > EfficiOS Inc. > > http://www.efficios.com > --000000000000995117059911d988 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Thanks so much for your great help.
I definitely=C2=A0= will look at those resources and papers!

One more= =C2=A0thing that I am confused
As I mentioned=C2=A0earlier, someo= ne said=C2=A0One key distinction is that both MVCC and RLU provide much str= onger consistency guarantees to readers than does RCU ...) (https://lwn.net/Articles/777036/).
<= div>I am not sure if the above statement=C2=A0is correct=C2=A0or not. But i= n general,=C2=A0
How can we compare RCU consistency=C2=A0guarante= es=C2=A0to other techniques (such as RLU)?
How=C2=A0to reason abo= ut which one has stronger or weaker guarantees?

Th= anks
Yuxin

On Fri, Dec 6, 2019 at 11:30 AM Paul E. McKenney = <paulmck@kernel.org> wrote:=
On Fri, Dec 06,= 2019 at 10:59:05AM -0500, Mathieu Desnoyers wrote:
> ----- On Dec 6, 2019, at 3:51 PM, Yuxin Ren <ryx@gwmail.gwu.edu> wrote:
>
> > On Fri, Dec 6, 2019 at 5:49 AM Mathieu Desnoyers < [
> > mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
> > wrote:
>
> >> ----- On Dec 5, 2019, at 8:17 PM, Yuxin Ren < [ mailto:ryx@gwmail.gwu.edu= |
> >> ryx@g= wmail.gwu.edu ] > wrote:
>
> >>> Hi,
> >>> I am a student, and learning RCU now, but still know very= little about it.
> >>> Are there any documents/papers/materials which (in)formal= ly define and explain
> >>> RCU consistency guarantees?
>
> >> You may want to have a look at
>
> >> User-Level Implementations of Read-Copy Update
> >> Article in IEEE Transactions on Parallel and Distributed Syst= ems 23(2):375 - 382
> >> =C2=B7 March 2012
>
> > Thanks for your info.
> > However, I do not think URCU talks about any consistency model fo= rmally.
>
> > From previous communication with Paul, he said RCU is not designe= d for
> > linearizability, and it is totally acceptable that RCU is not lin= earizable.
> > However, I am curious how to accurately/formally Characterize RCU= consistency
> > model/guarantees
>
> Adding Paul E. McKenney in CC.
>
> I am referring to the section "Overview of RCU semantics" in= the paper. Not sure it has the level of
> formality you are looking for though. Paul, do you have pointers to ad= ditional material ?

Indeed I do!=C2=A0 The Linux kernel memory model (LKMM) includes RCU.=C2=A0= It is
in tools/memory-model in recent kernel source trees, which includes
documentation.=C2=A0 This is an executable model, which means that you
can create litmus tests and have the model formally and automatically
evaluate them.

There are also a number of publications covering LKMM:

o=C2=A0 =C2=A0 =C2=A0 =C2=A0A formal kernel memory-ordering model
=C2=A0 =C2=A0 =C2=A0 =C2=A0 https://lwn.net/Articles/718628/
=C2=A0 =C2=A0 =C2=A0 =C2=A0 https://lwn.net/Articles/720550/

=C2=A0 =C2=A0 =C2=A0 =C2=A0 These cover the release stores and dependency o= rdering that
=C2=A0 =C2=A0 =C2=A0 =C2=A0 provide RCU's publish-subscribe guarantees.=

=C2=A0 =C2=A0 =C2=A0 =C2=A0 Backup material here:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinux= MM/

=C2=A0 =C2=A0 =C2=A0 =C2=A0 With these two likely being of particular inter= est:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 https://mirrors.edge.kernel.org/pub/linux/kernel/peopl= e/paulmck/LWNLinuxMM/RCUguarantees.html
=C2=A0 =C2=A0 =C2=A0 =C2=A0 https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck= /LWNLinuxMM/srcu.html

o=C2=A0 =C2=A0 =C2=A0 =C2=A0Frightening Small Children and Disconcerting Gr= own-ups: Concurrency in the Linux Kernel
=C2=A0 =C2=A0 =C2=A0 =C2=A0 https://dl.acm.org/citatio= n.cfm?id=3D3177156
=C2=A0 =C2=A0 =C2=A0 =C2=A0 http://www0.= cs.ucl.ac.uk/staff/j.alglave/papers/asplos18.pdf

=C2=A0 =C2=A0 =C2=A0 =C2=A0 Backup material:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 http://diy.inria.fr/linux/

o=C2=A0 =C2=A0 =C2=A0 =C2=A0Who's afraid of a big bad optimizing compil= er?
=C2=A0 =C2=A0 =C2=A0 =C2=A0 https://lwn.net/Articles/793253/

o=C2=A0 =C2=A0 =C2=A0 =C2=A0Calibrating your fear of big bad optimizing com= pilers
=C2=A0 =C2=A0 =C2=A0 =C2=A0 https://lwn.net/Articles/799218/

=C2=A0 =C2=A0 =C2=A0 =C2=A0 These last two justify use of normal C-language= assignment
=C2=A0 =C2=A0 =C2=A0 =C2=A0 statements to initialize and access data refere= nced by
=C2=A0 =C2=A0 =C2=A0 =C2=A0 RCU-protected pointers.

There is a large body of litmus tests (thousands of them) here:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 https://github.com/paulmckrcu/litmus=

Many of these litmus tests involve RCU, and these can be located by
search for files containing rcu_read_lock(), rcu_read_unlock(),
synchronize_rcu(), and so on.

Or were you looking for something else?

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Thanx, Paul

> Thanks,
>
> Mathieu
>
> >> as a starting point.
>
> >> Thanks,
>
> >> Mathieu
>
> >>> I know there are some consistency models in the database = area (such as PRAM,
> >>> Read Uncommitted, etc) from [ https://jepsen.io/consi= stency |
> >>> https://jepsen.io/consistency ] and [1].
> >>> How does RCU related to those consistency models?
>
> >>> I also found some comments online (One key distinction is= that both MVCC and RLU
> >>> provide much stronger consistency guarantees to readers t= han does RCU ...) ( [
> >>> https://lwn.net/Articles/777036/ | http= s://lwn.net/Articles/777036/ ] ).
> >>> I do not understand how we reason/dresibe/compare the con= sistency guarantees. (
> >>> I even do not know what consistency guarantees provided b= y RCU formally)
> >>> Could someone explain this to me?
>
> >>> [1] Bailis, P., Davidson, A., Fekete, A., Ghodsi, A., Hel= lerstein, J. M., &
> >>> Stoica, I. (2013). Highly available transactions: Virtues= and limitations.
> >>> Proceedings of the VLDB Endowment, 7(3), 181-192.
>
> >>> Thanks
> >>> Yuxin
>
> >>> _______________________________________________
> >>> lttng-dev mailing list
> >>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org ]
> >>> [ https://lists.lttng.or= g/cgi-bin/mailman/listinfo/lttng-dev |
> >>> https://lists.lttng.org/= cgi-bin/mailman/listinfo/lttng-dev ]
>
> >> --
> >> Mathieu Desnoyers
> >> EfficiOS Inc.
> >> [ http://www.efficios.com/ | http://www.efficios.com ] >
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
--000000000000995117059911d988-- --===============2512842037105845090== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev --===============2512842037105845090==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Yuxin Ren Subject: Re: RCU consistency guarantees Date: Sat, 7 Dec 2019 15:04:42 -0500 Message-ID: References: <194534011.751.1575629349181.JavaMail.zimbra@efficios.com> <512711764.1078.1575647945136.JavaMail.zimbra@efficios.com> <20191206163052.GG2889@paulmck-ThinkPad-P72> <20191207063730.GN2889@paulmck-ThinkPad-P72> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============4566789761914688777==" Return-path: Received: from mail-lj1-x241.google.com (mail-lj1-x241.google.com [IPv6:2a00:1450:4864:20::241]) by lists.lttng.org (Postfix) with ESMTPS id 47VgRG6dV4z131q for ; Sat, 7 Dec 2019 15:04:58 -0500 (EST) Received: by mail-lj1-x241.google.com with SMTP id 21so11328758ljr.0 for ; Sat, 07 Dec 2019 12:04:58 -0800 (PST) In-Reply-To: <20191207063730.GN2889@paulmck-ThinkPad-P72> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" To: paulmck@kernel.org Cc: lttng-dev List-Id: lttng-dev@lists.lttng.org --===============4566789761914688777== Content-Type: multipart/alternative; boundary="000000000000301204059922ad8b" --000000000000301204059922ad8b Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks a lot for your help. I have some questions below. On Sat, Dec 7, 2019 at 1:37 AM Paul E. McKenney wrote: > On Fri, Dec 06, 2019 at 07:00:13PM -0500, Yuxin Ren wrote: > > Thanks so much for your great help. > > I definitely will look at those resources and papers! > > > > One more thing that I am confused > > As I mentioned earlier, someone said One key distinction is that both > MVCC > > and RLU provide much stronger consistency guarantees to readers than do= es > > RCU ...) (https://lwn.net/Articles/777036/). > > That someone was in fact me. ;-) > > > I am not sure if the above statement is correct or not. But in general, > > How can we compare RCU consistency guarantees to other techniques (such > as > > RLU)? > > How to reason about which one has stronger or weaker guarantees? > > I suggest starting from the use case. For concreteness, let's assume > that we are using a hash table. At one extreme, imagine a use case in > which each event makes exactly one hash-table operation. No information > is carried from one event to the next. (This might well be the case > for simple web server.) Such a use case cannot tell the difference > between RCU on the one hand and MVCC/RLU on the other. > > At the other extreme, suppose that each event either accesses or updates > multiple entries in the hash table. In this case, MVCC/RLU will rule > out outcomes that RCU would permit. For example, suppose we had four > events accessing two different elements in different buckets of the > hash table: > > E1: Adds 32 to the hash table. > E2: Adds 1729 to the hash table. > E3: Within a read-side critical section, looks up 32 then 1729. > E4: Within a read-side critical section, looks up 1729 then 32. > > Given either MVCC or RLU, it will not be possible for E3 to find 32 but > not 1729 and at the same time for E4 to find 1729 but not 32. Given RCU, > this outcome is possible. > When you say "Within a read-side section", do you mean within a single same read section? such as > read_lock() > lookup(32) > lookup(1729) > read_unlock() How about putting two lookups into two read-side sections? Do we still have the problem, then? > read_lock() > lookup(32) > read_unlock() > read_lock() > lookup(1729) > read_unlock() Could you kindly give me more clues why RCU can see such reorder, while RLU can prevent it? > This is because MVCC and RLU provide readers a consistent view of > the updates, and RCU does not. Of course, it is often the case that a > consistent view is not needed, in which case the MVCC and RLU guarantees > are incurring read-side overhead for no reason. But if the use case > requires consistent readers, RCU is not an option. > > The reason a consistent view is not always needed is that speed-of-light > delays make it impossible to provide a consistent view of the outside > world. In the common case where the use case interacts with the > outside world, the algorithms absolutely must be designed to tolerate > inconsistency, which opens the door to things like RCU. > I am confused here. I think speed-of-light delays happen everywhere, not only bound to RCU, but also any other synchronization approach (such RLU). If so, how do others (RLU) provide consistent views? Thanks for your education. Yuxin > > Thanx, Paul > > > Thanks > > Yuxin > > > > On Fri, Dec 6, 2019 at 11:30 AM Paul E. McKenney > wrote: > > > > > On Fri, Dec 06, 2019 at 10:59:05AM -0500, Mathieu Desnoyers wrote: > > > > ----- On Dec 6, 2019, at 3:51 PM, Yuxin Ren > wrote: > > > > > > > > > On Fri, Dec 6, 2019 at 5:49 AM Mathieu Desnoyers < [ > > > > > mailto:mathieu.desnoyers@efficios.com | > mathieu.desnoyers@efficios.com > > > ] > > > > > > wrote: > > > > > > > > >> ----- On Dec 5, 2019, at 8:17 PM, Yuxin Ren < [ mailto: > > > ryx@gwmail.gwu.edu | > > > > >> ryx@gwmail.gwu.edu ] > wrote: > > > > > > > > >>> Hi, > > > > >>> I am a student, and learning RCU now, but still know very littl= e > > > about it. > > > > >>> Are there any documents/papers/materials which (in)formally > define > > > and explain > > > > >>> RCU consistency guarantees? > > > > > > > > >> You may want to have a look at > > > > > > > > >> User-Level Implementations of Read-Copy Update > > > > >> Article in IEEE Transactions on Parallel and Distributed Systems > > > 23(2):375 - 382 > > > > >> =C2=B7 March 2012 > > > > > > > > > Thanks for your info. > > > > > However, I do not think URCU talks about any consistency model > > > formally. > > > > > > > > > From previous communication with Paul, he said RCU is not designe= d > for > > > > > linearizability, and it is totally acceptable that RCU is not > > > linearizable. > > > > > However, I am curious how to accurately/formally Characterize RCU > > > consistency > > > > > model/guarantees > > > > > > > > Adding Paul E. McKenney in CC. > > > > > > > > I am referring to the section "Overview of RCU semantics" in the > paper. > > > Not sure it has the level of > > > > formality you are looking for though. Paul, do you have pointers to > > > additional material ? > > > > > > Indeed I do! The Linux kernel memory model (LKMM) includes RCU. It = is > > > in tools/memory-model in recent kernel source trees, which includes > > > documentation. This is an executable model, which means that you > > > can create litmus tests and have the model formally and automatically > > > evaluate them. > > > > > > There are also a number of publications covering LKMM: > > > > > > o A formal kernel memory-ordering model > > > https://lwn.net/Articles/718628/ > > > https://lwn.net/Articles/720550/ > > > > > > These cover the release stores and dependency ordering that > > > provide RCU's publish-subscribe guarantees. > > > > > > Backup material here: > > > > > > > > > > https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinuxM= M/ > > > > > > With these two likely being of particular interest: > > > > > > > > > > https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinuxM= M/RCUguarantees.html > > > > > > > https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinuxM= M/srcu.html > > > > > > o Frightening Small Children and Disconcerting Grown-ups: > > > Concurrency in the Linux Kernel > > > https://dl.acm.org/citation.cfm?id=3D3177156 > > > http://www0.cs.ucl.ac.uk/staff/j.alglave/papers/asplos18.pdf > > > > > > Backup material: > > > > > > http://diy.inria.fr/linux/ > > > > > > o Who's afraid of a big bad optimizing compiler? > > > https://lwn.net/Articles/793253/ > > > > > > o Calibrating your fear of big bad optimizing compilers > > > https://lwn.net/Articles/799218/ > > > > > > These last two justify use of normal C-language assignment > > > statements to initialize and access data referenced by > > > RCU-protected pointers. > > > > > > There is a large body of litmus tests (thousands of them) here: > > > > > > https://github.com/paulmckrcu/litmus > > > > > > Many of these litmus tests involve RCU, and these can be located by > > > search for files containing rcu_read_lock(), rcu_read_unlock(), > > > synchronize_rcu(), and so on. > > > > > > Or were you looking for something else? > > > > > > Thanx, Paul > > > > > > > Thanks, > > > > > > > > Mathieu > > > > > > > > >> as a starting point. > > > > > > > > >> Thanks, > > > > > > > > >> Mathieu > > > > > > > > >>> I know there are some consistency models in the database area > (such > > > as PRAM, > > > > >>> Read Uncommitted, etc) from [ https://jepsen.io/consistency | > > > > >>> https://jepsen.io/consistency ] and [1]. > > > > >>> How does RCU related to those consistency models? > > > > > > > > >>> I also found some comments online (One key distinction is that > both > > > MVCC and RLU > > > > >>> provide much stronger consistency guarantees to readers than do= es > > > RCU ...) ( [ > > > > >>> https://lwn.net/Articles/777036/ | > https://lwn.net/Articles/777036/ > > > ] ). > > > > >>> I do not understand how we reason/dresibe/compare the consisten= cy > > > guarantees. ( > > > > >>> I even do not know what consistency guarantees provided by RCU > > > formally) > > > > >>> Could someone explain this to me? > > > > > > > > >>> [1] Bailis, P., Davidson, A., Fekete, A., Ghodsi, A., > Hellerstein, > > > J. M., & > > > > >>> Stoica, I. (2013). Highly available transactions: Virtues and > > > limitations. > > > > >>> Proceedings of the VLDB Endowment, 7(3), 181-192. > > > > > > > > >>> Thanks > > > > >>> Yuxin > > > > > > > > >>> _______________________________________________ > > > > >>> lttng-dev mailing list > > > > >>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org = ] > > > > >>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev | > > > > >>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ] > > > > > > > > >> -- > > > > >> Mathieu Desnoyers > > > > >> EfficiOS Inc. > > > > >> [ http://www.efficios.com/ | http://www.efficios.com ] > > > > > > > > -- > > > > Mathieu Desnoyers > > > > EfficiOS Inc. > > > > http://www.efficios.com > > > > --000000000000301204059922ad8b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Thanks a lot for your help. I have some q= uestions below.

On Sat, Dec 7, 2019 at 1:37 AM Paul E. McKenney <paulmck@kernel.org> wrote:
<= blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-l= eft:1px solid rgb(204,204,204);padding-left:1ex">On Fri, Dec 06, 2019 at 07= :00:13PM -0500, Yuxin Ren wrote:
> Thanks so much for your great help.
> I definitely will look at those resources and papers!
>
> One more thing that I am confused
> As I mentioned earlier, someone said One key distinction is that both = MVCC
> and RLU provide much stronger consistency guarantees to readers than d= oes
> RCU ...) (https://lwn.net/Articles/777036/).

That someone was in fact me.=C2=A0 ;-)

> I am not sure if the above statement is correct or not. But in general= ,
> How can we compare RCU consistency guarantees to other techniques (suc= h as
> RLU)?
> How to reason about which one has stronger or weaker guarantees?

I suggest starting from the use case.=C2=A0 For concreteness, let's ass= ume
that we are using a hash table.=C2=A0 At one extreme, imagine a use case in=
which each event makes exactly one hash-table operation.=C2=A0 No informati= on
is carried from one event to the next.=C2=A0 (This might well be the case for simple web server.)=C2=A0 Such a use case cannot tell the difference between RCU on the one hand and MVCC/RLU on the other.

At the other extreme, suppose that each event either accesses or updates multiple entries in the hash table.=C2=A0 In this case, MVCC/RLU will rule<= br> out outcomes that RCU would permit.=C2=A0 For example, suppose we had four<= br> events accessing two different elements in different buckets of the
hash table:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 E1: Adds 32 to the hash table.
=C2=A0 =C2=A0 =C2=A0 =C2=A0 E2: Adds 1729 to the hash table.
=C2=A0 =C2=A0 =C2=A0 =C2=A0 E3: Within a read-side critical section, looks = up 32 then 1729.
=C2=A0 =C2=A0 =C2=A0 =C2=A0 E4: Within a read-side critical section, looks = up 1729 then 32.

Given either MVCC or RLU, it will not be possible for E3 to find 32 but
not 1729 and at the same time for E4 to find 1729 but not 32.=C2=A0 Given R= CU,
this outcome is possible.
When you say "Within a = read-side section", do you mean within a single same read section? suc= h as
read_lock()
= lookup(32)
lookup(1729)
read_unlock()

How about putting two lookups into two read-side sections? Do we still hav= e the problem, then?=C2=A0
read_lock()
lookup(32)
read_unlock()
read_lock()
loo= kup(1729)
read_unlock()

Could you kindly= give me more clues why RCU can see such reorder, while RLU can prevent it?=
=C2=A0
This is because MVCC and RLU provide readers a consistent view of
the updates, and RCU does not.=C2=A0 Of course, it is often the case that a=
consistent view is not needed, in which case the MVCC and RLU guarantees are incurring read-side overhead for no reason.=C2=A0 But if the use case requires consistent readers, RCU is not an option.

The reason a consistent view is not always needed is that speed-of-light delays make it impossible to provide a consistent view of the outside
world.=C2=A0 In the common case where the use case interacts with the
outside world, the algorithms absolutely must be designed to tolerate
inconsistency, which opens the door to things like RCU.

I am confused here. I think speed-of-light delays happen e= verywhere, not only bound to RCU, but also=C2=A0 any other synchronization= =C2=A0approach (such RLU).
If so, how do others (RLU) provide=C2= =A0consistent views?

Thanks for your education.
Yuxin

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Thanx, Paul

> Thanks
> Yuxin
>
> On Fri, Dec 6, 2019 at 11:30 AM Paul E. McKenney <paulmck@kernel.org> wrote: >
> > On Fri, Dec 06, 2019 at 10:59:05AM -0500, Mathieu Desnoyers wrote= :
> > > ----- On Dec 6, 2019, at 3:51 PM, Yuxin Ren <ryx@gwmail.gwu.edu> wrot= e:
> > >
> > > > On Fri, Dec 6, 2019 at 5:49 AM Mathieu Desnoyers < [=
> > > > mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficio= s.com
> > ] >
> > > > wrote:
> > >
> > > >> ----- On Dec 5, 2019, at 8:17 PM, Yuxin Ren < [ = mailto:
> > ryx@gwmai= l.gwu.edu |
> > > >> ryx@gwmail.gwu.edu ] > wrote:
> > >
> > > >>> Hi,
> > > >>> I am a student, and learning RCU now, but still= know very little
> > about it.
> > > >>> Are there any documents/papers/materials which = (in)formally define
> > and explain
> > > >>> RCU consistency guarantees?
> > >
> > > >> You may want to have a look at
> > >
> > > >> User-Level Implementations of Read-Copy Update
> > > >> Article in IEEE Transactions on Parallel and Distri= buted Systems
> > 23(2):375 - 382
> > > >> =C2=B7 March 2012
> > >
> > > > Thanks for your info.
> > > > However, I do not think URCU talks about any consistenc= y model
> > formally.
> > >
> > > > From previous communication with Paul, he said RCU is n= ot designed for
> > > > linearizability, and it is totally acceptable that RCU = is not
> > linearizable.
> > > > However, I am curious how to accurately/formally Charac= terize RCU
> > consistency
> > > > model/guarantees
> > >
> > > Adding Paul E. McKenney in CC.
> > >
> > > I am referring to the section "Overview of RCU semantic= s" in the paper.
> > Not sure it has the level of
> > > formality you are looking for though. Paul, do you have poin= ters to
> > additional material ?
> >
> > Indeed I do!=C2=A0 The Linux kernel memory model (LKMM) includes = RCU.=C2=A0 It is
> > in tools/memory-model in recent kernel source trees, which includ= es
> > documentation.=C2=A0 This is an executable model, which means tha= t you
> > can create litmus tests and have the model formally and automatic= ally
> > evaluate them.
> >
> > There are also a number of publications covering LKMM:
> >
> > o=C2=A0 =C2=A0 =C2=A0 =C2=A0A formal kernel memory-ordering model=
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0https://lwn.net/Articles/= 718628/
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0https://lwn.net/Articles/= 720550/
> >
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0These cover the release stores a= nd dependency ordering that
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0provide RCU's publish-subscr= ibe guarantees.
> >
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Backup material here:
> >
> >
> > https://mirrors= .edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinuxMM/
> >
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0With these two likely being of p= articular interest:
> >
> >
> > https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinux= MM/RCUguarantees.html
> >
> > https:= //mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinuxMM/srcu.h= tml
> >
> > o=C2=A0 =C2=A0 =C2=A0 =C2=A0Frightening Small Children and Discon= certing Grown-ups:
> > Concurrency in the Linux Kernel
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0https://dl.ac= m.org/citation.cfm?id=3D3177156
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0http://www0.cs.ucl.ac.uk/staff/j.alglave/papers/asplos18.pdf
> >
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Backup material:
> >
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0http://diy.inria.fr/linux/<= br> > >
> > o=C2=A0 =C2=A0 =C2=A0 =C2=A0Who's afraid of a big bad optimiz= ing compiler?
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0https://lwn.net/Articles/= 793253/
> >
> > o=C2=A0 =C2=A0 =C2=A0 =C2=A0Calibrating your fear of big bad opti= mizing compilers
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0https://lwn.net/Articles/= 799218/
> >
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0These last two justify use of no= rmal C-language assignment
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0statements to initialize and acc= ess data referenced by
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0RCU-protected pointers.
> >
> > There is a large body of litmus tests (thousands of them) here: > >
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0https://github.com/pa= ulmckrcu/litmus
> >
> > Many of these litmus tests involve RCU, and these can be located = by
> > search for files containing rcu_read_lock(), rcu_read_unlock(), > > synchronize_rcu(), and so on.
> >
> > Or were you looking for something else?
> >
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Thanx, Paul > >
> > > Thanks,
> > >
> > > Mathieu
> > >
> > > >> as a starting point.
> > >
> > > >> Thanks,
> > >
> > > >> Mathieu
> > >
> > > >>> I know there are some consistency models in the= database area (such
> > as PRAM,
> > > >>> Read Uncommitted, etc) from [ https://jepse= n.io/consistency |
> > > >>> https://jepsen.io/consistency ] and [= 1].
> > > >>> How does RCU related to those consistency model= s?
> > >
> > > >>> I also found some comments online (One key dist= inction is that both
> > MVCC and RLU
> > > >>> provide much stronger consistency guarantees to= readers than does
> > RCU ...) ( [
> > > >>> https://lwn.net/Articles/777036/ | <= a href=3D"https://lwn.net/Articles/777036/" rel=3D"noreferrer" target=3D"_b= lank">https://lwn.net/Articles/777036/
> > ] ).
> > > >>> I do not understand how we reason/dresibe/compa= re the consistency
> > guarantees. (
> > > >>> I even do not know what consistency guarantees = provided by RCU
> > formally)
> > > >>> Could someone explain this to me?
> > >
> > > >>> [1] Bailis, P., Davidson, A., Fekete, A., Ghods= i, A., Hellerstein,
> > J. M., &
> > > >>> Stoica, I. (2013). Highly available transaction= s: Virtues and
> > limitations.
> > > >>> Proceedings of the VLDB Endowment, 7(3), 181-19= 2.
> > >
> > > >>> Thanks
> > > >>> Yuxin
> > >
> > > >>> _______________________________________________=
> > > >>> lttng-dev mailing list
> > > >>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org = ]
> > > >>> [ https://list= s.lttng.org/cgi-bin/mailman/listinfo/lttng-dev |
> > > >>> https://lists.= lttng.org/cgi-bin/mailman/listinfo/lttng-dev ]
> > >
> > > >> --
> > > >> Mathieu Desnoyers
> > > >> EfficiOS Inc.
> > > >> [ http://www.efficios.com/ | http://www.efficios.c= om ]
> > >
> > > --
> > > Mathieu Desnoyers
> > > EfficiOS Inc.
> > > http://www.efficios.com
> >
--000000000000301204059922ad8b-- --===============4566789761914688777== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev --===============4566789761914688777==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Paul E. McKenney" Subject: Re: RCU consistency guarantees Date: Fri, 6 Dec 2019 22:37:30 -0800 Message-ID: <20191207063730.GN2889__40921.0663101234$1575916624$gmane$org@paulmck-ThinkPad-P72> References: <194534011.751.1575629349181.JavaMail.zimbra@efficios.com> <512711764.1078.1575647945136.JavaMail.zimbra@efficios.com> <20191206163052.GG2889@paulmck-ThinkPad-P72> Reply-To: paulmck@kernel.org Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by lists.lttng.org (Postfix) with ESMTPS id 47VKl03kFpz124g for ; Sat, 7 Dec 2019 01:47:23 -0500 (EST) Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" To: Yuxin Ren Cc: lttng-dev List-Id: lttng-dev@lists.lttng.org On Fri, Dec 06, 2019 at 07:00:13PM -0500, Yuxin Ren wrote: > Thanks so much for your great help. > I definitely will look at those resources and papers! > = > One more thing that I am confused > As I mentioned earlier, someone said One key distinction is that both MVCC > and RLU provide much stronger consistency guarantees to readers than does > RCU ...) (https://lwn.net/Articles/777036/). That someone was in fact me. ;-) > I am not sure if the above statement is correct or not. But in general, > How can we compare RCU consistency guarantees to other techniques (such as > RLU)? > How to reason about which one has stronger or weaker guarantees? I suggest starting from the use case. For concreteness, let's assume that we are using a hash table. At one extreme, imagine a use case in which each event makes exactly one hash-table operation. No information is carried from one event to the next. (This might well be the case for simple web server.) Such a use case cannot tell the difference between RCU on the one hand and MVCC/RLU on the other. At the other extreme, suppose that each event either accesses or updates multiple entries in the hash table. In this case, MVCC/RLU will rule out outcomes that RCU would permit. For example, suppose we had four events accessing two different elements in different buckets of the hash table: E1: Adds 32 to the hash table. E2: Adds 1729 to the hash table. E3: Within a read-side critical section, looks up 32 then 1729. E4: Within a read-side critical section, looks up 1729 then 32. Given either MVCC or RLU, it will not be possible for E3 to find 32 but not 1729 and at the same time for E4 to find 1729 but not 32. Given RCU, this outcome is possible. This is because MVCC and RLU provide readers a consistent view of the updates, and RCU does not. Of course, it is often the case that a consistent view is not needed, in which case the MVCC and RLU guarantees are incurring read-side overhead for no reason. But if the use case requires consistent readers, RCU is not an option. The reason a consistent view is not always needed is that speed-of-light delays make it impossible to provide a consistent view of the outside world. In the common case where the use case interacts with the outside world, the algorithms absolutely must be designed to tolerate inconsistency, which opens the door to things like RCU. Thanx, Paul > Thanks > Yuxin > = > On Fri, Dec 6, 2019 at 11:30 AM Paul E. McKenney wro= te: > = > > On Fri, Dec 06, 2019 at 10:59:05AM -0500, Mathieu Desnoyers wrote: > > > ----- On Dec 6, 2019, at 3:51 PM, Yuxin Ren wrot= e: > > > > > > > On Fri, Dec 6, 2019 at 5:49 AM Mathieu Desnoyers < [ > > > > mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.= com > > ] > > > > > wrote: > > > > > > >> ----- On Dec 5, 2019, at 8:17 PM, Yuxin Ren < [ mailto: > > ryx@gwmail.gwu.edu | > > > >> ryx@gwmail.gwu.edu ] > wrote: > > > > > > >>> Hi, > > > >>> I am a student, and learning RCU now, but still know very little > > about it. > > > >>> Are there any documents/papers/materials which (in)formally define > > and explain > > > >>> RCU consistency guarantees? > > > > > > >> You may want to have a look at > > > > > > >> User-Level Implementations of Read-Copy Update > > > >> Article in IEEE Transactions on Parallel and Distributed Systems > > 23(2):375 - 382 > > > >> =B7 March 2012 > > > > > > > Thanks for your info. > > > > However, I do not think URCU talks about any consistency model > > formally. > > > > > > > From previous communication with Paul, he said RCU is not designed = for > > > > linearizability, and it is totally acceptable that RCU is not > > linearizable. > > > > However, I am curious how to accurately/formally Characterize RCU > > consistency > > > > model/guarantees > > > > > > Adding Paul E. McKenney in CC. > > > > > > I am referring to the section "Overview of RCU semantics" in the pape= r. > > Not sure it has the level of > > > formality you are looking for though. Paul, do you have pointers to > > additional material ? > > > > Indeed I do! The Linux kernel memory model (LKMM) includes RCU. It is > > in tools/memory-model in recent kernel source trees, which includes > > documentation. This is an executable model, which means that you > > can create litmus tests and have the model formally and automatically > > evaluate them. > > > > There are also a number of publications covering LKMM: > > > > o A formal kernel memory-ordering model > > https://lwn.net/Articles/718628/ > > https://lwn.net/Articles/720550/ > > > > These cover the release stores and dependency ordering that > > provide RCU's publish-subscribe guarantees. > > > > Backup material here: > > > > > > https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinu= xMM/ > > > > With these two likely being of particular interest: > > > > > > https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinu= xMM/RCUguarantees.html > > > > https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinu= xMM/srcu.html > > > > o Frightening Small Children and Disconcerting Grown-ups: > > Concurrency in the Linux Kernel > > https://dl.acm.org/citation.cfm?id=3D3177156 > > http://www0.cs.ucl.ac.uk/staff/j.alglave/papers/asplos18.pdf > > > > Backup material: > > > > http://diy.inria.fr/linux/ > > > > o Who's afraid of a big bad optimizing compiler? > > https://lwn.net/Articles/793253/ > > > > o Calibrating your fear of big bad optimizing compilers > > https://lwn.net/Articles/799218/ > > > > These last two justify use of normal C-language assignment > > statements to initialize and access data referenced by > > RCU-protected pointers. > > > > There is a large body of litmus tests (thousands of them) here: > > > > https://github.com/paulmckrcu/litmus > > > > Many of these litmus tests involve RCU, and these can be located by > > search for files containing rcu_read_lock(), rcu_read_unlock(), > > synchronize_rcu(), and so on. > > > > Or were you looking for something else? > > > > Thanx, Paul > > > > > Thanks, > > > > > > Mathieu > > > > > > >> as a starting point. > > > > > > >> Thanks, > > > > > > >> Mathieu > > > > > > >>> I know there are some consistency models in the database area (su= ch > > as PRAM, > > > >>> Read Uncommitted, etc) from [ https://jepsen.io/consistency | > > > >>> https://jepsen.io/consistency ] and [1]. > > > >>> How does RCU related to those consistency models? > > > > > > >>> I also found some comments online (One key distinction is that bo= th > > MVCC and RLU > > > >>> provide much stronger consistency guarantees to readers than does > > RCU ...) ( [ > > > >>> https://lwn.net/Articles/777036/ | https://lwn.net/Articles/77703= 6/ > > ] ). > > > >>> I do not understand how we reason/dresibe/compare the consistency > > guarantees. ( > > > >>> I even do not know what consistency guarantees provided by RCU > > formally) > > > >>> Could someone explain this to me? > > > > > > >>> [1] Bailis, P., Davidson, A., Fekete, A., Ghodsi, A., Hellerstein, > > J. M., & > > > >>> Stoica, I. (2013). Highly available transactions: Virtues and > > limitations. > > > >>> Proceedings of the VLDB Endowment, 7(3), 181-192. > > > > > > >>> Thanks > > > >>> Yuxin > > > > > > >>> _______________________________________________ > > > >>> lttng-dev mailing list > > > >>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org ] > > > >>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev | > > > >>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ] > > > > > > >> -- > > > >> Mathieu Desnoyers > > > >> EfficiOS Inc. > > > >> [ http://www.efficios.com/ | http://www.efficios.com ] > > > > > > -- > > > Mathieu Desnoyers > > > EfficiOS Inc. > > > http://www.efficios.com > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Paul E. McKenney" Subject: Re: RCU consistency guarantees Date: Sat, 7 Dec 2019 14:42:32 -0800 Message-ID: <20191207224232.GR2889__41236.9331739441$1575916661$gmane$org@paulmck-ThinkPad-P72> References: <194534011.751.1575629349181.JavaMail.zimbra@efficios.com> <512711764.1078.1575647945136.JavaMail.zimbra@efficios.com> <20191206163052.GG2889@paulmck-ThinkPad-P72> <20191207063730.GN2889@paulmck-ThinkPad-P72> Reply-To: paulmck@kernel.org Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by lists.lttng.org (Postfix) with ESMTPS id 47Vkx60Mwlz12Yx for ; Sat, 7 Dec 2019 17:42:33 -0500 (EST) Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" To: Yuxin Ren Cc: lttng-dev List-Id: lttng-dev@lists.lttng.org On Sat, Dec 07, 2019 at 03:04:42PM -0500, Yuxin Ren wrote: > Thanks a lot for your help. I have some questions below. > = > On Sat, Dec 7, 2019 at 1:37 AM Paul E. McKenney wrot= e: > = > > On Fri, Dec 06, 2019 at 07:00:13PM -0500, Yuxin Ren wrote: > > > Thanks so much for your great help. > > > I definitely will look at those resources and papers! > > > > > > One more thing that I am confused > > > As I mentioned earlier, someone said One key distinction is that both > > MVCC > > > and RLU provide much stronger consistency guarantees to readers than = does > > > RCU ...) (https://lwn.net/Articles/777036/). > > > > That someone was in fact me. ;-) > > > > > I am not sure if the above statement is correct or not. But in genera= l, > > > How can we compare RCU consistency guarantees to other techniques (su= ch > > as > > > RLU)? > > > How to reason about which one has stronger or weaker guarantees? > > > > I suggest starting from the use case. For concreteness, let's assume > > that we are using a hash table. At one extreme, imagine a use case in > > which each event makes exactly one hash-table operation. No information > > is carried from one event to the next. (This might well be the case > > for simple web server.) Such a use case cannot tell the difference > > between RCU on the one hand and MVCC/RLU on the other. > > > > At the other extreme, suppose that each event either accesses or updates > > multiple entries in the hash table. In this case, MVCC/RLU will rule > > out outcomes that RCU would permit. For example, suppose we had four > > events accessing two different elements in different buckets of the > > hash table: > > > > E1: Adds 32 to the hash table. > > E2: Adds 1729 to the hash table. > > E3: Within a read-side critical section, looks up 32 then 1729. > > E4: Within a read-side critical section, looks up 1729 then 32. > > > > Given either MVCC or RLU, it will not be possible for E3 to find 32 but > > not 1729 and at the same time for E4 to find 1729 but not 32. Given RC= U, > > this outcome is possible. > > > When you say "Within a read-side section", do you mean within a single sa= me > read section? such as > = > > read_lock() > > lookup(32) > > lookup(1729) > > read_unlock() > = > = > How about putting two lookups into two read-side sections? Do we still ha= ve > the problem, then? > = > > read_lock() > > lookup(32) > > read_unlock() > > read_lock() > > lookup(1729) > > read_unlock() Without in any way agreeing with your characterization of this as a problem, because rcu_read_lock() and rcu_read_unlock() provide absolutely no ordering guarantees in the absence of a grace period, any non-grace-period-related reordering that can happen with a single RCU read-side critical section can also happen when that critical section is split in two as you have done above. > Could you kindly give me more clues why RCU can see such reorder, while R= LU > can prevent it? Here are minimal C-language implementations for RCU that can (and are) actually used: #define rcu_read_lock() #define rcu_read_unlock() Please compare these to the read-side markers presented in the RLU paper, and then tell me your thoughts on the answer to your question. ;-) > > This is because MVCC and RLU provide readers a consistent view of > > the updates, and RCU does not. Of course, it is often the case that a > > consistent view is not needed, in which case the MVCC and RLU guarantees > > are incurring read-side overhead for no reason. But if the use case > > requires consistent readers, RCU is not an option. > > > > The reason a consistent view is not always needed is that speed-of-light > > delays make it impossible to provide a consistent view of the outside > > world. In the common case where the use case interacts with the > > outside world, the algorithms absolutely must be designed to tolerate > > inconsistency, which opens the door to things like RCU. > = > I am confused here. I think speed-of-light delays happen everywhere, not > only bound to RCU, but also any other synchronization approach (such RLU= ). > If so, how do others (RLU) provide consistent views? You just stated the answer. Now it is only necessary for you to invest the time, effort, and thought to fully understand it. To help with this, the following paragraph provides another hint: Yes, you are quite right, speed-of-light delays between the outside world and the computer affect RLU just as surely as they do RCU. This means that the additional consistency guarantees provided by RLU must necessarily be limited to the confines of the computer system in question. Taking this one step further, there are also speed-of-light delays within the computer. Therefore, in the general case, RLU can provide its consistency guarantees, even within the confines of the computer system, only at the expense of incurring delays. Because RCU does not provide RLU's consistency guarantees, it need not incur RLU's delays. This is not a new line of reasoning, see for example: @article{Herlihy:1996:LCN:1063369.1063372 ,author =3D {Herlihy, Maurice and Shavit, Nir and Waarts, Orli} ,title =3D {Linearizable counting networks} ,journal =3D {Distrib. Comput.} ,volume =3D {9} ,issue =3D {4} ,month =3D {February} ,year =3D {1996} ,issn =3D {0178-2770} ,pages =3D {193--203} ,numpages =3D {11} ,url =3D {http://portal.acm.org/citation.cfm?id=3D1063369.1063372} ,doi =3D {10.1007/s004460050019} ,acmid =3D {1063372} ,publisher =3D {Springer-Verlag} ,address =3D {London, UK} ,keywords =3D {concurrency, contention, counting networks, data structures,= linearizability} } Thanx, Paul > Thanks for your education. > Yuxin > = > > > > Thanx, Paul > > > > > Thanks > > > Yuxin > > > > > > On Fri, Dec 6, 2019 at 11:30 AM Paul E. McKenney > > wrote: > > > > > > > On Fri, Dec 06, 2019 at 10:59:05AM -0500, Mathieu Desnoyers wrote: > > > > > ----- On Dec 6, 2019, at 3:51 PM, Yuxin Ren > > wrote: > > > > > > > > > > > On Fri, Dec 6, 2019 at 5:49 AM Mathieu Desnoyers < [ > > > > > > mailto:mathieu.desnoyers@efficios.com | > > mathieu.desnoyers@efficios.com > > > > ] > > > > > > > wrote: > > > > > > > > > > >> ----- On Dec 5, 2019, at 8:17 PM, Yuxin Ren < [ mailto: > > > > ryx@gwmail.gwu.edu | > > > > > >> ryx@gwmail.gwu.edu ] > wrote: > > > > > > > > > > >>> Hi, > > > > > >>> I am a student, and learning RCU now, but still know very lit= tle > > > > about it. > > > > > >>> Are there any documents/papers/materials which (in)formally > > define > > > > and explain > > > > > >>> RCU consistency guarantees? > > > > > > > > > > >> You may want to have a look at > > > > > > > > > > >> User-Level Implementations of Read-Copy Update > > > > > >> Article in IEEE Transactions on Parallel and Distributed Syste= ms > > > > 23(2):375 - 382 > > > > > >> =B7 March 2012 > > > > > > > > > > > Thanks for your info. > > > > > > However, I do not think URCU talks about any consistency model > > > > formally. > > > > > > > > > > > From previous communication with Paul, he said RCU is not desig= ned > > for > > > > > > linearizability, and it is totally acceptable that RCU is not > > > > linearizable. > > > > > > However, I am curious how to accurately/formally Characterize R= CU > > > > consistency > > > > > > model/guarantees > > > > > > > > > > Adding Paul E. McKenney in CC. > > > > > > > > > > I am referring to the section "Overview of RCU semantics" in the > > paper. > > > > Not sure it has the level of > > > > > formality you are looking for though. Paul, do you have pointers = to > > > > additional material ? > > > > > > > > Indeed I do! The Linux kernel memory model (LKMM) includes RCU. I= t is > > > > in tools/memory-model in recent kernel source trees, which includes > > > > documentation. This is an executable model, which means that you > > > > can create litmus tests and have the model formally and automatical= ly > > > > evaluate them. > > > > > > > > There are also a number of publications covering LKMM: > > > > > > > > o A formal kernel memory-ordering model > > > > https://lwn.net/Articles/718628/ > > > > https://lwn.net/Articles/720550/ > > > > > > > > These cover the release stores and dependency ordering that > > > > provide RCU's publish-subscribe guarantees. > > > > > > > > Backup material here: > > > > > > > > > > > > > > https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinu= xMM/ > > > > > > > > With these two likely being of particular interest: > > > > > > > > > > > > > > https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinu= xMM/RCUguarantees.html > > > > > > > > > > https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinu= xMM/srcu.html > > > > > > > > o Frightening Small Children and Disconcerting Grown-ups: > > > > Concurrency in the Linux Kernel > > > > https://dl.acm.org/citation.cfm?id=3D3177156 > > > > http://www0.cs.ucl.ac.uk/staff/j.alglave/papers/asplos18.pdf > > > > > > > > Backup material: > > > > > > > > http://diy.inria.fr/linux/ > > > > > > > > o Who's afraid of a big bad optimizing compiler? > > > > https://lwn.net/Articles/793253/ > > > > > > > > o Calibrating your fear of big bad optimizing compilers > > > > https://lwn.net/Articles/799218/ > > > > > > > > These last two justify use of normal C-language assignment > > > > statements to initialize and access data referenced by > > > > RCU-protected pointers. > > > > > > > > There is a large body of litmus tests (thousands of them) here: > > > > > > > > https://github.com/paulmckrcu/litmus > > > > > > > > Many of these litmus tests involve RCU, and these can be located by > > > > search for files containing rcu_read_lock(), rcu_read_unlock(), > > > > synchronize_rcu(), and so on. > > > > > > > > Or were you looking for something else? > > > > > > > > Thanx, Paul > > > > > > > > > Thanks, > > > > > > > > > > Mathieu > > > > > > > > > > >> as a starting point. > > > > > > > > > > >> Thanks, > > > > > > > > > > >> Mathieu > > > > > > > > > > >>> I know there are some consistency models in the database area > > (such > > > > as PRAM, > > > > > >>> Read Uncommitted, etc) from [ https://jepsen.io/consistency | > > > > > >>> https://jepsen.io/consistency ] and [1]. > > > > > >>> How does RCU related to those consistency models? > > > > > > > > > > >>> I also found some comments online (One key distinction is that > > both > > > > MVCC and RLU > > > > > >>> provide much stronger consistency guarantees to readers than = does > > > > RCU ...) ( [ > > > > > >>> https://lwn.net/Articles/777036/ | > > https://lwn.net/Articles/777036/ > > > > ] ). > > > > > >>> I do not understand how we reason/dresibe/compare the consist= ency > > > > guarantees. ( > > > > > >>> I even do not know what consistency guarantees provided by RCU > > > > formally) > > > > > >>> Could someone explain this to me? > > > > > > > > > > >>> [1] Bailis, P., Davidson, A., Fekete, A., Ghodsi, A., > > Hellerstein, > > > > J. M., & > > > > > >>> Stoica, I. (2013). Highly available transactions: Virtues and > > > > limitations. > > > > > >>> Proceedings of the VLDB Endowment, 7(3), 181-192. > > > > > > > > > > >>> Thanks > > > > > >>> Yuxin > > > > > > > > > > >>> _______________________________________________ > > > > > >>> lttng-dev mailing list > > > > > >>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.or= g ] > > > > > >>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev | > > > > > >>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ] > > > > > > > > > > >> -- > > > > > >> Mathieu Desnoyers > > > > > >> EfficiOS Inc. > > > > > >> [ http://www.efficios.com/ | http://www.efficios.com ] > > > > > > > > > > -- > > > > > Mathieu Desnoyers > > > > > EfficiOS Inc. > > > > > http://www.efficios.com > > > > > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Yuxin Ren Subject: Re: RCU consistency guarantees Date: Sat, 14 Dec 2019 01:31:31 -0500 Message-ID: References: <194534011.751.1575629349181.JavaMail.zimbra@efficios.com> <512711764.1078.1575647945136.JavaMail.zimbra@efficios.com> <20191206163052.GG2889@paulmck-ThinkPad-P72> <20191207063730.GN2889@paulmck-ThinkPad-P72> <20191207224232.GR2889@paulmck-ThinkPad-P72> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2564164649379268757==" Return-path: Received: from mail-lj1-x22f.google.com (mail-lj1-x22f.google.com [IPv6:2a00:1450:4864:20::22f]) by lists.lttng.org (Postfix) with ESMTPS id 47Zd3j6rR0z1873 for ; Sat, 14 Dec 2019 01:31:45 -0500 (EST) Received: by mail-lj1-x22f.google.com with SMTP id k8so1093585ljh.5 for ; Fri, 13 Dec 2019 22:31:45 -0800 (PST) In-Reply-To: <20191207224232.GR2889@paulmck-ThinkPad-P72> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" To: paulmck@kernel.org Cc: lttng-dev List-Id: lttng-dev@lists.lttng.org --===============2564164649379268757== Content-Type: multipart/alternative; boundary="000000000000ba52b10599a421a6" --000000000000ba52b10599a421a6 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Paul On Sat, Dec 7, 2019 at 5:42 PM Paul E. McKenney wrote: > On Sat, Dec 07, 2019 at 03:04:42PM -0500, Yuxin Ren wrote: > > Thanks a lot for your help. I have some questions below. > > > > On Sat, Dec 7, 2019 at 1:37 AM Paul E. McKenney > wrote: > > > > > On Fri, Dec 06, 2019 at 07:00:13PM -0500, Yuxin Ren wrote: > > > > Thanks so much for your great help. > > > > I definitely will look at those resources and papers! > > > > > > > > One more thing that I am confused > > > > As I mentioned earlier, someone said One key distinction is that bo= th > > > MVCC > > > > and RLU provide much stronger consistency guarantees to readers tha= n > does > > > > RCU ...) (https://lwn.net/Articles/777036/). > > > > > > That someone was in fact me. ;-) > > > > > > > I am not sure if the above statement is correct or not. But in > general, > > > > How can we compare RCU consistency guarantees to other techniques > (such > > > as > > > > RLU)? > > > > How to reason about which one has stronger or weaker guarantees? > > > > > > I suggest starting from the use case. For concreteness, let's assume > > > that we are using a hash table. At one extreme, imagine a use case i= n > > > which each event makes exactly one hash-table operation. No > information > > > is carried from one event to the next. (This might well be the case > > > for simple web server.) Such a use case cannot tell the difference > > > between RCU on the one hand and MVCC/RLU on the other. > > > > > > At the other extreme, suppose that each event either accesses or > updates > > > multiple entries in the hash table. In this case, MVCC/RLU will rule > > > out outcomes that RCU would permit. For example, suppose we had four > > > events accessing two different elements in different buckets of the > > > hash table: > > > > > > E1: Adds 32 to the hash table. > > > E2: Adds 1729 to the hash table. > > > E3: Within a read-side critical section, looks up 32 then 172= 9. > > > E4: Within a read-side critical section, looks up 1729 then 3= 2. > > > > > > Given either MVCC or RLU, it will not be possible for E3 to find 32 b= ut > > > not 1729 and at the same time for E4 to find 1729 but not 32. Given > RCU, > > > this outcome is possible. > > > > > When you say "Within a read-side section", do you mean within a single > same > > read section? such as > > > > > read_lock() > > > lookup(32) > > > lookup(1729) > > > read_unlock() > > > > > > How about putting two lookups into two read-side sections? Do we still > have > > the problem, then? > > > > > read_lock() > > > lookup(32) > > > read_unlock() > > > read_lock() > > > lookup(1729) > > > read_unlock() > > Without in any way agreeing with your characterization of this as a > problem, because rcu_read_lock() and rcu_read_unlock() provide > absolutely no ordering guarantees in the absence of a grace period, > any non-grace-period-related reordering that can happen with a single > RCU read-side critical section can also happen when that critical > section is split in two as you have done above. > > > Could you kindly give me more clues why RCU can see such reorder, while > RLU > > can prevent it? > > Here are minimal C-language implementations for RCU that can (and are) > actually used: > Great. We use the same thing in our real-time work [1] > #define rcu_read_lock() > #define rcu_read_unlock() > > Please compare these to the read-side markers presented in the RLU paper, > and then tell me your thoughts on the answer to your question. ;-) > I submit my homework here, but I do not think I did it well. 1. I believe in the default URCU implementation, it has memory barrier inside the read_lock / read_unlock. 2. From the RLU paper, it shows the code for read-side operation. RLU_READER_LOCK(ctx) ctx.is-writer=E2=86=90false ctx.run-cnt=E2=86=90ctx.run-cnt+1. memory fence ctx.local-clock=E2=86=90global-clock RLU_READER_UNLOCK(ctx) ctx.run-cnt=E2=86=90ctx.run-cnt+1 if (ctx.is-writer) RLU_COMMIT_WRITE_LOG(ctx) We can ignore the writer check, as in our case, the reader never do update. My understanding is that read-side operations are only used to facilitate the quiescence detection. the run cnt is used to decide if a thread is active (if it is currently inside a read-side section). the local clock is used to check if the active thread goes into the read-side section very late, so it does not prevent reclaiming memory unlinked before it enters the read section. However i see no difference between RLU and RCU read-side operation regarding consistency guarantee. Could you kindly teach me more? BTW, the RLU read-side ops are similar, but not efficient, comparing to our Parsec work [1, 2] [1] Yuxin Ren, G. Liu, G. Parmer, B. Brandenburg, Scalable Memory Reclamation for Multi-Core, Real-Time Systems, in Proceedings of the 24th IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS), 2018 [2] Q. Wang, T. Stamler, and G. Parmer, Parallel Sections: Scaling System-Level Data-Structures, in Proceedings of European Conference on Computer Systems (EuroSys), 2016 Thanks, Best, Yuxin > > This is because MVCC and RLU provide readers a consistent view of > > > the updates, and RCU does not. Of course, it is often the case that = a > > > consistent view is not needed, in which case the MVCC and RLU > guarantees > > > are incurring read-side overhead for no reason. But if the use case > > > requires consistent readers, RCU is not an option. > > > > > > The reason a consistent view is not always needed is that > speed-of-light > > > delays make it impossible to provide a consistent view of the outside > > > world. In the common case where the use case interacts with the > > > outside world, the algorithms absolutely must be designed to tolerate > > > inconsistency, which opens the door to things like RCU. > > > > I am confused here. I think speed-of-light delays happen everywhere, no= t > > only bound to RCU, but also any other synchronization approach (such > RLU). > > If so, how do others (RLU) provide consistent views? > > You just stated the answer. Now it is only necessary for you to invest > the time, effort, and thought to fully understand it. To help with this, > the following paragraph provides another hint: > > Yes, you are quite right, speed-of-light delays between the > outside world and the computer affect RLU just as surely as they > do RCU. This means that the additional consistency guarantees > provided by RLU must necessarily be limited to the confines of th= e > computer system in question. Taking this one step further, there > are also speed-of-light delays within the computer. Therefore, > in the general case, RLU can provide its consistency guarantees, > even within the confines of the computer system, only at the > expense of incurring delays. Because RCU does not provide RLU's > consistency guarantees, it need not incur RLU's delays. > > This is not a new line of reasoning, see for example: > > @article{Herlihy:1996:LCN:1063369.1063372 > ,author =3D {Herlihy, Maurice and Shavit, Nir and Waarts, Orli} > ,title =3D {Linearizable counting networks} > ,journal =3D {Distrib. Comput.} > ,volume =3D {9} > ,issue =3D {4} > ,month =3D {February} > ,year =3D {1996} > ,issn =3D {0178-2770} > ,pages =3D {193--203} > ,numpages =3D {11} > ,url =3D {http://portal.acm.org/citation.cfm?id=3D1063369.1063372} > ,doi =3D {10.1007/s004460050019} > ,acmid =3D {1063372} > ,publisher =3D {Springer-Verlag} > ,address =3D {London, UK} > ,keywords =3D {concurrency, contention, counting networks, data structure= s, > linearizability} > } > > Thanx, Paul > > > Thanks for your education. > > Yuxin > > > > > > > > Thanx, Paul > > > > > > > Thanks > > > > Yuxin > > > > > > > > On Fri, Dec 6, 2019 at 11:30 AM Paul E. McKenney > > > > wrote: > > > > > > > > > On Fri, Dec 06, 2019 at 10:59:05AM -0500, Mathieu Desnoyers wrote= : > > > > > > ----- On Dec 6, 2019, at 3:51 PM, Yuxin Ren > > > wrote: > > > > > > > > > > > > > On Fri, Dec 6, 2019 at 5:49 AM Mathieu Desnoyers < [ > > > > > > > mailto:mathieu.desnoyers@efficios.com | > > > mathieu.desnoyers@efficios.com > > > > > ] > > > > > > > > wrote: > > > > > > > > > > > > >> ----- On Dec 5, 2019, at 8:17 PM, Yuxin Ren < [ mailto: > > > > > ryx@gwmail.gwu.edu | > > > > > > >> ryx@gwmail.gwu.edu ] > wrote: > > > > > > > > > > > > >>> Hi, > > > > > > >>> I am a student, and learning RCU now, but still know very > little > > > > > about it. > > > > > > >>> Are there any documents/papers/materials which (in)formally > > > define > > > > > and explain > > > > > > >>> RCU consistency guarantees? > > > > > > > > > > > > >> You may want to have a look at > > > > > > > > > > > > >> User-Level Implementations of Read-Copy Update > > > > > > >> Article in IEEE Transactions on Parallel and Distributed > Systems > > > > > 23(2):375 - 382 > > > > > > >> =C2=B7 March 2012 > > > > > > > > > > > > > Thanks for your info. > > > > > > > However, I do not think URCU talks about any consistency mode= l > > > > > formally. > > > > > > > > > > > > > From previous communication with Paul, he said RCU is not > designed > > > for > > > > > > > linearizability, and it is totally acceptable that RCU is not > > > > > linearizable. > > > > > > > However, I am curious how to accurately/formally Characterize > RCU > > > > > consistency > > > > > > > model/guarantees > > > > > > > > > > > > Adding Paul E. McKenney in CC. > > > > > > > > > > > > I am referring to the section "Overview of RCU semantics" in th= e > > > paper. > > > > > Not sure it has the level of > > > > > > formality you are looking for though. Paul, do you have pointer= s > to > > > > > additional material ? > > > > > > > > > > Indeed I do! The Linux kernel memory model (LKMM) includes RCU. > It is > > > > > in tools/memory-model in recent kernel source trees, which includ= es > > > > > documentation. This is an executable model, which means that you > > > > > can create litmus tests and have the model formally and > automatically > > > > > evaluate them. > > > > > > > > > > There are also a number of publications covering LKMM: > > > > > > > > > > o A formal kernel memory-ordering model > > > > > https://lwn.net/Articles/718628/ > > > > > https://lwn.net/Articles/720550/ > > > > > > > > > > These cover the release stores and dependency ordering th= at > > > > > provide RCU's publish-subscribe guarantees. > > > > > > > > > > Backup material here: > > > > > > > > > > > > > > > > > > > https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinuxM= M/ > > > > > > > > > > With these two likely being of particular interest: > > > > > > > > > > > > > > > > > > > https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinuxM= M/RCUguarantees.html > > > > > > > > > > > > > > https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinuxM= M/srcu.html > > > > > > > > > > o Frightening Small Children and Disconcerting Grown-ups: > > > > > Concurrency in the Linux Kernel > > > > > https://dl.acm.org/citation.cfm?id=3D3177156 > > > > > > http://www0.cs.ucl.ac.uk/staff/j.alglave/papers/asplos18.pdf > > > > > > > > > > Backup material: > > > > > > > > > > http://diy.inria.fr/linux/ > > > > > > > > > > o Who's afraid of a big bad optimizing compiler? > > > > > https://lwn.net/Articles/793253/ > > > > > > > > > > o Calibrating your fear of big bad optimizing compilers > > > > > https://lwn.net/Articles/799218/ > > > > > > > > > > These last two justify use of normal C-language assignmen= t > > > > > statements to initialize and access data referenced by > > > > > RCU-protected pointers. > > > > > > > > > > There is a large body of litmus tests (thousands of them) here: > > > > > > > > > > https://github.com/paulmckrcu/litmus > > > > > > > > > > Many of these litmus tests involve RCU, and these can be located = by > > > > > search for files containing rcu_read_lock(), rcu_read_unlock(), > > > > > synchronize_rcu(), and so on. > > > > > > > > > > Or were you looking for something else? > > > > > > > > > > Thanx, Pa= ul > > > > > > > > > > > Thanks, > > > > > > > > > > > > Mathieu > > > > > > > > > > > > >> as a starting point. > > > > > > > > > > > > >> Thanks, > > > > > > > > > > > > >> Mathieu > > > > > > > > > > > > >>> I know there are some consistency models in the database ar= ea > > > (such > > > > > as PRAM, > > > > > > >>> Read Uncommitted, etc) from [ https://jepsen.io/consistency > | > > > > > > >>> https://jepsen.io/consistency ] and [1]. > > > > > > >>> How does RCU related to those consistency models? > > > > > > > > > > > > >>> I also found some comments online (One key distinction is > that > > > both > > > > > MVCC and RLU > > > > > > >>> provide much stronger consistency guarantees to readers tha= n > does > > > > > RCU ...) ( [ > > > > > > >>> https://lwn.net/Articles/777036/ | > > > https://lwn.net/Articles/777036/ > > > > > ] ). > > > > > > >>> I do not understand how we reason/dresibe/compare the > consistency > > > > > guarantees. ( > > > > > > >>> I even do not know what consistency guarantees provided by > RCU > > > > > formally) > > > > > > >>> Could someone explain this to me? > > > > > > > > > > > > >>> [1] Bailis, P., Davidson, A., Fekete, A., Ghodsi, A., > > > Hellerstein, > > > > > J. M., & > > > > > > >>> Stoica, I. (2013). Highly available transactions: Virtues a= nd > > > > > limitations. > > > > > > >>> Proceedings of the VLDB Endowment, 7(3), 181-192. > > > > > > > > > > > > >>> Thanks > > > > > > >>> Yuxin > > > > > > > > > > > > >>> _______________________________________________ > > > > > > >>> lttng-dev mailing list > > > > > > >>> [ mailto:lttng-dev@lists.lttng.org | > lttng-dev@lists.lttng.org ] > > > > > > >>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-de= v > | > > > > > > >>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev = ] > > > > > > > > > > > > >> -- > > > > > > >> Mathieu Desnoyers > > > > > > >> EfficiOS Inc. > > > > > > >> [ http://www.efficios.com/ | http://www.efficios.com ] > > > > > > > > > > > > -- > > > > > > Mathieu Desnoyers > > > > > > EfficiOS Inc. > > > > > > http://www.efficios.com > > > > > > > > > --000000000000ba52b10599a421a6 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Paul

On Sat, Dec 7, 2019 at 5:42 PM Pau= l E. McKenney <p= aulmck@kernel.org> wrote:
On Sat, Dec 07, 2019 at 03:04:42PM -0500, Yuxin Ren wrote:=
> Thanks a lot for your help. I have some questions below.
>
> On Sat, Dec 7, 2019 at 1:37 AM Paul E. McKenney <paulmck@kernel.org> wrote:
>
> > On Fri, Dec 06, 2019 at 07:00:13PM -0500, Yuxin Ren wrote:
> > > Thanks so much for your great help.
> > > I definitely will look at those resources and papers!
> > >
> > > One more thing that I am confused
> > > As I mentioned earlier, someone said One key distinction is = that both
> > MVCC
> > > and RLU provide much stronger consistency guarantees to read= ers than does
> > > RCU ...) (https://lwn.net/Articles/777036/). > >
> > That someone was in fact me.=C2=A0 ;-)
> >
> > > I am not sure if the above statement is correct or not. But = in general,
> > > How can we compare RCU consistency guarantees to other techn= iques (such
> > as
> > > RLU)?
> > > How to reason about which one has stronger or weaker guarant= ees?
> >
> > I suggest starting from the use case.=C2=A0 For concreteness, let= 's assume
> > that we are using a hash table.=C2=A0 At one extreme, imagine a u= se case in
> > which each event makes exactly one hash-table operation.=C2=A0 No= information
> > is carried from one event to the next.=C2=A0 (This might well be = the case
> > for simple web server.)=C2=A0 Such a use case cannot tell the dif= ference
> > between RCU on the one hand and MVCC/RLU on the other.
> >
> > At the other extreme, suppose that each event either accesses or = updates
> > multiple entries in the hash table.=C2=A0 In this case, MVCC/RLU = will rule
> > out outcomes that RCU would permit.=C2=A0 For example, suppose we= had four
> > events accessing two different elements in different buckets of t= he
> > hash table:
> >
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0E1: Adds 32 to the hash table. > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0E2: Adds 1729 to the hash table.=
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0E3: Within a read-side critical = section, looks up 32 then 1729.
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0E4: Within a read-side critical = section, looks up 1729 then 32.
> >
> > Given either MVCC or RLU, it will not be possible for E3 to find = 32 but
> > not 1729 and at the same time for E4 to find 1729 but not 32.=C2= =A0 Given RCU,
> > this outcome is possible.
> >
> When you say "Within a read-side section", do you mean withi= n a single same
> read section? such as
>
> > read_lock()
> > lookup(32)
> > lookup(1729)
> > read_unlock()
>
>
> How about putting two lookups into two read-side sections? Do we still= have
> the problem, then?
>
> > read_lock()
> > lookup(32)
> > read_unlock()
> > read_lock()
> > lookup(1729)
> > read_unlock()

Without in any way agreeing with your characterization of this as a
problem, because rcu_read_lock() and rcu_read_unlock() provide
absolutely no ordering guarantees in the absence of a grace period,
any non-grace-period-related reordering that can happen with a single
RCU read-side critical section can also happen when that critical
section is split in two as you have done above.

> Could you kindly give me more clues why RCU can see such reorder, whil= e RLU
> can prevent it?

Here are minimal C-language implementations for RCU that can (and are)
actually used:
Great. We use the same thing in our rea= l-time work [1]


#define rcu_read_lock()
#define rcu_read_unlock()

Please compare these to the read-side markers presented in the RLU paper, and then tell me your thoughts on the answer to your question.=C2=A0 ;-)
I submit my homework here, but I do not think I did it w= ell.
1. I believe in the default URCU implementation, it has memo= ry barrier inside the read_lock / read_unlock.
2. From the RLU pa= per, it shows the code for read-side operation.
RLU_READER_LOCK(c= tx)
=C2=A0=C2=A0=C2=A0 ctx.is-writer=E2=86=90false
=C2= =A0=C2=A0=C2=A0 ctx.run-cnt=E2=86=90ctx.run-cnt+1.
=C2=A0=C2=A0= =C2=A0 memory fence
=C2=A0=C2=A0 ctx.local-clock=E2=86=90global-c= lock
RLU_READER_UNLOCK(ctx)
=C2=A0=C2=A0=C2=A0 ctx.run-= cnt=E2=86=90ctx.run-cnt+1
=C2=A0=C2=A0=C2=A0 if (ctx.is-writer)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 RLU_COMMIT_WRITE_LOG(ct= x)

We can ignore the writer check, as in our case,= the reader never do update.
My understanding is that read-side o= perations are only used to facilitate the quiescence detection.
t= he run cnt is used to decide if a thread is active (if it is currently insi= de a read-side section).
the local clock is used to check if the = active thread goes into the read-side section very late, so it does not pre= vent reclaiming memory unlinked before it enters the read section.
However i see no difference between RLU and RCU read-side operation regar= ding consistency guarantee.
Could you kindly teach me more?
=

BTW, the RLU read-side ops are similar, but not efficie= nt, comparing to our Parsec work [1, 2]
[1] Yuxin Ren, G. Liu, G.= Parmer, B. Brandenburg, Scalable Memory Reclamation for Multi-Core, Real-T= ime Systems, in Proceedings of the 24th IEEE Real-Time and Embedded Technol= ogy and Applications Symposium (RTAS), 2018
[2] Q. Wang, T. Staml= er, and G. Parmer, Parallel Sections: Scaling System-Level Data-Structures,= in Proceedings of European Conference on Computer Systems (EuroSys), 2016<= /div>

Thanks,
Best,
Yuxin
<= div>

> > This is because MVCC and RLU provide readers a consistent view of=
> > the updates, and RCU does not.=C2=A0 Of course, it is often the c= ase that a
> > consistent view is not needed, in which case the MVCC and RLU gua= rantees
> > are incurring read-side overhead for no reason.=C2=A0 But if the = use case
> > requires consistent readers, RCU is not an option.
> >
> > The reason a consistent view is not always needed is that speed-o= f-light
> > delays make it impossible to provide a consistent view of the out= side
> > world.=C2=A0 In the common case where the use case interacts with= the
> > outside world, the algorithms absolutely must be designed to tole= rate
> > inconsistency, which opens the door to things like RCU.
>
> I am confused here. I think speed-of-light delays happen everywhere, n= ot
> only bound to RCU, but also=C2=A0 any other synchronization approach (= such RLU).
> If so, how do others (RLU) provide consistent views?

You just stated the answer.=C2=A0 Now it is only necessary for you to inves= t
the time, effort, and thought to fully understand it.=C2=A0 To help with th= is,
the following paragraph provides another hint:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 Yes, you are quite right, speed-of-light delays= between the
=C2=A0 =C2=A0 =C2=A0 =C2=A0 outside world and the computer affect RLU just = as surely as they
=C2=A0 =C2=A0 =C2=A0 =C2=A0 do RCU.=C2=A0 This means that the additional co= nsistency guarantees
=C2=A0 =C2=A0 =C2=A0 =C2=A0 provided by RLU must necessarily be limited to = the confines of the
=C2=A0 =C2=A0 =C2=A0 =C2=A0 computer system in question.=C2=A0 Taking this = one step further, there
=C2=A0 =C2=A0 =C2=A0 =C2=A0 are also speed-of-light delays within the compu= ter.=C2=A0 Therefore,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 in the general case, RLU can provide its consis= tency guarantees,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 even within the confines of the computer system= , only at the
=C2=A0 =C2=A0 =C2=A0 =C2=A0 expense of incurring delays.=C2=A0 Because RCU = does not provide RLU's
=C2=A0 =C2=A0 =C2=A0 =C2=A0 consistency guarantees, it need not incur RLU&#= 39;s delays.

This is not a new line of reasoning, see for example:

@article{Herlihy:1996:LCN:1063369.1063372
,author =3D {Herlihy, Maurice and Shavit, Nir and Waarts, Orli}
,title =3D {Linearizable counting networks}
,journal =3D {Distrib. Comput.}
,volume =3D {9}
,issue =3D {4}
,month =3D {February}
,year =3D {1996}
,issn =3D {0178-2770}
,pages =3D {193--203}
,numpages =3D {11}
,url =3D {http://portal.acm.org/citation.cfm?= id=3D1063369.1063372}
,doi =3D {10.1007/s004460050019}
,acmid =3D {1063372}
,publisher =3D {Springer-Verlag}
,address =3D {London, UK}
,keywords =3D {concurrency, contention, counting networks, data structures,= linearizability}
}

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Thanx, Paul

> Thanks for your education.
> Yuxin
>
> >
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Thanx, Paul > >
> > > Thanks
> > > Yuxin
> > >
> > > On Fri, Dec 6, 2019 at 11:30 AM Paul E. McKenney <paulmck@kernel.org>=
> > wrote:
> > >
> > > > On Fri, Dec 06, 2019 at 10:59:05AM -0500, Mathieu Desno= yers wrote:
> > > > > ----- On Dec 6, 2019, at 3:51 PM, Yuxin Ren <ryx@gwmail.gwu.edu>
> > wrote:
> > > > >
> > > > > > On Fri, Dec 6, 2019 at 5:49 AM Mathieu Desnoy= ers < [
> > > > > > mailto:mathieu.desnoyers@efficios.com |
> > mathieu.desnoyers@efficios.com
> > > > ] >
> > > > > > wrote:
> > > > >
> > > > > >> ----- On Dec 5, 2019, at 8:17 PM, Yuxin R= en < [ mailto:
> > > > ryx@gwmail.gwu.edu |
> > > > > >> ryx@gwmail.gwu.edu ] > wrote:
> > > > >
> > > > > >>> Hi,
> > > > > >>> I am a student, and learning RCU now,= but still know very little
> > > > about it.
> > > > > >>> Are there any documents/papers/materi= als which (in)formally
> > define
> > > > and explain
> > > > > >>> RCU consistency guarantees?
> > > > >
> > > > > >> You may want to have a look at
> > > > >
> > > > > >> User-Level Implementations of Read-Copy U= pdate
> > > > > >> Article in IEEE Transactions on Parallel = and Distributed Systems
> > > > 23(2):375 - 382
> > > > > >> =C2=B7 March 2012
> > > > >
> > > > > > Thanks for your info.
> > > > > > However, I do not think URCU talks about any = consistency model
> > > > formally.
> > > > >
> > > > > > From previous communication with Paul, he sai= d RCU is not designed
> > for
> > > > > > linearizability, and it is totally acceptable= that RCU is not
> > > > linearizable.
> > > > > > However, I am curious how to accurately/forma= lly Characterize RCU
> > > > consistency
> > > > > > model/guarantees
> > > > >
> > > > > Adding Paul E. McKenney in CC.
> > > > >
> > > > > I am referring to the section "Overview of RC= U semantics" in the
> > paper.
> > > > Not sure it has the level of
> > > > > formality you are looking for though. Paul, do you= have pointers to
> > > > additional material ?
> > > >
> > > > Indeed I do!=C2=A0 The Linux kernel memory model (LKMM)= includes RCU.=C2=A0 It is
> > > > in tools/memory-model in recent kernel source trees, wh= ich includes
> > > > documentation.=C2=A0 This is an executable model, which= means that you
> > > > can create litmus tests and have the model formally and= automatically
> > > > evaluate them.
> > > >
> > > > There are also a number of publications covering LKMM:<= br> > > > >
> > > > o=C2=A0 =C2=A0 =C2=A0 =C2=A0A formal kernel memory-orde= ring model
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0https://lwn.net= /Articles/718628/
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0https://lwn.net= /Articles/720550/
> > > >
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0These cover the releas= e stores and dependency ordering that
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0provide RCU's publ= ish-subscribe guarantees.
> > > >
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Backup material here:<= br> > > > >
> > > >
> > > >
> > https://mirrors= .edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinuxMM/
> > > >
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0With these two likely = being of particular interest:
> > > >
> > > >
> > > >
> > https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinux= MM/RCUguarantees.html
> > > >
> > > >
> > https:= //mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinuxMM/srcu.h= tml
> > > >
> > > > o=C2=A0 =C2=A0 =C2=A0 =C2=A0Frightening Small Children = and Disconcerting Grown-ups:
> > > > Concurrency in the Linux Kernel
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0htt= ps://dl.acm.org/citation.cfm?id=3D3177156
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0http://www0.cs.ucl.ac.uk/staff/j.alglave/papers/asplos18.pdf
> > > >
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Backup material:
> > > >
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0http://diy.inria.fr/l= inux/
> > > >
> > > > o=C2=A0 =C2=A0 =C2=A0 =C2=A0Who's afraid of a big b= ad optimizing compiler?
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0https://lwn.net= /Articles/793253/
> > > >
> > > > o=C2=A0 =C2=A0 =C2=A0 =C2=A0Calibrating your fear of bi= g bad optimizing compilers
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0https://lwn.net= /Articles/799218/
> > > >
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0These last two justify= use of normal C-language assignment
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0statements to initiali= ze and access data referenced by
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0RCU-protected pointers= .
> > > >
> > > > There is a large body of litmus tests (thousands of the= m) here:
> > > >
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0https://git= hub.com/paulmckrcu/litmus
> > > >
> > > > Many of these litmus tests involve RCU, and these can b= e located by
> > > > search for files containing rcu_read_lock(), rcu_read_u= nlock(),
> > > > synchronize_rcu(), and so on.
> > > >
> > > > Or were you looking for something else?
> > > >
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Th= anx, Paul
> > > >
> > > > > Thanks,
> > > > >
> > > > > Mathieu
> > > > >
> > > > > >> as a starting point.
> > > > >
> > > > > >> Thanks,
> > > > >
> > > > > >> Mathieu
> > > > >
> > > > > >>> I know there are some consistency mod= els in the database area
> > (such
> > > > as PRAM,
> > > > > >>> Read Uncommitted, etc) from [ htt= ps://jepsen.io/consistency |
> > > > > >>> https://jepsen.io/consistency= ] and [1].
> > > > > >>> How does RCU related to those consist= ency models?
> > > > >
> > > > > >>> I also found some comments online (On= e key distinction is that
> > both
> > > > MVCC and RLU
> > > > > >>> provide much stronger consistency gua= rantees to readers than does
> > > > RCU ...) ( [
> > > > > >>> https://lwn.net/Articles/77703= 6/ |
> > https://lwn.net/Articles/777036/
> > > > ] ).
> > > > > >>> I do not understand how we reason/dre= sibe/compare the consistency
> > > > guarantees. (
> > > > > >>> I even do not know what consistency g= uarantees provided by RCU
> > > > formally)
> > > > > >>> Could someone explain this to me?
> > > > >
> > > > > >>> [1] Bailis, P., Davidson, A., Fekete,= A., Ghodsi, A.,
> > Hellerstein,
> > > > J. M., &
> > > > > >>> Stoica, I. (2013). Highly available t= ransactions: Virtues and
> > > > limitations.
> > > > > >>> Proceedings of the VLDB Endowment, 7(= 3), 181-192.
> > > > >
> > > > > >>> Thanks
> > > > > >>> Yuxin
> > > > >
> > > > > >>> _____________________________________= __________
> > > > > >>> lttng-dev mailing list
> > > > > >>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.ltt= ng.org ]
> > > > > >>> [ ht= tps://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev |
> > > > > >>> http= s://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ]
> > > > >
> > > > > >> --
> > > > > >> Mathieu Desnoyers
> > > > > >> EfficiOS Inc.
> > > > > >> [ http://www.efficios.com/ | http://ww= w.efficios.com ]
> > > > >
> > > > > --
> > > > > Mathieu Desnoyers
> > > > > EfficiOS Inc.
> > > > > http://www.efficios.com
> > > >
> >
--000000000000ba52b10599a421a6-- --===============2564164649379268757== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev --===============2564164649379268757==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Paul E. McKenney" Subject: Re: RCU consistency guarantees Date: Sun, 15 Dec 2019 12:30:19 -0800 Message-ID: <20191215203019.GN2889__3855.293691025$1576441840$gmane$org@paulmck-ThinkPad-P72> References: <194534011.751.1575629349181.JavaMail.zimbra@efficios.com> <512711764.1078.1575647945136.JavaMail.zimbra@efficios.com> <20191206163052.GG2889@paulmck-ThinkPad-P72> <20191207063730.GN2889@paulmck-ThinkPad-P72> <20191207224232.GR2889@paulmck-ThinkPad-P72> Reply-To: paulmck@kernel.org Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by lists.lttng.org (Postfix) with ESMTPS id 47bbcs0p7Jz19K0 for ; Sun, 15 Dec 2019 15:30:20 -0500 (EST) Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" To: Yuxin Ren Cc: lttng-dev List-Id: lttng-dev@lists.lttng.org T24gU2F0LCBEZWMgMTQsIDIwMTkgYXQgMDE6MzE6MzFBTSAtMDUwMCwgWXV4aW4gUmVuIHdyb3Rl Ogo+IEhpIFBhdWwKPiAKPiBPbiBTYXQsIERlYyA3LCAyMDE5IGF0IDU6NDIgUE0gUGF1bCBFLiBN Y0tlbm5leSA8cGF1bG1ja0BrZXJuZWwub3JnPiB3cm90ZToKPiAKPiA+IE9uIFNhdCwgRGVjIDA3 LCAyMDE5IGF0IDAzOjA0OjQyUE0gLTA1MDAsIFl1eGluIFJlbiB3cm90ZToKPiA+ID4gVGhhbmtz IGEgbG90IGZvciB5b3VyIGhlbHAuIEkgaGF2ZSBzb21lIHF1ZXN0aW9ucyBiZWxvdy4KPiA+ID4K PiA+ID4gT24gU2F0LCBEZWMgNywgMjAxOSBhdCAxOjM3IEFNIFBhdWwgRS4gTWNLZW5uZXkgPHBh dWxtY2tAa2VybmVsLm9yZz4KPiA+IHdyb3RlOgo+ID4gPgo+ID4gPiA+IE9uIEZyaSwgRGVjIDA2 LCAyMDE5IGF0IDA3OjAwOjEzUE0gLTA1MDAsIFl1eGluIFJlbiB3cm90ZToKPiA+ID4gPiA+IFRo YW5rcyBzbyBtdWNoIGZvciB5b3VyIGdyZWF0IGhlbHAuCj4gPiA+ID4gPiBJIGRlZmluaXRlbHkg d2lsbCBsb29rIGF0IHRob3NlIHJlc291cmNlcyBhbmQgcGFwZXJzIQo+ID4gPiA+ID4KPiA+ID4g PiA+IE9uZSBtb3JlIHRoaW5nIHRoYXQgSSBhbSBjb25mdXNlZAo+ID4gPiA+ID4gQXMgSSBtZW50 aW9uZWQgZWFybGllciwgc29tZW9uZSBzYWlkIE9uZSBrZXkgZGlzdGluY3Rpb24gaXMgdGhhdCBi b3RoCj4gPiA+ID4gTVZDQwo+ID4gPiA+ID4gYW5kIFJMVSBwcm92aWRlIG11Y2ggc3Ryb25nZXIg Y29uc2lzdGVuY3kgZ3VhcmFudGVlcyB0byByZWFkZXJzIHRoYW4KPiA+IGRvZXMKPiA+ID4gPiA+ IFJDVSAuLi4pIChodHRwczovL2x3bi5uZXQvQXJ0aWNsZXMvNzc3MDM2LykuCj4gPiA+ID4KPiA+ ID4gPiBUaGF0IHNvbWVvbmUgd2FzIGluIGZhY3QgbWUuICA7LSkKPiA+ID4gPgo+ID4gPiA+ID4g SSBhbSBub3Qgc3VyZSBpZiB0aGUgYWJvdmUgc3RhdGVtZW50IGlzIGNvcnJlY3Qgb3Igbm90LiBC dXQgaW4KPiA+IGdlbmVyYWwsCj4gPiA+ID4gPiBIb3cgY2FuIHdlIGNvbXBhcmUgUkNVIGNvbnNp c3RlbmN5IGd1YXJhbnRlZXMgdG8gb3RoZXIgdGVjaG5pcXVlcwo+ID4gKHN1Y2gKPiA+ID4gPiBh cwo+ID4gPiA+ID4gUkxVKT8KPiA+ID4gPiA+IEhvdyB0byByZWFzb24gYWJvdXQgd2hpY2ggb25l IGhhcyBzdHJvbmdlciBvciB3ZWFrZXIgZ3VhcmFudGVlcz8KPiA+ID4gPgo+ID4gPiA+IEkgc3Vn Z2VzdCBzdGFydGluZyBmcm9tIHRoZSB1c2UgY2FzZS4gIEZvciBjb25jcmV0ZW5lc3MsIGxldCdz IGFzc3VtZQo+ID4gPiA+IHRoYXQgd2UgYXJlIHVzaW5nIGEgaGFzaCB0YWJsZS4gIEF0IG9uZSBl eHRyZW1lLCBpbWFnaW5lIGEgdXNlIGNhc2UgaW4KPiA+ID4gPiB3aGljaCBlYWNoIGV2ZW50IG1h a2VzIGV4YWN0bHkgb25lIGhhc2gtdGFibGUgb3BlcmF0aW9uLiAgTm8KPiA+IGluZm9ybWF0aW9u Cj4gPiA+ID4gaXMgY2FycmllZCBmcm9tIG9uZSBldmVudCB0byB0aGUgbmV4dC4gIChUaGlzIG1p Z2h0IHdlbGwgYmUgdGhlIGNhc2UKPiA+ID4gPiBmb3Igc2ltcGxlIHdlYiBzZXJ2ZXIuKSAgU3Vj aCBhIHVzZSBjYXNlIGNhbm5vdCB0ZWxsIHRoZSBkaWZmZXJlbmNlCj4gPiA+ID4gYmV0d2VlbiBS Q1Ugb24gdGhlIG9uZSBoYW5kIGFuZCBNVkNDL1JMVSBvbiB0aGUgb3RoZXIuCj4gPiA+ID4KPiA+ ID4gPiBBdCB0aGUgb3RoZXIgZXh0cmVtZSwgc3VwcG9zZSB0aGF0IGVhY2ggZXZlbnQgZWl0aGVy IGFjY2Vzc2VzIG9yCj4gPiB1cGRhdGVzCj4gPiA+ID4gbXVsdGlwbGUgZW50cmllcyBpbiB0aGUg aGFzaCB0YWJsZS4gIEluIHRoaXMgY2FzZSwgTVZDQy9STFUgd2lsbCBydWxlCj4gPiA+ID4gb3V0 IG91dGNvbWVzIHRoYXQgUkNVIHdvdWxkIHBlcm1pdC4gIEZvciBleGFtcGxlLCBzdXBwb3NlIHdl IGhhZCBmb3VyCj4gPiA+ID4gZXZlbnRzIGFjY2Vzc2luZyB0d28gZGlmZmVyZW50IGVsZW1lbnRz IGluIGRpZmZlcmVudCBidWNrZXRzIG9mIHRoZQo+ID4gPiA+IGhhc2ggdGFibGU6Cj4gPiA+ID4K PiA+ID4gPiAgICAgICAgIEUxOiBBZGRzIDMyIHRvIHRoZSBoYXNoIHRhYmxlLgo+ID4gPiA+ICAg ICAgICAgRTI6IEFkZHMgMTcyOSB0byB0aGUgaGFzaCB0YWJsZS4KPiA+ID4gPiAgICAgICAgIEUz OiBXaXRoaW4gYSByZWFkLXNpZGUgY3JpdGljYWwgc2VjdGlvbiwgbG9va3MgdXAgMzIgdGhlbiAx NzI5Lgo+ID4gPiA+ICAgICAgICAgRTQ6IFdpdGhpbiBhIHJlYWQtc2lkZSBjcml0aWNhbCBzZWN0 aW9uLCBsb29rcyB1cCAxNzI5IHRoZW4gMzIuCj4gPiA+ID4KPiA+ID4gPiBHaXZlbiBlaXRoZXIg TVZDQyBvciBSTFUsIGl0IHdpbGwgbm90IGJlIHBvc3NpYmxlIGZvciBFMyB0byBmaW5kIDMyIGJ1 dAo+ID4gPiA+IG5vdCAxNzI5IGFuZCBhdCB0aGUgc2FtZSB0aW1lIGZvciBFNCB0byBmaW5kIDE3 MjkgYnV0IG5vdCAzMi4gIEdpdmVuCj4gPiBSQ1UsCj4gPiA+ID4gdGhpcyBvdXRjb21lIGlzIHBv c3NpYmxlLgo+ID4gPiA+Cj4gPiA+IFdoZW4geW91IHNheSAiV2l0aGluIGEgcmVhZC1zaWRlIHNl Y3Rpb24iLCBkbyB5b3UgbWVhbiB3aXRoaW4gYSBzaW5nbGUKPiA+IHNhbWUKPiA+ID4gcmVhZCBz ZWN0aW9uPyBzdWNoIGFzCj4gPiA+Cj4gPiA+ID4gcmVhZF9sb2NrKCkKPiA+ID4gPiBsb29rdXAo MzIpCj4gPiA+ID4gbG9va3VwKDE3MjkpCj4gPiA+ID4gcmVhZF91bmxvY2soKQo+ID4gPgo+ID4g Pgo+ID4gPiBIb3cgYWJvdXQgcHV0dGluZyB0d28gbG9va3VwcyBpbnRvIHR3byByZWFkLXNpZGUg c2VjdGlvbnM/IERvIHdlIHN0aWxsCj4gPiBoYXZlCj4gPiA+IHRoZSBwcm9ibGVtLCB0aGVuPwo+ ID4gPgo+ID4gPiA+IHJlYWRfbG9jaygpCj4gPiA+ID4gbG9va3VwKDMyKQo+ID4gPiA+IHJlYWRf dW5sb2NrKCkKPiA+ID4gPiByZWFkX2xvY2soKQo+ID4gPiA+IGxvb2t1cCgxNzI5KQo+ID4gPiA+ IHJlYWRfdW5sb2NrKCkKPiA+Cj4gPiBXaXRob3V0IGluIGFueSB3YXkgYWdyZWVpbmcgd2l0aCB5 b3VyIGNoYXJhY3Rlcml6YXRpb24gb2YgdGhpcyBhcyBhCj4gPiBwcm9ibGVtLCBiZWNhdXNlIHJj dV9yZWFkX2xvY2soKSBhbmQgcmN1X3JlYWRfdW5sb2NrKCkgcHJvdmlkZQo+ID4gYWJzb2x1dGVs eSBubyBvcmRlcmluZyBndWFyYW50ZWVzIGluIHRoZSBhYnNlbmNlIG9mIGEgZ3JhY2UgcGVyaW9k LAo+ID4gYW55IG5vbi1ncmFjZS1wZXJpb2QtcmVsYXRlZCByZW9yZGVyaW5nIHRoYXQgY2FuIGhh cHBlbiB3aXRoIGEgc2luZ2xlCj4gPiBSQ1UgcmVhZC1zaWRlIGNyaXRpY2FsIHNlY3Rpb24gY2Fu IGFsc28gaGFwcGVuIHdoZW4gdGhhdCBjcml0aWNhbAo+ID4gc2VjdGlvbiBpcyBzcGxpdCBpbiB0 d28gYXMgeW91IGhhdmUgZG9uZSBhYm92ZS4KPiA+Cj4gPiA+IENvdWxkIHlvdSBraW5kbHkgZ2l2 ZSBtZSBtb3JlIGNsdWVzIHdoeSBSQ1UgY2FuIHNlZSBzdWNoIHJlb3JkZXIsIHdoaWxlCj4gPiBS TFUKPiA+ID4gY2FuIHByZXZlbnQgaXQ/Cj4gPgo+ID4gSGVyZSBhcmUgbWluaW1hbCBDLWxhbmd1 YWdlIGltcGxlbWVudGF0aW9ucyBmb3IgUkNVIHRoYXQgY2FuIChhbmQgYXJlKQo+ID4gYWN0dWFs bHkgdXNlZDoKPiA+Cj4gR3JlYXQuIFdlIHVzZSB0aGUgc2FtZSB0aGluZyBpbiBvdXIgcmVhbC10 aW1lIHdvcmsgWzFdCgpJdCBoYXMgYmVlbiBhIHBvcHVsYXIgY2hvaWNlIGZvciA0MCB5ZWFycyBu b3cuICA7LSkKCj4gPiAjZGVmaW5lIHJjdV9yZWFkX2xvY2soKQo+ID4gI2RlZmluZSByY3VfcmVh ZF91bmxvY2soKQo+ID4KPiA+IFBsZWFzZSBjb21wYXJlIHRoZXNlIHRvIHRoZSByZWFkLXNpZGUg bWFya2VycyBwcmVzZW50ZWQgaW4gdGhlIFJMVSBwYXBlciwKPiA+IGFuZCB0aGVuIHRlbGwgbWUg eW91ciB0aG91Z2h0cyBvbiB0aGUgYW5zd2VyIHRvIHlvdXIgcXVlc3Rpb24uICA7LSkKPiA+Cj4g SSBzdWJtaXQgbXkgaG9tZXdvcmsgaGVyZSwgYnV0IEkgZG8gbm90IHRoaW5rIEkgZGlkIGl0IHdl bGwuCj4gMS4gSSBiZWxpZXZlIGluIHRoZSBkZWZhdWx0IFVSQ1UgaW1wbGVtZW50YXRpb24sIGl0 IGhhcyBtZW1vcnkgYmFycmllcgo+IGluc2lkZSB0aGUgcmVhZF9sb2NrIC8gcmVhZF91bmxvY2su CgpJdCBjZXJ0YWlubHkgd2FzIGF0IG9uZSB0aW1lLiAgQnV0IHlvdSB3b3VsZCBvZiBjb3Vyc2Ug bmVlZCB0byBjaGVjawp0byBzZWUgd2hhdCBhbnkgZ2l2ZW4gd29ya2VyIGFjdHVhbGx5IHVzZWQu Cgo+IDIuIEZyb20gdGhlIFJMVSBwYXBlciwgaXQgc2hvd3MgdGhlIGNvZGUgZm9yIHJlYWQtc2lk ZSBvcGVyYXRpb24uCj4gUkxVX1JFQURFUl9MT0NLKGN0eCkKPiAgICAgY3R4LmlzLXdyaXRlcuKG kGZhbHNlCj4gICAgIGN0eC5ydW4tY2504oaQY3R4LnJ1bi1jbnQrMS4KPiAgICAgbWVtb3J5IGZl bmNlCj4gICAgY3R4LmxvY2FsLWNsb2Nr4oaQZ2xvYmFsLWNsb2NrCj4gUkxVX1JFQURFUl9VTkxP Q0soY3R4KQo+ICAgICBjdHgucnVuLWNudOKGkGN0eC5ydW4tY250KzEKPiAgICAgaWYgKGN0eC5p cy13cml0ZXIpCj4gICAgICAgICBSTFVfQ09NTUlUX1dSSVRFX0xPRyhjdHgpCj4gCj4gV2UgY2Fu IGlnbm9yZSB0aGUgd3JpdGVyIGNoZWNrLCBhcyBpbiBvdXIgY2FzZSwgdGhlIHJlYWRlciBuZXZl ciBkbyB1cGRhdGUuCj4gTXkgdW5kZXJzdGFuZGluZyBpcyB0aGF0IHJlYWQtc2lkZSBvcGVyYXRp b25zIGFyZSBvbmx5IHVzZWQgdG8gZmFjaWxpdGF0ZQo+IHRoZSBxdWllc2NlbmNlIGRldGVjdGlv bi4KPiB0aGUgcnVuIGNudCBpcyB1c2VkIHRvIGRlY2lkZSBpZiBhIHRocmVhZCBpcyBhY3RpdmUg KGlmIGl0IGlzIGN1cnJlbnRseQo+IGluc2lkZSBhIHJlYWQtc2lkZSBzZWN0aW9uKS4KPiB0aGUg bG9jYWwgY2xvY2sgaXMgdXNlZCB0byBjaGVjayBpZiB0aGUgYWN0aXZlIHRocmVhZCBnb2VzIGlu dG8gdGhlCj4gcmVhZC1zaWRlIHNlY3Rpb24gdmVyeSBsYXRlLCBzbyBpdCBkb2VzIG5vdCBwcmV2 ZW50IHJlY2xhaW1pbmcgbWVtb3J5Cj4gdW5saW5rZWQgYmVmb3JlIGl0IGVudGVycyB0aGUgcmVh ZCBzZWN0aW9uLgo+IEhvd2V2ZXIgaSBzZWUgbm8gZGlmZmVyZW5jZSBiZXR3ZWVuIFJMVSBhbmQg UkNVIHJlYWQtc2lkZSBvcGVyYXRpb24KPiByZWdhcmRpbmcgY29uc2lzdGVuY3kgZ3VhcmFudGVl Lgo+IENvdWxkIHlvdSBraW5kbHkgdGVhY2ggbWUgbW9yZT8KCkluIG9yZGVyIHRvIHRlYWNoIHlv dSBtb3JlLCBJIGRvIG5lZWQgc29tZSBoZWxwIGZyb20geW91LiAgSSBoYXZlIHBvc3RlZAphIG51 bWJlciBvZiBVUkxzIGVhcmxpZXIgaW4gdGhpcyBlbWFpbCB0aHJlYWQuICBDb3VsZCB5b3UgcGxl YXNlIHRlbGwKbWUgd2hhdCB5b3UgbGVhcm5lZCBmcm9tIGVhY2ggb2YgdGhlbT8KCgkJCQkJCQkJ VGhhbngsIFBhdWwKCj4gQlRXLCB0aGUgUkxVIHJlYWQtc2lkZSBvcHMgYXJlIHNpbWlsYXIsIGJ1 dCBub3QgZWZmaWNpZW50LCBjb21wYXJpbmcgdG8gb3VyCj4gUGFyc2VjIHdvcmsgWzEsIDJdCj4g WzFdIFl1eGluIFJlbiwgRy4gTGl1LCBHLiBQYXJtZXIsIEIuIEJyYW5kZW5idXJnLCBTY2FsYWJs ZSBNZW1vcnkKPiBSZWNsYW1hdGlvbiBmb3IgTXVsdGktQ29yZSwgUmVhbC1UaW1lIFN5c3RlbXMs IGluIFByb2NlZWRpbmdzIG9mIHRoZSAyNHRoCj4gSUVFRSBSZWFsLVRpbWUgYW5kIEVtYmVkZGVk IFRlY2hub2xvZ3kgYW5kIEFwcGxpY2F0aW9ucyBTeW1wb3NpdW0gKFJUQVMpLAo+IDIwMTgKPiBb Ml0gUS4gV2FuZywgVC4gU3RhbWxlciwgYW5kIEcuIFBhcm1lciwgUGFyYWxsZWwgU2VjdGlvbnM6 IFNjYWxpbmcKPiBTeXN0ZW0tTGV2ZWwgRGF0YS1TdHJ1Y3R1cmVzLCBpbiBQcm9jZWVkaW5ncyBv ZiBFdXJvcGVhbiBDb25mZXJlbmNlIG9uCj4gQ29tcHV0ZXIgU3lzdGVtcyAoRXVyb1N5cyksIDIw MTYKPiAKPiBUaGFua3MsCj4gQmVzdCwKPiBZdXhpbgo+IAo+IAo+ID4gPiBUaGlzIGlzIGJlY2F1 c2UgTVZDQyBhbmQgUkxVIHByb3ZpZGUgcmVhZGVycyBhIGNvbnNpc3RlbnQgdmlldyBvZgo+ID4g PiA+IHRoZSB1cGRhdGVzLCBhbmQgUkNVIGRvZXMgbm90LiAgT2YgY291cnNlLCBpdCBpcyBvZnRl biB0aGUgY2FzZSB0aGF0IGEKPiA+ID4gPiBjb25zaXN0ZW50IHZpZXcgaXMgbm90IG5lZWRlZCwg aW4gd2hpY2ggY2FzZSB0aGUgTVZDQyBhbmQgUkxVCj4gPiBndWFyYW50ZWVzCj4gPiA+ID4gYXJl IGluY3VycmluZyByZWFkLXNpZGUgb3ZlcmhlYWQgZm9yIG5vIHJlYXNvbi4gIEJ1dCBpZiB0aGUg dXNlIGNhc2UKPiA+ID4gPiByZXF1aXJlcyBjb25zaXN0ZW50IHJlYWRlcnMsIFJDVSBpcyBub3Qg YW4gb3B0aW9uLgo+ID4gPiA+Cj4gPiA+ID4gVGhlIHJlYXNvbiBhIGNvbnNpc3RlbnQgdmlldyBp cyBub3QgYWx3YXlzIG5lZWRlZCBpcyB0aGF0Cj4gPiBzcGVlZC1vZi1saWdodAo+ID4gPiA+IGRl bGF5cyBtYWtlIGl0IGltcG9zc2libGUgdG8gcHJvdmlkZSBhIGNvbnNpc3RlbnQgdmlldyBvZiB0 aGUgb3V0c2lkZQo+ID4gPiA+IHdvcmxkLiAgSW4gdGhlIGNvbW1vbiBjYXNlIHdoZXJlIHRoZSB1 c2UgY2FzZSBpbnRlcmFjdHMgd2l0aCB0aGUKPiA+ID4gPiBvdXRzaWRlIHdvcmxkLCB0aGUgYWxn b3JpdGhtcyBhYnNvbHV0ZWx5IG11c3QgYmUgZGVzaWduZWQgdG8gdG9sZXJhdGUKPiA+ID4gPiBp bmNvbnNpc3RlbmN5LCB3aGljaCBvcGVucyB0aGUgZG9vciB0byB0aGluZ3MgbGlrZSBSQ1UuCj4g PiA+Cj4gPiA+IEkgYW0gY29uZnVzZWQgaGVyZS4gSSB0aGluayBzcGVlZC1vZi1saWdodCBkZWxh eXMgaGFwcGVuIGV2ZXJ5d2hlcmUsIG5vdAo+ID4gPiBvbmx5IGJvdW5kIHRvIFJDVSwgYnV0IGFs c28gIGFueSBvdGhlciBzeW5jaHJvbml6YXRpb24gYXBwcm9hY2ggKHN1Y2gKPiA+IFJMVSkuCj4g PiA+IElmIHNvLCBob3cgZG8gb3RoZXJzIChSTFUpIHByb3ZpZGUgY29uc2lzdGVudCB2aWV3cz8K PiA+Cj4gPiBZb3UganVzdCBzdGF0ZWQgdGhlIGFuc3dlci4gIE5vdyBpdCBpcyBvbmx5IG5lY2Vz c2FyeSBmb3IgeW91IHRvIGludmVzdAo+ID4gdGhlIHRpbWUsIGVmZm9ydCwgYW5kIHRob3VnaHQg dG8gZnVsbHkgdW5kZXJzdGFuZCBpdC4gIFRvIGhlbHAgd2l0aCB0aGlzLAo+ID4gdGhlIGZvbGxv d2luZyBwYXJhZ3JhcGggcHJvdmlkZXMgYW5vdGhlciBoaW50Ogo+ID4KPiA+ICAgICAgICAgWWVz LCB5b3UgYXJlIHF1aXRlIHJpZ2h0LCBzcGVlZC1vZi1saWdodCBkZWxheXMgYmV0d2VlbiB0aGUK PiA+ICAgICAgICAgb3V0c2lkZSB3b3JsZCBhbmQgdGhlIGNvbXB1dGVyIGFmZmVjdCBSTFUganVz dCBhcyBzdXJlbHkgYXMgdGhleQo+ID4gICAgICAgICBkbyBSQ1UuICBUaGlzIG1lYW5zIHRoYXQg dGhlIGFkZGl0aW9uYWwgY29uc2lzdGVuY3kgZ3VhcmFudGVlcwo+ID4gICAgICAgICBwcm92aWRl ZCBieSBSTFUgbXVzdCBuZWNlc3NhcmlseSBiZSBsaW1pdGVkIHRvIHRoZSBjb25maW5lcyBvZiB0 aGUKPiA+ICAgICAgICAgY29tcHV0ZXIgc3lzdGVtIGluIHF1ZXN0aW9uLiAgVGFraW5nIHRoaXMg b25lIHN0ZXAgZnVydGhlciwgdGhlcmUKPiA+ICAgICAgICAgYXJlIGFsc28gc3BlZWQtb2YtbGln aHQgZGVsYXlzIHdpdGhpbiB0aGUgY29tcHV0ZXIuICBUaGVyZWZvcmUsCj4gPiAgICAgICAgIGlu IHRoZSBnZW5lcmFsIGNhc2UsIFJMVSBjYW4gcHJvdmlkZSBpdHMgY29uc2lzdGVuY3kgZ3VhcmFu dGVlcywKPiA+ICAgICAgICAgZXZlbiB3aXRoaW4gdGhlIGNvbmZpbmVzIG9mIHRoZSBjb21wdXRl ciBzeXN0ZW0sIG9ubHkgYXQgdGhlCj4gPiAgICAgICAgIGV4cGVuc2Ugb2YgaW5jdXJyaW5nIGRl bGF5cy4gIEJlY2F1c2UgUkNVIGRvZXMgbm90IHByb3ZpZGUgUkxVJ3MKPiA+ICAgICAgICAgY29u c2lzdGVuY3kgZ3VhcmFudGVlcywgaXQgbmVlZCBub3QgaW5jdXIgUkxVJ3MgZGVsYXlzLgo+ID4K PiA+IFRoaXMgaXMgbm90IGEgbmV3IGxpbmUgb2YgcmVhc29uaW5nLCBzZWUgZm9yIGV4YW1wbGU6 Cj4gPgo+ID4gQGFydGljbGV7SGVybGloeToxOTk2OkxDTjoxMDYzMzY5LjEwNjMzNzIKPiA+ICxh dXRob3IgPSB7SGVybGloeSwgTWF1cmljZSBhbmQgU2hhdml0LCBOaXIgYW5kIFdhYXJ0cywgT3Js aX0KPiA+ICx0aXRsZSA9IHtMaW5lYXJpemFibGUgY291bnRpbmcgbmV0d29ya3N9Cj4gPiAsam91 cm5hbCA9IHtEaXN0cmliLiBDb21wdXQufQo+ID4gLHZvbHVtZSA9IHs5fQo+ID4gLGlzc3VlID0g ezR9Cj4gPiAsbW9udGggPSB7RmVicnVhcnl9Cj4gPiAseWVhciA9IHsxOTk2fQo+ID4gLGlzc24g PSB7MDE3OC0yNzcwfQo+ID4gLHBhZ2VzID0gezE5My0tMjAzfQo+ID4gLG51bXBhZ2VzID0gezEx fQo+ID4gLHVybCA9IHtodHRwOi8vcG9ydGFsLmFjbS5vcmcvY2l0YXRpb24uY2ZtP2lkPTEwNjMz NjkuMTA2MzM3Mn0KPiA+ICxkb2kgPSB7MTAuMTAwNy9zMDA0NDYwMDUwMDE5fQo+ID4gLGFjbWlk ID0gezEwNjMzNzJ9Cj4gPiAscHVibGlzaGVyID0ge1NwcmluZ2VyLVZlcmxhZ30KPiA+ICxhZGRy ZXNzID0ge0xvbmRvbiwgVUt9Cj4gPiAsa2V5d29yZHMgPSB7Y29uY3VycmVuY3ksIGNvbnRlbnRp b24sIGNvdW50aW5nIG5ldHdvcmtzLCBkYXRhIHN0cnVjdHVyZXMsCj4gPiBsaW5lYXJpemFiaWxp dHl9Cj4gPiB9Cj4gPgo+ID4gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICBUaGFueCwgUGF1bAo+ID4KPiA+ID4gVGhhbmtzIGZvciB5b3VyIGVk dWNhdGlvbi4KPiA+ID4gWXV4aW4KPiA+ID4KPiA+ID4gPgo+ID4gPiA+ICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgVGhhbngsIFBhdWwKPiA+ ID4gPgo+ID4gPiA+ID4gVGhhbmtzCj4gPiA+ID4gPiBZdXhpbgo+ID4gPiA+ID4KPiA+ID4gPiA+ IE9uIEZyaSwgRGVjIDYsIDIwMTkgYXQgMTE6MzAgQU0gUGF1bCBFLiBNY0tlbm5leSA8cGF1bG1j a0BrZXJuZWwub3JnCj4gPiA+Cj4gPiA+ID4gd3JvdGU6Cj4gPiA+ID4gPgo+ID4gPiA+ID4gPiBP biBGcmksIERlYyAwNiwgMjAxOSBhdCAxMDo1OTowNUFNIC0wNTAwLCBNYXRoaWV1IERlc25veWVy cyB3cm90ZToKPiA+ID4gPiA+ID4gPiAtLS0tLSBPbiBEZWMgNiwgMjAxOSwgYXQgMzo1MSBQTSwg WXV4aW4gUmVuIDxyeXhAZ3dtYWlsLmd3dS5lZHU+Cj4gPiA+ID4gd3JvdGU6Cj4gPiA+ID4gPiA+ ID4KPiA+ID4gPiA+ID4gPiA+IE9uIEZyaSwgRGVjIDYsIDIwMTkgYXQgNTo0OSBBTSBNYXRoaWV1 IERlc25veWVycyA8IFsKPiA+ID4gPiA+ID4gPiA+IG1haWx0bzptYXRoaWV1LmRlc25veWVyc0Bl ZmZpY2lvcy5jb20gfAo+ID4gPiA+IG1hdGhpZXUuZGVzbm95ZXJzQGVmZmljaW9zLmNvbQo+ID4g PiA+ID4gPiBdID4KPiA+ID4gPiA+ID4gPiA+IHdyb3RlOgo+ID4gPiA+ID4gPiA+Cj4gPiA+ID4g PiA+ID4gPj4gLS0tLS0gT24gRGVjIDUsIDIwMTksIGF0IDg6MTcgUE0sIFl1eGluIFJlbiA8IFsg bWFpbHRvOgo+ID4gPiA+ID4gPiByeXhAZ3dtYWlsLmd3dS5lZHUgfAo+ID4gPiA+ID4gPiA+ID4+ IHJ5eEBnd21haWwuZ3d1LmVkdSBdID4gd3JvdGU6Cj4gPiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4g PiA+Pj4gSGksCj4gPiA+ID4gPiA+ID4gPj4+IEkgYW0gYSBzdHVkZW50LCBhbmQgbGVhcm5pbmcg UkNVIG5vdywgYnV0IHN0aWxsIGtub3cgdmVyeQo+ID4gbGl0dGxlCj4gPiA+ID4gPiA+IGFib3V0 IGl0Lgo+ID4gPiA+ID4gPiA+ID4+PiBBcmUgdGhlcmUgYW55IGRvY3VtZW50cy9wYXBlcnMvbWF0 ZXJpYWxzIHdoaWNoIChpbilmb3JtYWxseQo+ID4gPiA+IGRlZmluZQo+ID4gPiA+ID4gPiBhbmQg ZXhwbGFpbgo+ID4gPiA+ID4gPiA+ID4+PiBSQ1UgY29uc2lzdGVuY3kgZ3VhcmFudGVlcz8KPiA+ ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiA+ID4+IFlvdSBtYXkgd2FudCB0byBoYXZlIGEgbG9vayBh dAo+ID4gPiA+ID4gPiA+Cj4gPiA+ID4gPiA+ID4gPj4gVXNlci1MZXZlbCBJbXBsZW1lbnRhdGlv bnMgb2YgUmVhZC1Db3B5IFVwZGF0ZQo+ID4gPiA+ID4gPiA+ID4+IEFydGljbGUgaW4gSUVFRSBU cmFuc2FjdGlvbnMgb24gUGFyYWxsZWwgYW5kIERpc3RyaWJ1dGVkCj4gPiBTeXN0ZW1zCj4gPiA+ ID4gPiA+IDIzKDIpOjM3NSAtIDM4Mgo+ID4gPiA+ID4gPiA+ID4+IMK3IE1hcmNoIDIwMTIKPiA+ ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiA+ID4gVGhhbmtzIGZvciB5b3VyIGluZm8uCj4gPiA+ID4g PiA+ID4gPiBIb3dldmVyLCBJIGRvIG5vdCB0aGluayBVUkNVIHRhbGtzIGFib3V0IGFueSBjb25z aXN0ZW5jeSBtb2RlbAo+ID4gPiA+ID4gPiBmb3JtYWxseS4KPiA+ID4gPiA+ID4gPgo+ID4gPiA+ ID4gPiA+ID4gRnJvbSBwcmV2aW91cyBjb21tdW5pY2F0aW9uIHdpdGggUGF1bCwgaGUgc2FpZCBS Q1UgaXMgbm90Cj4gPiBkZXNpZ25lZAo+ID4gPiA+IGZvcgo+ID4gPiA+ID4gPiA+ID4gbGluZWFy aXphYmlsaXR5LCBhbmQgaXQgaXMgdG90YWxseSBhY2NlcHRhYmxlIHRoYXQgUkNVIGlzIG5vdAo+ ID4gPiA+ID4gPiBsaW5lYXJpemFibGUuCj4gPiA+ID4gPiA+ID4gPiBIb3dldmVyLCBJIGFtIGN1 cmlvdXMgaG93IHRvIGFjY3VyYXRlbHkvZm9ybWFsbHkgQ2hhcmFjdGVyaXplCj4gPiBSQ1UKPiA+ ID4gPiA+ID4gY29uc2lzdGVuY3kKPiA+ID4gPiA+ID4gPiA+IG1vZGVsL2d1YXJhbnRlZXMKPiA+ ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiA+IEFkZGluZyBQYXVsIEUuIE1jS2VubmV5IGluIENDLgo+ ID4gPiA+ID4gPiA+Cj4gPiA+ID4gPiA+ID4gSSBhbSByZWZlcnJpbmcgdG8gdGhlIHNlY3Rpb24g Ik92ZXJ2aWV3IG9mIFJDVSBzZW1hbnRpY3MiIGluIHRoZQo+ID4gPiA+IHBhcGVyLgo+ID4gPiA+ ID4gPiBOb3Qgc3VyZSBpdCBoYXMgdGhlIGxldmVsIG9mCj4gPiA+ID4gPiA+ID4gZm9ybWFsaXR5 IHlvdSBhcmUgbG9va2luZyBmb3IgdGhvdWdoLiBQYXVsLCBkbyB5b3UgaGF2ZSBwb2ludGVycwo+ ID4gdG8KPiA+ID4gPiA+ID4gYWRkaXRpb25hbCBtYXRlcmlhbCA/Cj4gPiA+ID4gPiA+Cj4gPiA+ ID4gPiA+IEluZGVlZCBJIGRvISAgVGhlIExpbnV4IGtlcm5lbCBtZW1vcnkgbW9kZWwgKExLTU0p IGluY2x1ZGVzIFJDVS4KPiA+IEl0IGlzCj4gPiA+ID4gPiA+IGluIHRvb2xzL21lbW9yeS1tb2Rl bCBpbiByZWNlbnQga2VybmVsIHNvdXJjZSB0cmVlcywgd2hpY2ggaW5jbHVkZXMKPiA+ID4gPiA+ ID4gZG9jdW1lbnRhdGlvbi4gIFRoaXMgaXMgYW4gZXhlY3V0YWJsZSBtb2RlbCwgd2hpY2ggbWVh bnMgdGhhdCB5b3UKPiA+ID4gPiA+ID4gY2FuIGNyZWF0ZSBsaXRtdXMgdGVzdHMgYW5kIGhhdmUg dGhlIG1vZGVsIGZvcm1hbGx5IGFuZAo+ID4gYXV0b21hdGljYWxseQo+ID4gPiA+ID4gPiBldmFs dWF0ZSB0aGVtLgo+ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiBUaGVyZSBhcmUgYWxzbyBhIG51bWJl ciBvZiBwdWJsaWNhdGlvbnMgY292ZXJpbmcgTEtNTToKPiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4g byAgICAgICBBIGZvcm1hbCBrZXJuZWwgbWVtb3J5LW9yZGVyaW5nIG1vZGVsCj4gPiA+ID4gPiA+ ICAgICAgICAgaHR0cHM6Ly9sd24ubmV0L0FydGljbGVzLzcxODYyOC8KPiA+ID4gPiA+ID4gICAg ICAgICBodHRwczovL2x3bi5uZXQvQXJ0aWNsZXMvNzIwNTUwLwo+ID4gPiA+ID4gPgo+ID4gPiA+ ID4gPiAgICAgICAgIFRoZXNlIGNvdmVyIHRoZSByZWxlYXNlIHN0b3JlcyBhbmQgZGVwZW5kZW5j eSBvcmRlcmluZyB0aGF0Cj4gPiA+ID4gPiA+ICAgICAgICAgcHJvdmlkZSBSQ1UncyBwdWJsaXNo LXN1YnNjcmliZSBndWFyYW50ZWVzLgo+ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiAgICAgICAgIEJh Y2t1cCBtYXRlcmlhbCBoZXJlOgo+ID4gPiA+ID4gPgo+ID4gPiA+ID4gPgo+ID4gPiA+ID4gPgo+ ID4gPiA+Cj4gPiBodHRwczovL21pcnJvcnMuZWRnZS5rZXJuZWwub3JnL3B1Yi9saW51eC9rZXJu ZWwvcGVvcGxlL3BhdWxtY2svTFdOTGludXhNTS8KPiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4gICAg ICAgICBXaXRoIHRoZXNlIHR3byBsaWtlbHkgYmVpbmcgb2YgcGFydGljdWxhciBpbnRlcmVzdDoK PiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4KPiA+ID4gPgo+ID4gaHR0cHM6Ly9t aXJyb3JzLmVkZ2Uua2VybmVsLm9yZy9wdWIvbGludXgva2VybmVsL3Blb3BsZS9wYXVsbWNrL0xX TkxpbnV4TU0vUkNVZ3VhcmFudGVlcy5odG1sCj4gPiA+ID4gPiA+Cj4gPiA+ID4gPiA+Cj4gPiA+ ID4KPiA+IGh0dHBzOi8vbWlycm9ycy5lZGdlLmtlcm5lbC5vcmcvcHViL2xpbnV4L2tlcm5lbC9w ZW9wbGUvcGF1bG1jay9MV05MaW51eE1NL3NyY3UuaHRtbAo+ID4gPiA+ID4gPgo+ID4gPiA+ID4g PiBvICAgICAgIEZyaWdodGVuaW5nIFNtYWxsIENoaWxkcmVuIGFuZCBEaXNjb25jZXJ0aW5nIEdy b3duLXVwczoKPiA+ID4gPiA+ID4gQ29uY3VycmVuY3kgaW4gdGhlIExpbnV4IEtlcm5lbAo+ID4g PiA+ID4gPiAgICAgICAgIGh0dHBzOi8vZGwuYWNtLm9yZy9jaXRhdGlvbi5jZm0/aWQ9MzE3NzE1 Ngo+ID4gPiA+ID4gPgo+ID4gaHR0cDovL3d3dzAuY3MudWNsLmFjLnVrL3N0YWZmL2ouYWxnbGF2 ZS9wYXBlcnMvYXNwbG9zMTgucGRmCj4gPiA+ID4gPiA+Cj4gPiA+ID4gPiA+ICAgICAgICAgQmFj a3VwIG1hdGVyaWFsOgo+ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiAgICAgICAgIGh0dHA6Ly9kaXku aW5yaWEuZnIvbGludXgvCj4gPiA+ID4gPiA+Cj4gPiA+ID4gPiA+IG8gICAgICAgV2hvJ3MgYWZy YWlkIG9mIGEgYmlnIGJhZCBvcHRpbWl6aW5nIGNvbXBpbGVyPwo+ID4gPiA+ID4gPiAgICAgICAg IGh0dHBzOi8vbHduLm5ldC9BcnRpY2xlcy83OTMyNTMvCj4gPiA+ID4gPiA+Cj4gPiA+ID4gPiA+ IG8gICAgICAgQ2FsaWJyYXRpbmcgeW91ciBmZWFyIG9mIGJpZyBiYWQgb3B0aW1pemluZyBjb21w aWxlcnMKPiA+ID4gPiA+ID4gICAgICAgICBodHRwczovL2x3bi5uZXQvQXJ0aWNsZXMvNzk5MjE4 Lwo+ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiAgICAgICAgIFRoZXNlIGxhc3QgdHdvIGp1c3RpZnkg dXNlIG9mIG5vcm1hbCBDLWxhbmd1YWdlIGFzc2lnbm1lbnQKPiA+ID4gPiA+ID4gICAgICAgICBz dGF0ZW1lbnRzIHRvIGluaXRpYWxpemUgYW5kIGFjY2VzcyBkYXRhIHJlZmVyZW5jZWQgYnkKPiA+ ID4gPiA+ID4gICAgICAgICBSQ1UtcHJvdGVjdGVkIHBvaW50ZXJzLgo+ID4gPiA+ID4gPgo+ID4g PiA+ID4gPiBUaGVyZSBpcyBhIGxhcmdlIGJvZHkgb2YgbGl0bXVzIHRlc3RzICh0aG91c2FuZHMg b2YgdGhlbSkgaGVyZToKPiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4gICAgICAgICBodHRwczovL2dp dGh1Yi5jb20vcGF1bG1ja3JjdS9saXRtdXMKPiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4gTWFueSBv ZiB0aGVzZSBsaXRtdXMgdGVzdHMgaW52b2x2ZSBSQ1UsIGFuZCB0aGVzZSBjYW4gYmUgbG9jYXRl ZCBieQo+ID4gPiA+ID4gPiBzZWFyY2ggZm9yIGZpbGVzIGNvbnRhaW5pbmcgcmN1X3JlYWRfbG9j aygpLCByY3VfcmVhZF91bmxvY2soKSwKPiA+ID4gPiA+ID4gc3luY2hyb25pemVfcmN1KCksIGFu ZCBzbyBvbi4KPiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4gT3Igd2VyZSB5b3UgbG9va2luZyBmb3Ig c29tZXRoaW5nIGVsc2U/Cj4gPiA+ID4gPiA+Cj4gPiA+ID4gPiA+ICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgVGhhbngsIFBhdWwKPiA+ID4g PiA+ID4KPiA+ID4gPiA+ID4gPiBUaGFua3MsCj4gPiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4gPiBN YXRoaWV1Cj4gPiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4gPiA+PiBhcyBhIHN0YXJ0aW5nIHBvaW50 Lgo+ID4gPiA+ID4gPiA+Cj4gPiA+ID4gPiA+ID4gPj4gVGhhbmtzLAo+ID4gPiA+ID4gPiA+Cj4g PiA+ID4gPiA+ID4gPj4gTWF0aGlldQo+ID4gPiA+ID4gPiA+Cj4gPiA+ID4gPiA+ID4gPj4+IEkg a25vdyB0aGVyZSBhcmUgc29tZSBjb25zaXN0ZW5jeSBtb2RlbHMgaW4gdGhlIGRhdGFiYXNlIGFy ZWEKPiA+ID4gPiAoc3VjaAo+ID4gPiA+ID4gPiBhcyBQUkFNLAo+ID4gPiA+ID4gPiA+ID4+PiBS ZWFkIFVuY29tbWl0dGVkLCBldGMpIGZyb20gWyBodHRwczovL2plcHNlbi5pby9jb25zaXN0ZW5j eQo+ID4gfAo+ID4gPiA+ID4gPiA+ID4+PiBodHRwczovL2plcHNlbi5pby9jb25zaXN0ZW5jeSBd IGFuZCBbMV0uCj4gPiA+ID4gPiA+ID4gPj4+IEhvdyBkb2VzIFJDVSByZWxhdGVkIHRvIHRob3Nl IGNvbnNpc3RlbmN5IG1vZGVscz8KPiA+ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiA+ID4+PiBJIGFs c28gZm91bmQgc29tZSBjb21tZW50cyBvbmxpbmUgKE9uZSBrZXkgZGlzdGluY3Rpb24gaXMKPiA+ IHRoYXQKPiA+ID4gPiBib3RoCj4gPiA+ID4gPiA+IE1WQ0MgYW5kIFJMVQo+ID4gPiA+ID4gPiA+ ID4+PiBwcm92aWRlIG11Y2ggc3Ryb25nZXIgY29uc2lzdGVuY3kgZ3VhcmFudGVlcyB0byByZWFk ZXJzIHRoYW4KPiA+IGRvZXMKPiA+ID4gPiA+ID4gUkNVIC4uLikgKCBbCj4gPiA+ID4gPiA+ID4g Pj4+IGh0dHBzOi8vbHduLm5ldC9BcnRpY2xlcy83NzcwMzYvIHwKPiA+ID4gPiBodHRwczovL2x3 bi5uZXQvQXJ0aWNsZXMvNzc3MDM2Lwo+ID4gPiA+ID4gPiBdICkuCj4gPiA+ID4gPiA+ID4gPj4+ IEkgZG8gbm90IHVuZGVyc3RhbmQgaG93IHdlIHJlYXNvbi9kcmVzaWJlL2NvbXBhcmUgdGhlCj4g PiBjb25zaXN0ZW5jeQo+ID4gPiA+ID4gPiBndWFyYW50ZWVzLiAoCj4gPiA+ID4gPiA+ID4gPj4+ IEkgZXZlbiBkbyBub3Qga25vdyB3aGF0IGNvbnNpc3RlbmN5IGd1YXJhbnRlZXMgcHJvdmlkZWQg YnkKPiA+IFJDVQo+ID4gPiA+ID4gPiBmb3JtYWxseSkKPiA+ID4gPiA+ID4gPiA+Pj4gQ291bGQg c29tZW9uZSBleHBsYWluIHRoaXMgdG8gbWU/Cj4gPiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4gPiA+ Pj4gWzFdIEJhaWxpcywgUC4sIERhdmlkc29uLCBBLiwgRmVrZXRlLCBBLiwgR2hvZHNpLCBBLiwK PiA+ID4gPiBIZWxsZXJzdGVpbiwKPiA+ID4gPiA+ID4gSi4gTS4sICYKPiA+ID4gPiA+ID4gPiA+ Pj4gU3RvaWNhLCBJLiAoMjAxMykuIEhpZ2hseSBhdmFpbGFibGUgdHJhbnNhY3Rpb25zOiBWaXJ0 dWVzIGFuZAo+ID4gPiA+ID4gPiBsaW1pdGF0aW9ucy4KPiA+ID4gPiA+ID4gPiA+Pj4gUHJvY2Vl ZGluZ3Mgb2YgdGhlIFZMREIgRW5kb3dtZW50LCA3KDMpLCAxODEtMTkyLgo+ID4gPiA+ID4gPiA+ Cj4gPiA+ID4gPiA+ID4gPj4+IFRoYW5rcwo+ID4gPiA+ID4gPiA+ID4+PiBZdXhpbgo+ID4gPiA+ ID4gPiA+Cj4gPiA+ID4gPiA+ID4gPj4+IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fCj4gPiA+ID4gPiA+ID4gPj4+IGx0dG5nLWRldiBtYWlsaW5nIGxpc3QK PiA+ID4gPiA+ID4gPiA+Pj4gWyBtYWlsdG86bHR0bmctZGV2QGxpc3RzLmx0dG5nLm9yZyB8Cj4g PiBsdHRuZy1kZXZAbGlzdHMubHR0bmcub3JnIF0KPiA+ID4gPiA+ID4gPiA+Pj4gWyBodHRwczov L2xpc3RzLmx0dG5nLm9yZy9jZ2ktYmluL21haWxtYW4vbGlzdGluZm8vbHR0bmctZGV2Cj4gPiB8 Cj4gPiA+ID4gPiA+ID4gPj4+IGh0dHBzOi8vbGlzdHMubHR0bmcub3JnL2NnaS1iaW4vbWFpbG1h bi9saXN0aW5mby9sdHRuZy1kZXYgXQo+ID4gPiA+ID4gPiA+Cj4gPiA+ID4gPiA+ID4gPj4gLS0K PiA+ID4gPiA+ID4gPiA+PiBNYXRoaWV1IERlc25veWVycwo+ID4gPiA+ID4gPiA+ID4+IEVmZmlj aU9TIEluYy4KPiA+ID4gPiA+ID4gPiA+PiBbIGh0dHA6Ly93d3cuZWZmaWNpb3MuY29tLyB8IGh0 dHA6Ly93d3cuZWZmaWNpb3MuY29tIF0KPiA+ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiA+IC0tCj4g PiA+ID4gPiA+ID4gTWF0aGlldSBEZXNub3llcnMKPiA+ID4gPiA+ID4gPiBFZmZpY2lPUyBJbmMu Cj4gPiA+ID4gPiA+ID4gaHR0cDovL3d3dy5lZmZpY2lvcy5jb20KPiA+ID4gPiA+ID4KPiA+ID4g Pgo+ID4KX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbHR0 bmctZGV2IG1haWxpbmcgbGlzdApsdHRuZy1kZXZAbGlzdHMubHR0bmcub3JnCmh0dHBzOi8vbGlz dHMubHR0bmcub3JnL2NnaS1iaW4vbWFpbG1hbi9saXN0aW5mby9sdHRuZy1kZXYK From mboxrd@z Thu Jan 1 00:00:00 1970 From: Yuxin Ren Subject: Re: RCU consistency guarantees Date: Sun, 15 Dec 2019 17:10:11 -0500 Message-ID: References: <194534011.751.1575629349181.JavaMail.zimbra@efficios.com> <512711764.1078.1575647945136.JavaMail.zimbra@efficios.com> <20191206163052.GG2889@paulmck-ThinkPad-P72> <20191207063730.GN2889@paulmck-ThinkPad-P72> <20191207224232.GR2889@paulmck-ThinkPad-P72> <20191215203019.GN2889@paulmck-ThinkPad-P72> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0938664408393169190==" Return-path: Received: from mail-lf1-x132.google.com (mail-lf1-x132.google.com [IPv6:2a00:1450:4864:20::132]) by lists.lttng.org (Postfix) with ESMTPS id 47bdrY0mx4z19MT for ; Sun, 15 Dec 2019 17:10:36 -0500 (EST) Received: by mail-lf1-x132.google.com with SMTP id y19so2821180lfl.9 for ; Sun, 15 Dec 2019 14:10:36 -0800 (PST) In-Reply-To: <20191215203019.GN2889@paulmck-ThinkPad-P72> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" To: paulmck@kernel.org Cc: lttng-dev List-Id: lttng-dev@lists.lttng.org --===============0938664408393169190== Content-Type: multipart/alternative; boundary="0000000000003b8c6b0599c55dcd" --0000000000003b8c6b0599c55dcd Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sun, Dec 15, 2019 at 3:30 PM Paul E. McKenney wrote= : > On Sat, Dec 14, 2019 at 01:31:31AM -0500, Yuxin Ren wrote: > > Hi Paul > > > > On Sat, Dec 7, 2019 at 5:42 PM Paul E. McKenney > wrote: > > > > > On Sat, Dec 07, 2019 at 03:04:42PM -0500, Yuxin Ren wrote: > > > > Thanks a lot for your help. I have some questions below. > > > > > > > > On Sat, Dec 7, 2019 at 1:37 AM Paul E. McKenney > > > wrote: > > > > > > > > > On Fri, Dec 06, 2019 at 07:00:13PM -0500, Yuxin Ren wrote: > > > > > > Thanks so much for your great help. > > > > > > I definitely will look at those resources and papers! > > > > > > > > > > > > One more thing that I am confused > > > > > > As I mentioned earlier, someone said One key distinction is tha= t > both > > > > > MVCC > > > > > > and RLU provide much stronger consistency guarantees to readers > than > > > does > > > > > > RCU ...) (https://lwn.net/Articles/777036/). > > > > > > > > > > That someone was in fact me. ;-) > > > > > > > > > > > I am not sure if the above statement is correct or not. But in > > > general, > > > > > > How can we compare RCU consistency guarantees to other techniqu= es > > > (such > > > > > as > > > > > > RLU)? > > > > > > How to reason about which one has stronger or weaker guarantees= ? > > > > > > > > > > I suggest starting from the use case. For concreteness, let's > assume > > > > > that we are using a hash table. At one extreme, imagine a use > case in > > > > > which each event makes exactly one hash-table operation. No > > > information > > > > > is carried from one event to the next. (This might well be the > case > > > > > for simple web server.) Such a use case cannot tell the differen= ce > > > > > between RCU on the one hand and MVCC/RLU on the other. > > > > > > > > > > At the other extreme, suppose that each event either accesses or > > > updates > > > > > multiple entries in the hash table. In this case, MVCC/RLU will > rule > > > > > out outcomes that RCU would permit. For example, suppose we had > four > > > > > events accessing two different elements in different buckets of t= he > > > > > hash table: > > > > > > > > > > E1: Adds 32 to the hash table. > > > > > E2: Adds 1729 to the hash table. > > > > > E3: Within a read-side critical section, looks up 32 then > 1729. > > > > > E4: Within a read-side critical section, looks up 1729 > then 32. > > > > > > > > > > Given either MVCC or RLU, it will not be possible for E3 to find > 32 but > > > > > not 1729 and at the same time for E4 to find 1729 but not 32. > Given > > > RCU, > > > > > this outcome is possible. > > > > > > > > > When you say "Within a read-side section", do you mean within a > single > > > same > > > > read section? such as > > > > > > > > > read_lock() > > > > > lookup(32) > > > > > lookup(1729) > > > > > read_unlock() > > > > > > > > > > > > How about putting two lookups into two read-side sections? Do we > still > > > have > > > > the problem, then? > > > > > > > > > read_lock() > > > > > lookup(32) > > > > > read_unlock() > > > > > read_lock() > > > > > lookup(1729) > > > > > read_unlock() > > > > > > Without in any way agreeing with your characterization of this as a > > > problem, because rcu_read_lock() and rcu_read_unlock() provide > > > absolutely no ordering guarantees in the absence of a grace period, > > > any non-grace-period-related reordering that can happen with a single > > > RCU read-side critical section can also happen when that critical > > > section is split in two as you have done above. > > > > > > > Could you kindly give me more clues why RCU can see such reorder, > while > > > RLU > > > > can prevent it? > > > > > > Here are minimal C-language implementations for RCU that can (and are= ) > > > actually used: > > > > > Great. We use the same thing in our real-time work [1] > > It has been a popular choice for 40 years now. ;-) > > > > #define rcu_read_lock() > > > #define rcu_read_unlock() > > > > > > Please compare these to the read-side markers presented in the RLU > paper, > > > and then tell me your thoughts on the answer to your question. ;-) > > > > > I submit my homework here, but I do not think I did it well. > > 1. I believe in the default URCU implementation, it has memory barrier > > inside the read_lock / read_unlock. > > It certainly was at one time. But you would of course need to check > to see what any given worker actually used. > > > 2. From the RLU paper, it shows the code for read-side operation. > > RLU_READER_LOCK(ctx) > > ctx.is-writer=E2=86=90false > > ctx.run-cnt=E2=86=90ctx.run-cnt+1. > > memory fence > > ctx.local-clock=E2=86=90global-clock > > RLU_READER_UNLOCK(ctx) > > ctx.run-cnt=E2=86=90ctx.run-cnt+1 > > if (ctx.is-writer) > > RLU_COMMIT_WRITE_LOG(ctx) > > > > We can ignore the writer check, as in our case, the reader never do > update. > > My understanding is that read-side operations are only used to facilita= te > > the quiescence detection. > > the run cnt is used to decide if a thread is active (if it is currently > > inside a read-side section). > > the local clock is used to check if the active thread goes into the > > read-side section very late, so it does not prevent reclaiming memory > > unlinked before it enters the read section. > > However i see no difference between RLU and RCU read-side operation > > regarding consistency guarantee. > > Could you kindly teach me more? > > In order to teach you more, I do need some help from you. I have posted > a number of URLs earlier in this email thread. Could you please tell > me what you learned from each of them? > Sure, I will definitely learn those resources you provided, and come back to you if I have more questions. Simply speaking, I failed to find a clear and formal consistency model/guarantee of RCU quickly. The difficulty for me is, for most people, at their first glance, RCU only has relax/weak consistency guarantee. So whenever I suggest to use RCU to others, they always ask like Does RCU provide Serializability? .. maybe not. Does RCU provide Linearizability? ... maybe not ... OK, what does RCU provide? .. I am not sure. I even cannot find a single document which has an accurate answer. Let me give more examples. 1. When I recommend using read-write lock, people are happy to use it. However, when I further suggest to replace rw lock with RCU, then they hesitate to do so. Because they feel RCU is weaker than rw lock, and do not know what precisely RCU guarantee. 2. In contrast, in the database area, it has relative clear definition of consistency levels (such as https://jepsen.io/consistency), and users are easy to assert if they can use certain database configuration based on its consistency model. Again, I really appreciate your patience and communication, hope those does not bother you a lot. Thanks Best, Yuxin > > Thanx, Pa= ul > > > BTW, the RLU read-side ops are similar, but not efficient, comparing to > our > > Parsec work [1, 2] > > [1] Yuxin Ren, G. Liu, G. Parmer, B. Brandenburg, Scalable Memory > > Reclamation for Multi-Core, Real-Time Systems, in Proceedings of the 24= th > > IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS= ), > > 2018 > > [2] Q. Wang, T. Stamler, and G. Parmer, Parallel Sections: Scaling > > System-Level Data-Structures, in Proceedings of European Conference on > > Computer Systems (EuroSys), 2016 > > > > Thanks, > > Best, > > Yuxin > > > > > > > > This is because MVCC and RLU provide readers a consistent view of > > > > > the updates, and RCU does not. Of course, it is often the case > that a > > > > > consistent view is not needed, in which case the MVCC and RLU > > > guarantees > > > > > are incurring read-side overhead for no reason. But if the use > case > > > > > requires consistent readers, RCU is not an option. > > > > > > > > > > The reason a consistent view is not always needed is that > > > speed-of-light > > > > > delays make it impossible to provide a consistent view of the > outside > > > > > world. In the common case where the use case interacts with the > > > > > outside world, the algorithms absolutely must be designed to > tolerate > > > > > inconsistency, which opens the door to things like RCU. > > > > > > > > I am confused here. I think speed-of-light delays happen everywhere= , > not > > > > only bound to RCU, but also any other synchronization approach (su= ch > > > RLU). > > > > If so, how do others (RLU) provide consistent views? > > > > > > You just stated the answer. Now it is only necessary for you to inve= st > > > the time, effort, and thought to fully understand it. To help with > this, > > > the following paragraph provides another hint: > > > > > > Yes, you are quite right, speed-of-light delays between the > > > outside world and the computer affect RLU just as surely as > they > > > do RCU. This means that the additional consistency guarantee= s > > > provided by RLU must necessarily be limited to the confines o= f > the > > > computer system in question. Taking this one step further, > there > > > are also speed-of-light delays within the computer. Therefor= e, > > > in the general case, RLU can provide its consistency > guarantees, > > > even within the confines of the computer system, only at the > > > expense of incurring delays. Because RCU does not provide > RLU's > > > consistency guarantees, it need not incur RLU's delays. > > > > > > This is not a new line of reasoning, see for example: > > > > > > @article{Herlihy:1996:LCN:1063369.1063372 > > > ,author =3D {Herlihy, Maurice and Shavit, Nir and Waarts, Orli} > > > ,title =3D {Linearizable counting networks} > > > ,journal =3D {Distrib. Comput.} > > > ,volume =3D {9} > > > ,issue =3D {4} > > > ,month =3D {February} > > > ,year =3D {1996} > > > ,issn =3D {0178-2770} > > > ,pages =3D {193--203} > > > ,numpages =3D {11} > > > ,url =3D {http://portal.acm.org/citation.cfm?id=3D1063369.1063372} > > > ,doi =3D {10.1007/s004460050019} > > > ,acmid =3D {1063372} > > > ,publisher =3D {Springer-Verlag} > > > ,address =3D {London, UK} > > > ,keywords =3D {concurrency, contention, counting networks, data > structures, > > > linearizability} > > > } > > > > > > Thanx, Paul > > > > > > > Thanks for your education. > > > > Yuxin > > > > > > > > > > > > > > Thanx, Pa= ul > > > > > > > > > > > Thanks > > > > > > Yuxin > > > > > > > > > > > > On Fri, Dec 6, 2019 at 11:30 AM Paul E. McKenney < > paulmck@kernel.org > > > > > > > > > wrote: > > > > > > > > > > > > > On Fri, Dec 06, 2019 at 10:59:05AM -0500, Mathieu Desnoyers > wrote: > > > > > > > > ----- On Dec 6, 2019, at 3:51 PM, Yuxin Ren < > ryx@gwmail.gwu.edu> > > > > > wrote: > > > > > > > > > > > > > > > > > On Fri, Dec 6, 2019 at 5:49 AM Mathieu Desnoyers < [ > > > > > > > > > mailto:mathieu.desnoyers@efficios.com | > > > > > mathieu.desnoyers@efficios.com > > > > > > > ] > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > >> ----- On Dec 5, 2019, at 8:17 PM, Yuxin Ren < [ mailto: > > > > > > > ryx@gwmail.gwu.edu | > > > > > > > > >> ryx@gwmail.gwu.edu ] > wrote: > > > > > > > > > > > > > > > > >>> Hi, > > > > > > > > >>> I am a student, and learning RCU now, but still know ve= ry > > > little > > > > > > > about it. > > > > > > > > >>> Are there any documents/papers/materials which > (in)formally > > > > > define > > > > > > > and explain > > > > > > > > >>> RCU consistency guarantees? > > > > > > > > > > > > > > > > >> You may want to have a look at > > > > > > > > > > > > > > > > >> User-Level Implementations of Read-Copy Update > > > > > > > > >> Article in IEEE Transactions on Parallel and Distributed > > > Systems > > > > > > > 23(2):375 - 382 > > > > > > > > >> =C2=B7 March 2012 > > > > > > > > > > > > > > > > > Thanks for your info. > > > > > > > > > However, I do not think URCU talks about any consistency > model > > > > > > > formally. > > > > > > > > > > > > > > > > > From previous communication with Paul, he said RCU is not > > > designed > > > > > for > > > > > > > > > linearizability, and it is totally acceptable that RCU is > not > > > > > > > linearizable. > > > > > > > > > However, I am curious how to accurately/formally > Characterize > > > RCU > > > > > > > consistency > > > > > > > > > model/guarantees > > > > > > > > > > > > > > > > Adding Paul E. McKenney in CC. > > > > > > > > > > > > > > > > I am referring to the section "Overview of RCU semantics" i= n > the > > > > > paper. > > > > > > > Not sure it has the level of > > > > > > > > formality you are looking for though. Paul, do you have > pointers > > > to > > > > > > > additional material ? > > > > > > > > > > > > > > Indeed I do! The Linux kernel memory model (LKMM) includes > RCU. > > > It is > > > > > > > in tools/memory-model in recent kernel source trees, which > includes > > > > > > > documentation. This is an executable model, which means that > you > > > > > > > can create litmus tests and have the model formally and > > > automatically > > > > > > > evaluate them. > > > > > > > > > > > > > > There are also a number of publications covering LKMM: > > > > > > > > > > > > > > o A formal kernel memory-ordering model > > > > > > > https://lwn.net/Articles/718628/ > > > > > > > https://lwn.net/Articles/720550/ > > > > > > > > > > > > > > These cover the release stores and dependency orderin= g > that > > > > > > > provide RCU's publish-subscribe guarantees. > > > > > > > > > > > > > > Backup material here: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinuxM= M/ > > > > > > > > > > > > > > With these two likely being of particular interest: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinuxM= M/RCUguarantees.html > > > > > > > > > > > > > > > > > > > > > > > https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinuxM= M/srcu.html > > > > > > > > > > > > > > o Frightening Small Children and Disconcerting Grown-up= s: > > > > > > > Concurrency in the Linux Kernel > > > > > > > https://dl.acm.org/citation.cfm?id=3D3177156 > > > > > > > > > > http://www0.cs.ucl.ac.uk/staff/j.alglave/papers/asplos18.pdf > > > > > > > > > > > > > > Backup material: > > > > > > > > > > > > > > http://diy.inria.fr/linux/ > > > > > > > > > > > > > > o Who's afraid of a big bad optimizing compiler? > > > > > > > https://lwn.net/Articles/793253/ > > > > > > > > > > > > > > o Calibrating your fear of big bad optimizing compilers > > > > > > > https://lwn.net/Articles/799218/ > > > > > > > > > > > > > > These last two justify use of normal C-language > assignment > > > > > > > statements to initialize and access data referenced b= y > > > > > > > RCU-protected pointers. > > > > > > > > > > > > > > There is a large body of litmus tests (thousands of them) her= e: > > > > > > > > > > > > > > https://github.com/paulmckrcu/litmus > > > > > > > > > > > > > > Many of these litmus tests involve RCU, and these can be > located by > > > > > > > search for files containing rcu_read_lock(), rcu_read_unlock(= ), > > > > > > > synchronize_rcu(), and so on. > > > > > > > > > > > > > > Or were you looking for something else? > > > > > > > > > > > > > > Thanx= , > Paul > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Mathieu > > > > > > > > > > > > > > > > >> as a starting point. > > > > > > > > > > > > > > > > >> Thanks, > > > > > > > > > > > > > > > > >> Mathieu > > > > > > > > > > > > > > > > >>> I know there are some consistency models in the databas= e > area > > > > > (such > > > > > > > as PRAM, > > > > > > > > >>> Read Uncommitted, etc) from [ > https://jepsen.io/consistency > > > | > > > > > > > > >>> https://jepsen.io/consistency ] and [1]. > > > > > > > > >>> How does RCU related to those consistency models? > > > > > > > > > > > > > > > > >>> I also found some comments online (One key distinction = is > > > that > > > > > both > > > > > > > MVCC and RLU > > > > > > > > >>> provide much stronger consistency guarantees to readers > than > > > does > > > > > > > RCU ...) ( [ > > > > > > > > >>> https://lwn.net/Articles/777036/ | > > > > > https://lwn.net/Articles/777036/ > > > > > > > ] ). > > > > > > > > >>> I do not understand how we reason/dresibe/compare the > > > consistency > > > > > > > guarantees. ( > > > > > > > > >>> I even do not know what consistency guarantees provided > by > > > RCU > > > > > > > formally) > > > > > > > > >>> Could someone explain this to me? > > > > > > > > > > > > > > > > >>> [1] Bailis, P., Davidson, A., Fekete, A., Ghodsi, A., > > > > > Hellerstein, > > > > > > > J. M., & > > > > > > > > >>> Stoica, I. (2013). Highly available transactions: > Virtues and > > > > > > > limitations. > > > > > > > > >>> Proceedings of the VLDB Endowment, 7(3), 181-192. > > > > > > > > > > > > > > > > >>> Thanks > > > > > > > > >>> Yuxin > > > > > > > > > > > > > > > > >>> _______________________________________________ > > > > > > > > >>> lttng-dev mailing list > > > > > > > > >>> [ mailto:lttng-dev@lists.lttng.org | > > > lttng-dev@lists.lttng.org ] > > > > > > > > >>> [ > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > > > | > > > > > > > > >>> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ] > > > > > > > > > > > > > > > > >> -- > > > > > > > > >> Mathieu Desnoyers > > > > > > > > >> EfficiOS Inc. > > > > > > > > >> [ http://www.efficios.com/ | http://www.efficios.com ] > > > > > > > > > > > > > > > > -- > > > > > > > > Mathieu Desnoyers > > > > > > > > EfficiOS Inc. > > > > > > > > http://www.efficios.com > > > > > > > > > > > > > > > > --0000000000003b8c6b0599c55dcd Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Sun, Dec 15, 2019 at 3:30 PM Paul = E. McKenney <pau= lmck@kernel.org> wrote:
On Sat, Dec 14, 2019 at 01:31:31AM -0500, Yuxin Ren wrote: > Hi Paul
>
> On Sat, Dec 7, 2019 at 5:42 PM Paul E. McKenney <paulmck@kernel.org> wrote:
>
> > On Sat, Dec 07, 2019 at 03:04:42PM -0500, Yuxin Ren wrote:
> > > Thanks a lot for your help. I have some questions below.
> > >
> > > On Sat, Dec 7, 2019 at 1:37 AM Paul E. McKenney <paulmck@kernel.org>=
> > wrote:
> > >
> > > > On Fri, Dec 06, 2019 at 07:00:13PM -0500, Yuxin Ren wro= te:
> > > > > Thanks so much for your great help.
> > > > > I definitely will look at those resources and pape= rs!
> > > > >
> > > > > One more thing that I am confused
> > > > > As I mentioned earlier, someone said One key disti= nction is that both
> > > > MVCC
> > > > > and RLU provide much stronger consistency guarante= es to readers than
> > does
> > > > > RCU ...) (https://lwn.net/Articles/777036/<= /a>).
> > > >
> > > > That someone was in fact me.=C2=A0 ;-)
> > > >
> > > > > I am not sure if the above statement is correct or= not. But in
> > general,
> > > > > How can we compare RCU consistency guarantees to o= ther techniques
> > (such
> > > > as
> > > > > RLU)?
> > > > > How to reason about which one has stronger or weak= er guarantees?
> > > >
> > > > I suggest starting from the use case.=C2=A0 For concret= eness, let's assume
> > > > that we are using a hash table.=C2=A0 At one extreme, i= magine a use case in
> > > > which each event makes exactly one hash-table operation= .=C2=A0 No
> > information
> > > > is carried from one event to the next.=C2=A0 (This migh= t well be the case
> > > > for simple web server.)=C2=A0 Such a use case cannot te= ll the difference
> > > > between RCU on the one hand and MVCC/RLU on the other.<= br> > > > >
> > > > At the other extreme, suppose that each event either ac= cesses or
> > updates
> > > > multiple entries in the hash table.=C2=A0 In this case,= MVCC/RLU will rule
> > > > out outcomes that RCU would permit.=C2=A0 For example, = suppose we had four
> > > > events accessing two different elements in different bu= ckets of the
> > > > hash table:
> > > >
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0E1: Adds 32 to the has= h table.
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0E2: Adds 1729 to the h= ash table.
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0E3: Within a read-side= critical section, looks up 32 then 1729.
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0E4: Within a read-side= critical section, looks up 1729 then 32.
> > > >
> > > > Given either MVCC or RLU, it will not be possible for E= 3 to find 32 but
> > > > not 1729 and at the same time for E4 to find 1729 but n= ot 32.=C2=A0 Given
> > RCU,
> > > > this outcome is possible.
> > > >
> > > When you say "Within a read-side section", do you = mean within a single
> > same
> > > read section? such as
> > >
> > > > read_lock()
> > > > lookup(32)
> > > > lookup(1729)
> > > > read_unlock()
> > >
> > >
> > > How about putting two lookups into two read-side sections? D= o we still
> > have
> > > the problem, then?
> > >
> > > > read_lock()
> > > > lookup(32)
> > > > read_unlock()
> > > > read_lock()
> > > > lookup(1729)
> > > > read_unlock()
> >
> > Without in any way agreeing with your characterization of this as= a
> > problem, because rcu_read_lock() and rcu_read_unlock() provide > > absolutely no ordering guarantees in the absence of a grace perio= d,
> > any non-grace-period-related reordering that can happen with a si= ngle
> > RCU read-side critical section can also happen when that critical=
> > section is split in two as you have done above.
> >
> > > Could you kindly give me more clues why RCU can see such reo= rder, while
> > RLU
> > > can prevent it?
> >
> > Here are minimal C-language implementations for RCU that can (and= are)
> > actually used:
> >
> Great. We use the same thing in our real-time work [1]

It has been a popular choice for 40 years now.=C2=A0 ;-)

> > #define rcu_read_lock()
> > #define rcu_read_unlock()
> >
> > Please compare these to the read-side markers presented in the RL= U paper,
> > and then tell me your thoughts on the answer to your question.=C2= =A0 ;-)
> >
> I submit my homework here, but I do not think I did it well.
> 1. I believe in the default URCU implementation, it has memory barrier=
> inside the read_lock / read_unlock.

It certainly was at one time.=C2=A0 But you would of course need to check to see what any given worker actually used.

> 2. From the RLU paper, it shows the code for read-side operation.
> RLU_READER_LOCK(ctx)
>=C2=A0 =C2=A0 =C2=A0ctx.is-writer=E2=86=90false
>=C2=A0 =C2=A0 =C2=A0ctx.run-cnt=E2=86=90ctx.run-cnt+1.
>=C2=A0 =C2=A0 =C2=A0memory fence
>=C2=A0 =C2=A0 ctx.local-clock=E2=86=90global-clock
> RLU_READER_UNLOCK(ctx)
>=C2=A0 =C2=A0 =C2=A0ctx.run-cnt=E2=86=90ctx.run-cnt+1
>=C2=A0 =C2=A0 =C2=A0if (ctx.is-writer)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0RLU_COMMIT_WRITE_LOG(ctx)
>
> We can ignore the writer check, as in our case, the reader never do up= date.
> My understanding is that read-side operations are only used to facilit= ate
> the quiescence detection.
> the run cnt is used to decide if a thread is active (if it is currentl= y
> inside a read-side section).
> the local clock is used to check if the active thread goes into the > read-side section very late, so it does not prevent reclaiming memory<= br> > unlinked before it enters the read section.
> However i see no difference between RLU and RCU read-side operation > regarding consistency guarantee.
> Could you kindly teach me more?

In order to teach you more, I do need some help from you.=C2=A0 I have post= ed
a number of URLs earlier in this email thread.=C2=A0 Could you please tell<= br> me what you learned from each of them?

2. In= contrast, in the database area, it has relative clear definition=C2=A0of c= onsistency levels (such as=C2=A0h= ttps://jepsen.io/consistency), and users are easy to assert if they can= use certain database configuration based on its consistency model.

Again, I really appreciate=C2=A0your patience and communi= cation, hope those does not bother=C2=A0you a lot.

Thanks
Best,
Yuxin

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 Thanx, Paul

> BTW, the RLU read-side ops are similar, but not efficient, comparing t= o our
> Parsec work [1, 2]
> [1] Yuxin Ren, G. Liu, G. Parmer, B. Brandenburg, Scalable Memory
> Reclamation for Multi-Core, Real-Time Systems, in Proceedings of the 2= 4th
> IEEE Real-Time and Embedded Technology and Applications Symposium (RTA= S),
> 2018
> [2] Q. Wang, T. Stamler, and G. Parmer, Parallel Sections: Scaling
> System-Level Data-Structures, in Proceedings of European Conference on=
> Computer Systems (EuroSys), 2016
>
> Thanks,
> Best,
> Yuxin
>
>
> > > This is because MVCC and RLU provide readers a consistent vi= ew of
> > > > the updates, and RCU does not.=C2=A0 Of course, it is o= ften the case that a
> > > > consistent view is not needed, in which case the MVCC a= nd RLU
> > guarantees
> > > > are incurring read-side overhead for no reason.=C2=A0 B= ut if the use case
> > > > requires consistent readers, RCU is not an option.
> > > >
> > > > The reason a consistent view is not always needed is th= at
> > speed-of-light
> > > > delays make it impossible to provide a consistent view = of the outside
> > > > world.=C2=A0 In the common case where the use case inte= racts with the
> > > > outside world, the algorithms absolutely must be design= ed to tolerate
> > > > inconsistency, which opens the door to things like RCU.=
> > >
> > > I am confused here. I think speed-of-light delays happen eve= rywhere, not
> > > only bound to RCU, but also=C2=A0 any other synchronization = approach (such
> > RLU).
> > > If so, how do others (RLU) provide consistent views?
> >
> > You just stated the answer.=C2=A0 Now it is only necessary for yo= u to invest
> > the time, effort, and thought to fully understand it.=C2=A0 To he= lp with this,
> > the following paragraph provides another hint:
> >
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Yes, you are quite right, speed-= of-light delays between the
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0outside world and the computer a= ffect RLU just as surely as they
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0do RCU.=C2=A0 This means that th= e additional consistency guarantees
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0provided by RLU must necessarily= be limited to the confines of the
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0computer system in question.=C2= =A0 Taking this one step further, there
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0are also speed-of-light delays w= ithin the computer.=C2=A0 Therefore,
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0in the general case, RLU can pro= vide its consistency guarantees,
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0even within the confines of the = computer system, only at the
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0expense of incurring delays.=C2= =A0 Because RCU does not provide RLU's
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0consistency guarantees, it need = not incur RLU's delays.
> >
> > This is not a new line of reasoning, see for example:
> >
> > @article{Herlihy:1996:LCN:1063369.1063372
> > ,author =3D {Herlihy, Maurice and Shavit, Nir and Waarts, Orli} > > ,title =3D {Linearizable counting networks}
> > ,journal =3D {Distrib. Comput.}
> > ,volume =3D {9}
> > ,issue =3D {4}
> > ,month =3D {February}
> > ,year =3D {1996}
> > ,issn =3D {0178-2770}
> > ,pages =3D {193--203}
> > ,numpages =3D {11}
> > ,url =3D {http://portal.acm.org/cit= ation.cfm?id=3D1063369.1063372}
> > ,doi =3D {10.1007/s004460050019}
> > ,acmid =3D {1063372}
> > ,publisher =3D {Springer-Verlag}
> > ,address =3D {London, UK}
> > ,keywords =3D {concurrency, contention, counting networks, data s= tructures,
> > linearizability}
> > }
> >
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Thanx, Paul > >
> > > Thanks for your education.
> > > Yuxin
> > >
> > > >
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Th= anx, Paul
> > > >
> > > > > Thanks
> > > > > Yuxin
> > > > >
> > > > > On Fri, Dec 6, 2019 at 11:30 AM Paul E. McKenney &= lt;paulmck@kernel.o= rg
> > >
> > > > wrote:
> > > > >
> > > > > > On Fri, Dec 06, 2019 at 10:59:05AM -0500, Mat= hieu Desnoyers wrote:
> > > > > > > ----- On Dec 6, 2019, at 3:51 PM, Yuxin = Ren <ryx@gwmail.= gwu.edu>
> > > > wrote:
> > > > > > >
> > > > > > > > On Fri, Dec 6, 2019 at 5:49 AM Math= ieu Desnoyers < [
> > > > > > > > mailto:mathieu.desnoyers@efficios.com = |
> > > > mathieu.desnoyers@efficios.com
> > > > > > ] >
> > > > > > > > wrote:
> > > > > > >
> > > > > > > >> ----- On Dec 5, 2019, at 8:17 P= M, Yuxin Ren < [ mailto:
> > > > > > ryx@gwmail.gwu.edu |
> > > > > > > >> ryx@gwmail.gwu.edu ] > wrote:
> > > > > > >
> > > > > > > >>> Hi,
> > > > > > > >>> I am a student, and learnin= g RCU now, but still know very
> > little
> > > > > > about it.
> > > > > > > >>> Are there any documents/pap= ers/materials which (in)formally
> > > > define
> > > > > > and explain
> > > > > > > >>> RCU consistency guarantees?=
> > > > > > >
> > > > > > > >> You may want to have a look at<= br> > > > > > > >
> > > > > > > >> User-Level Implementations of R= ead-Copy Update
> > > > > > > >> Article in IEEE Transactions on= Parallel and Distributed
> > Systems
> > > > > > 23(2):375 - 382
> > > > > > > >> =C2=B7 March 2012
> > > > > > >
> > > > > > > > Thanks for your info.
> > > > > > > > However, I do not think URCU talks = about any consistency model
> > > > > > formally.
> > > > > > >
> > > > > > > > From previous communication with Pa= ul, he said RCU is not
> > designed
> > > > for
> > > > > > > > linearizability, and it is totally = acceptable that RCU is not
> > > > > > linearizable.
> > > > > > > > However, I am curious how to accura= tely/formally Characterize
> > RCU
> > > > > > consistency
> > > > > > > > model/guarantees
> > > > > > >
> > > > > > > Adding Paul E. McKenney in CC.
> > > > > > >
> > > > > > > I am referring to the section "Over= view of RCU semantics" in the
> > > > paper.
> > > > > > Not sure it has the level of
> > > > > > > formality you are looking for though. Pa= ul, do you have pointers
> > to
> > > > > > additional material ?
> > > > > >
> > > > > > Indeed I do!=C2=A0 The Linux kernel memory mo= del (LKMM) includes RCU.
> > It is
> > > > > > in tools/memory-model in recent kernel source= trees, which includes
> > > > > > documentation.=C2=A0 This is an executable mo= del, which means that you
> > > > > > can create litmus tests and have the model fo= rmally and
> > automatically
> > > > > > evaluate them.
> > > > > >
> > > > > > There are also a number of publications cover= ing LKMM:
> > > > > >
> > > > > > o=C2=A0 =C2=A0 =C2=A0 =C2=A0A formal kernel m= emory-ordering model
> > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0https= ://lwn.net/Articles/718628/
> > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0https= ://lwn.net/Articles/720550/
> > > > > >
> > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0These cover = the release stores and dependency ordering that
> > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0provide RCU&= #39;s publish-subscribe guarantees.
> > > > > >
> > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Backup mater= ial here:
> > > > > >
> > > > > >
> > > > > >
> > > >
> > https://mirrors= .edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinuxMM/
> > > > > >
> > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0With these t= wo likely being of particular interest:
> > > > > >
> > > > > >
> > > > > >
> > > >
> > https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinux= MM/RCUguarantees.html
> > > > > >
> > > > > >
> > > >
> > https:= //mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinuxMM/srcu.h= tml
> > > > > >
> > > > > > o=C2=A0 =C2=A0 =C2=A0 =C2=A0Frightening Small= Children and Disconcerting Grown-ups:
> > > > > > Concurrency in the Linux Kernel
> > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0https://dl.acm.org/citation.cfm?id=3D3177156
> > > > > >
> > http://www0.cs.ucl.ac.uk/staff= /j.alglave/papers/asplos18.pdf
> > > > > >
> > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Backup mater= ial:
> > > > > >
> > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0http://diy.= inria.fr/linux/
> > > > > >
> > > > > > o=C2=A0 =C2=A0 =C2=A0 =C2=A0Who's afraid = of a big bad optimizing compiler?
> > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0https= ://lwn.net/Articles/793253/
> > > > > >
> > > > > > o=C2=A0 =C2=A0 =C2=A0 =C2=A0Calibrating your = fear of big bad optimizing compilers
> > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0https= ://lwn.net/Articles/799218/
> > > > > >
> > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0These last t= wo justify use of normal C-language assignment
> > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0statements t= o initialize and access data referenced by
> > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0RCU-protecte= d pointers.
> > > > > >
> > > > > > There is a large body of litmus tests (thousa= nds of them) here:
> > > > > >
> > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0h= ttps://github.com/paulmckrcu/litmus
> > > > > >
> > > > > > Many of these litmus tests involve RCU, and t= hese can be located by
> > > > > > search for files containing rcu_read_lock(), = rcu_read_unlock(),
> > > > > > synchronize_rcu(), and so on.
> > > > > >
> > > > > > Or were you looking for something else?
> > > > > >
> > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0Thanx, Paul
> > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Mathieu
> > > > > > >
> > > > > > > >> as a starting point.
> > > > > > >
> > > > > > > >> Thanks,
> > > > > > >
> > > > > > > >> Mathieu
> > > > > > >
> > > > > > > >>> I know there are some consi= stency models in the database area
> > > > (such
> > > > > > as PRAM,
> > > > > > > >>> Read Uncommitted, etc) from= [ https://jepsen.io/consistency
> > |
> > > > > > > >>> https://jepsen.io/consi= stency ] and [1].
> > > > > > > >>> How does RCU related to tho= se consistency models?
> > > > > > >
> > > > > > > >>> I also found some comments = online (One key distinction is
> > that
> > > > both
> > > > > > MVCC and RLU
> > > > > > > >>> provide much stronger consi= stency guarantees to readers than
> > does
> > > > > > RCU ...) ( [
> > > > > > > >>> https://lwn.net/Arti= cles/777036/ |
> > > > https://lwn.net/Articles/777036/
> > > > > > ] ).
> > > > > > > >>> I do not understand how we = reason/dresibe/compare the
> > consistency
> > > > > > guarantees. (
> > > > > > > >>> I even do not know what con= sistency guarantees provided by
> > RCU
> > > > > > formally)
> > > > > > > >>> Could someone explain this = to me?
> > > > > > >
> > > > > > > >>> [1] Bailis, P., Davidson, A= ., Fekete, A., Ghodsi, A.,
> > > > Hellerstein,
> > > > > > J. M., &
> > > > > > > >>> Stoica, I. (2013). Highly a= vailable transactions: Virtues and
> > > > > > limitations.
> > > > > > > >>> Proceedings of the VLDB End= owment, 7(3), 181-192.
> > > > > > >
> > > > > > > >>> Thanks
> > > > > > > >>> Yuxin
> > > > > > >
> > > > > > > >>> ___________________________= ____________________
> > > > > > > >>> lttng-dev mailing list
> > > > > > > >>> [ mailto:lttng-dev@lists.lttng.org = |
> > lt= tng-dev@lists.lttng.org ]
> > > > > > > >>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> > |
> > > > > > > >>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ]
> > > > > > >
> > > > > > > >> --
> > > > > > > >> Mathieu Desnoyers
> > > > > > > >> EfficiOS Inc.
> > > > > > > >> [ http://www.efficios.com/ |= h= ttp://www.efficios.com ]
> > > > > > >
> > > > > > > --
> > > > > > > Mathieu Desnoyers
> > > > > > > EfficiOS Inc.
> > > > > > > http://www.efficios.com
> > > > > >
> > > >
> >
--0000000000003b8c6b0599c55dcd-- --===============0938664408393169190== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev --===============0938664408393169190==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Paul E. McKenney" Subject: Re: RCU consistency guarantees Date: Sun, 15 Dec 2019 16:57:39 -0800 Message-ID: <20191216005739.GP2889__31933.4573506907$1576457879$gmane$org@paulmck-ThinkPad-P72> References: <512711764.1078.1575647945136.JavaMail.zimbra@efficios.com> <20191206163052.GG2889@paulmck-ThinkPad-P72> <20191207063730.GN2889@paulmck-ThinkPad-P72> <20191207224232.GR2889@paulmck-ThinkPad-P72> <20191215203019.GN2889@paulmck-ThinkPad-P72> Reply-To: paulmck@kernel.org Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by lists.lttng.org (Postfix) with ESMTPS id 47bjYJ6JVrz18xq for ; Sun, 15 Dec 2019 19:57:40 -0500 (EST) Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" To: Yuxin Ren Cc: lttng-dev List-Id: lttng-dev@lists.lttng.org T24gU3VuLCBEZWMgMTUsIDIwMTkgYXQgMDU6MTA6MTFQTSAtMDUwMCwgWXV4aW4gUmVuIHdyb3Rl Ogo+IE9uIFN1biwgRGVjIDE1LCAyMDE5IGF0IDM6MzAgUE0gUGF1bCBFLiBNY0tlbm5leSA8cGF1 bG1ja0BrZXJuZWwub3JnPiB3cm90ZToKPiAKPiA+IE9uIFNhdCwgRGVjIDE0LCAyMDE5IGF0IDAx OjMxOjMxQU0gLTA1MDAsIFl1eGluIFJlbiB3cm90ZToKPiA+ID4gSGkgUGF1bAo+ID4gPgo+ID4g PiBPbiBTYXQsIERlYyA3LCAyMDE5IGF0IDU6NDIgUE0gUGF1bCBFLiBNY0tlbm5leSA8cGF1bG1j a0BrZXJuZWwub3JnPgo+ID4gd3JvdGU6Cj4gPiA+Cj4gPiA+ID4gT24gU2F0LCBEZWMgMDcsIDIw MTkgYXQgMDM6MDQ6NDJQTSAtMDUwMCwgWXV4aW4gUmVuIHdyb3RlOgo+ID4gPiA+ID4gVGhhbmtz IGEgbG90IGZvciB5b3VyIGhlbHAuIEkgaGF2ZSBzb21lIHF1ZXN0aW9ucyBiZWxvdy4KPiA+ID4g PiA+Cj4gPiA+ID4gPiBPbiBTYXQsIERlYyA3LCAyMDE5IGF0IDE6MzcgQU0gUGF1bCBFLiBNY0tl bm5leSA8cGF1bG1ja0BrZXJuZWwub3JnPgo+ID4gPiA+IHdyb3RlOgo+ID4gPiA+ID4KPiA+ID4g PiA+ID4gT24gRnJpLCBEZWMgMDYsIDIwMTkgYXQgMDc6MDA6MTNQTSAtMDUwMCwgWXV4aW4gUmVu IHdyb3RlOgo+ID4gPiA+ID4gPiA+IFRoYW5rcyBzbyBtdWNoIGZvciB5b3VyIGdyZWF0IGhlbHAu Cj4gPiA+ID4gPiA+ID4gSSBkZWZpbml0ZWx5IHdpbGwgbG9vayBhdCB0aG9zZSByZXNvdXJjZXMg YW5kIHBhcGVycyEKPiA+ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiA+IE9uZSBtb3JlIHRoaW5nIHRo YXQgSSBhbSBjb25mdXNlZAo+ID4gPiA+ID4gPiA+IEFzIEkgbWVudGlvbmVkIGVhcmxpZXIsIHNv bWVvbmUgc2FpZCBPbmUga2V5IGRpc3RpbmN0aW9uIGlzIHRoYXQKPiA+IGJvdGgKPiA+ID4gPiA+ ID4gTVZDQwo+ID4gPiA+ID4gPiA+IGFuZCBSTFUgcHJvdmlkZSBtdWNoIHN0cm9uZ2VyIGNvbnNp c3RlbmN5IGd1YXJhbnRlZXMgdG8gcmVhZGVycwo+ID4gdGhhbgo+ID4gPiA+IGRvZXMKPiA+ID4g PiA+ID4gPiBSQ1UgLi4uKSAoaHR0cHM6Ly9sd24ubmV0L0FydGljbGVzLzc3NzAzNi8pLgo+ID4g PiA+ID4gPgo+ID4gPiA+ID4gPiBUaGF0IHNvbWVvbmUgd2FzIGluIGZhY3QgbWUuICA7LSkKPiA+ ID4gPiA+ID4KPiA+ID4gPiA+ID4gPiBJIGFtIG5vdCBzdXJlIGlmIHRoZSBhYm92ZSBzdGF0ZW1l bnQgaXMgY29ycmVjdCBvciBub3QuIEJ1dCBpbgo+ID4gPiA+IGdlbmVyYWwsCj4gPiA+ID4gPiA+ ID4gSG93IGNhbiB3ZSBjb21wYXJlIFJDVSBjb25zaXN0ZW5jeSBndWFyYW50ZWVzIHRvIG90aGVy IHRlY2huaXF1ZXMKPiA+ID4gPiAoc3VjaAo+ID4gPiA+ID4gPiBhcwo+ID4gPiA+ID4gPiA+IFJM VSk/Cj4gPiA+ID4gPiA+ID4gSG93IHRvIHJlYXNvbiBhYm91dCB3aGljaCBvbmUgaGFzIHN0cm9u Z2VyIG9yIHdlYWtlciBndWFyYW50ZWVzPwo+ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiBJIHN1Z2dl c3Qgc3RhcnRpbmcgZnJvbSB0aGUgdXNlIGNhc2UuICBGb3IgY29uY3JldGVuZXNzLCBsZXQncwo+ ID4gYXNzdW1lCj4gPiA+ID4gPiA+IHRoYXQgd2UgYXJlIHVzaW5nIGEgaGFzaCB0YWJsZS4gIEF0 IG9uZSBleHRyZW1lLCBpbWFnaW5lIGEgdXNlCj4gPiBjYXNlIGluCj4gPiA+ID4gPiA+IHdoaWNo IGVhY2ggZXZlbnQgbWFrZXMgZXhhY3RseSBvbmUgaGFzaC10YWJsZSBvcGVyYXRpb24uICBObwo+ ID4gPiA+IGluZm9ybWF0aW9uCj4gPiA+ID4gPiA+IGlzIGNhcnJpZWQgZnJvbSBvbmUgZXZlbnQg dG8gdGhlIG5leHQuICAoVGhpcyBtaWdodCB3ZWxsIGJlIHRoZQo+ID4gY2FzZQo+ID4gPiA+ID4g PiBmb3Igc2ltcGxlIHdlYiBzZXJ2ZXIuKSAgU3VjaCBhIHVzZSBjYXNlIGNhbm5vdCB0ZWxsIHRo ZSBkaWZmZXJlbmNlCj4gPiA+ID4gPiA+IGJldHdlZW4gUkNVIG9uIHRoZSBvbmUgaGFuZCBhbmQg TVZDQy9STFUgb24gdGhlIG90aGVyLgo+ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiBBdCB0aGUgb3Ro ZXIgZXh0cmVtZSwgc3VwcG9zZSB0aGF0IGVhY2ggZXZlbnQgZWl0aGVyIGFjY2Vzc2VzIG9yCj4g PiA+ID4gdXBkYXRlcwo+ID4gPiA+ID4gPiBtdWx0aXBsZSBlbnRyaWVzIGluIHRoZSBoYXNoIHRh YmxlLiAgSW4gdGhpcyBjYXNlLCBNVkNDL1JMVSB3aWxsCj4gPiBydWxlCj4gPiA+ID4gPiA+IG91 dCBvdXRjb21lcyB0aGF0IFJDVSB3b3VsZCBwZXJtaXQuICBGb3IgZXhhbXBsZSwgc3VwcG9zZSB3 ZSBoYWQKPiA+IGZvdXIKPiA+ID4gPiA+ID4gZXZlbnRzIGFjY2Vzc2luZyB0d28gZGlmZmVyZW50 IGVsZW1lbnRzIGluIGRpZmZlcmVudCBidWNrZXRzIG9mIHRoZQo+ID4gPiA+ID4gPiBoYXNoIHRh YmxlOgo+ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiAgICAgICAgIEUxOiBBZGRzIDMyIHRvIHRoZSBo YXNoIHRhYmxlLgo+ID4gPiA+ID4gPiAgICAgICAgIEUyOiBBZGRzIDE3MjkgdG8gdGhlIGhhc2gg dGFibGUuCj4gPiA+ID4gPiA+ICAgICAgICAgRTM6IFdpdGhpbiBhIHJlYWQtc2lkZSBjcml0aWNh bCBzZWN0aW9uLCBsb29rcyB1cCAzMiB0aGVuCj4gPiAxNzI5Lgo+ID4gPiA+ID4gPiAgICAgICAg IEU0OiBXaXRoaW4gYSByZWFkLXNpZGUgY3JpdGljYWwgc2VjdGlvbiwgbG9va3MgdXAgMTcyOQo+ ID4gdGhlbiAzMi4KPiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4gR2l2ZW4gZWl0aGVyIE1WQ0Mgb3Ig UkxVLCBpdCB3aWxsIG5vdCBiZSBwb3NzaWJsZSBmb3IgRTMgdG8gZmluZAo+ID4gMzIgYnV0Cj4g PiA+ID4gPiA+IG5vdCAxNzI5IGFuZCBhdCB0aGUgc2FtZSB0aW1lIGZvciBFNCB0byBmaW5kIDE3 MjkgYnV0IG5vdCAzMi4KPiA+IEdpdmVuCj4gPiA+ID4gUkNVLAo+ID4gPiA+ID4gPiB0aGlzIG91 dGNvbWUgaXMgcG9zc2libGUuCj4gPiA+ID4gPiA+Cj4gPiA+ID4gPiBXaGVuIHlvdSBzYXkgIldp dGhpbiBhIHJlYWQtc2lkZSBzZWN0aW9uIiwgZG8geW91IG1lYW4gd2l0aGluIGEKPiA+IHNpbmds ZQo+ID4gPiA+IHNhbWUKPiA+ID4gPiA+IHJlYWQgc2VjdGlvbj8gc3VjaCBhcwo+ID4gPiA+ID4K PiA+ID4gPiA+ID4gcmVhZF9sb2NrKCkKPiA+ID4gPiA+ID4gbG9va3VwKDMyKQo+ID4gPiA+ID4g PiBsb29rdXAoMTcyOSkKPiA+ID4gPiA+ID4gcmVhZF91bmxvY2soKQo+ID4gPiA+ID4KPiA+ID4g PiA+Cj4gPiA+ID4gPiBIb3cgYWJvdXQgcHV0dGluZyB0d28gbG9va3VwcyBpbnRvIHR3byByZWFk LXNpZGUgc2VjdGlvbnM/IERvIHdlCj4gPiBzdGlsbAo+ID4gPiA+IGhhdmUKPiA+ID4gPiA+IHRo ZSBwcm9ibGVtLCB0aGVuPwo+ID4gPiA+ID4KPiA+ID4gPiA+ID4gcmVhZF9sb2NrKCkKPiA+ID4g PiA+ID4gbG9va3VwKDMyKQo+ID4gPiA+ID4gPiByZWFkX3VubG9jaygpCj4gPiA+ID4gPiA+IHJl YWRfbG9jaygpCj4gPiA+ID4gPiA+IGxvb2t1cCgxNzI5KQo+ID4gPiA+ID4gPiByZWFkX3VubG9j aygpCj4gPiA+ID4KPiA+ID4gPiBXaXRob3V0IGluIGFueSB3YXkgYWdyZWVpbmcgd2l0aCB5b3Vy IGNoYXJhY3Rlcml6YXRpb24gb2YgdGhpcyBhcyBhCj4gPiA+ID4gcHJvYmxlbSwgYmVjYXVzZSBy Y3VfcmVhZF9sb2NrKCkgYW5kIHJjdV9yZWFkX3VubG9jaygpIHByb3ZpZGUKPiA+ID4gPiBhYnNv bHV0ZWx5IG5vIG9yZGVyaW5nIGd1YXJhbnRlZXMgaW4gdGhlIGFic2VuY2Ugb2YgYSBncmFjZSBw ZXJpb2QsCj4gPiA+ID4gYW55IG5vbi1ncmFjZS1wZXJpb2QtcmVsYXRlZCByZW9yZGVyaW5nIHRo YXQgY2FuIGhhcHBlbiB3aXRoIGEgc2luZ2xlCj4gPiA+ID4gUkNVIHJlYWQtc2lkZSBjcml0aWNh bCBzZWN0aW9uIGNhbiBhbHNvIGhhcHBlbiB3aGVuIHRoYXQgY3JpdGljYWwKPiA+ID4gPiBzZWN0 aW9uIGlzIHNwbGl0IGluIHR3byBhcyB5b3UgaGF2ZSBkb25lIGFib3ZlLgo+ID4gPiA+Cj4gPiA+ ID4gPiBDb3VsZCB5b3Uga2luZGx5IGdpdmUgbWUgbW9yZSBjbHVlcyB3aHkgUkNVIGNhbiBzZWUg c3VjaCByZW9yZGVyLAo+ID4gd2hpbGUKPiA+ID4gPiBSTFUKPiA+ID4gPiA+IGNhbiBwcmV2ZW50 IGl0Pwo+ID4gPiA+Cj4gPiA+ID4gSGVyZSBhcmUgbWluaW1hbCBDLWxhbmd1YWdlIGltcGxlbWVu dGF0aW9ucyBmb3IgUkNVIHRoYXQgY2FuIChhbmQgYXJlKQo+ID4gPiA+IGFjdHVhbGx5IHVzZWQ6 Cj4gPiA+ID4KPiA+ID4gR3JlYXQuIFdlIHVzZSB0aGUgc2FtZSB0aGluZyBpbiBvdXIgcmVhbC10 aW1lIHdvcmsgWzFdCj4gPgo+ID4gSXQgaGFzIGJlZW4gYSBwb3B1bGFyIGNob2ljZSBmb3IgNDAg eWVhcnMgbm93LiAgOy0pCj4gPgo+ID4gPiA+ICNkZWZpbmUgcmN1X3JlYWRfbG9jaygpCj4gPiA+ ID4gI2RlZmluZSByY3VfcmVhZF91bmxvY2soKQo+ID4gPiA+Cj4gPiA+ID4gUGxlYXNlIGNvbXBh cmUgdGhlc2UgdG8gdGhlIHJlYWQtc2lkZSBtYXJrZXJzIHByZXNlbnRlZCBpbiB0aGUgUkxVCj4g PiBwYXBlciwKPiA+ID4gPiBhbmQgdGhlbiB0ZWxsIG1lIHlvdXIgdGhvdWdodHMgb24gdGhlIGFu c3dlciB0byB5b3VyIHF1ZXN0aW9uLiAgOy0pCj4gPiA+ID4KPiA+ID4gSSBzdWJtaXQgbXkgaG9t ZXdvcmsgaGVyZSwgYnV0IEkgZG8gbm90IHRoaW5rIEkgZGlkIGl0IHdlbGwuCj4gPiA+IDEuIEkg YmVsaWV2ZSBpbiB0aGUgZGVmYXVsdCBVUkNVIGltcGxlbWVudGF0aW9uLCBpdCBoYXMgbWVtb3J5 IGJhcnJpZXIKPiA+ID4gaW5zaWRlIHRoZSByZWFkX2xvY2sgLyByZWFkX3VubG9jay4KPiA+Cj4g PiBJdCBjZXJ0YWlubHkgd2FzIGF0IG9uZSB0aW1lLiAgQnV0IHlvdSB3b3VsZCBvZiBjb3Vyc2Ug bmVlZCB0byBjaGVjawo+ID4gdG8gc2VlIHdoYXQgYW55IGdpdmVuIHdvcmtlciBhY3R1YWxseSB1 c2VkLgo+ID4KPiA+ID4gMi4gRnJvbSB0aGUgUkxVIHBhcGVyLCBpdCBzaG93cyB0aGUgY29kZSBm b3IgcmVhZC1zaWRlIG9wZXJhdGlvbi4KPiA+ID4gUkxVX1JFQURFUl9MT0NLKGN0eCkKPiA+ID4g ICAgIGN0eC5pcy13cml0ZXLihpBmYWxzZQo+ID4gPiAgICAgY3R4LnJ1bi1jbnTihpBjdHgucnVu LWNudCsxLgo+ID4gPiAgICAgbWVtb3J5IGZlbmNlCj4gPiA+ICAgIGN0eC5sb2NhbC1jbG9ja+KG kGdsb2JhbC1jbG9jawo+ID4gPiBSTFVfUkVBREVSX1VOTE9DSyhjdHgpCj4gPiA+ICAgICBjdHgu cnVuLWNudOKGkGN0eC5ydW4tY250KzEKPiA+ID4gICAgIGlmIChjdHguaXMtd3JpdGVyKQo+ID4g PiAgICAgICAgIFJMVV9DT01NSVRfV1JJVEVfTE9HKGN0eCkKPiA+ID4KPiA+ID4gV2UgY2FuIGln bm9yZSB0aGUgd3JpdGVyIGNoZWNrLCBhcyBpbiBvdXIgY2FzZSwgdGhlIHJlYWRlciBuZXZlciBk bwo+ID4gdXBkYXRlLgo+ID4gPiBNeSB1bmRlcnN0YW5kaW5nIGlzIHRoYXQgcmVhZC1zaWRlIG9w ZXJhdGlvbnMgYXJlIG9ubHkgdXNlZCB0byBmYWNpbGl0YXRlCj4gPiA+IHRoZSBxdWllc2NlbmNl IGRldGVjdGlvbi4KPiA+ID4gdGhlIHJ1biBjbnQgaXMgdXNlZCB0byBkZWNpZGUgaWYgYSB0aHJl YWQgaXMgYWN0aXZlIChpZiBpdCBpcyBjdXJyZW50bHkKPiA+ID4gaW5zaWRlIGEgcmVhZC1zaWRl IHNlY3Rpb24pLgo+ID4gPiB0aGUgbG9jYWwgY2xvY2sgaXMgdXNlZCB0byBjaGVjayBpZiB0aGUg YWN0aXZlIHRocmVhZCBnb2VzIGludG8gdGhlCj4gPiA+IHJlYWQtc2lkZSBzZWN0aW9uIHZlcnkg bGF0ZSwgc28gaXQgZG9lcyBub3QgcHJldmVudCByZWNsYWltaW5nIG1lbW9yeQo+ID4gPiB1bmxp bmtlZCBiZWZvcmUgaXQgZW50ZXJzIHRoZSByZWFkIHNlY3Rpb24uCj4gPiA+IEhvd2V2ZXIgaSBz ZWUgbm8gZGlmZmVyZW5jZSBiZXR3ZWVuIFJMVSBhbmQgUkNVIHJlYWQtc2lkZSBvcGVyYXRpb24K PiA+ID4gcmVnYXJkaW5nIGNvbnNpc3RlbmN5IGd1YXJhbnRlZS4KPiA+ID4gQ291bGQgeW91IGtp bmRseSB0ZWFjaCBtZSBtb3JlPwo+ID4KPiA+IEluIG9yZGVyIHRvIHRlYWNoIHlvdSBtb3JlLCBJ IGRvIG5lZWQgc29tZSBoZWxwIGZyb20geW91LiAgSSBoYXZlIHBvc3RlZAo+ID4gYSBudW1iZXIg b2YgVVJMcyBlYXJsaWVyIGluIHRoaXMgZW1haWwgdGhyZWFkLiAgQ291bGQgeW91IHBsZWFzZSB0 ZWxsCj4gPiBtZSB3aGF0IHlvdSBsZWFybmVkIGZyb20gZWFjaCBvZiB0aGVtPwo+ID4KPiBTdXJl LCBJIHdpbGwgZGVmaW5pdGVseSBsZWFybiB0aG9zZSByZXNvdXJjZXMgeW91IHByb3ZpZGVkLCBh bmQgY29tZSBiYWNrCj4gdG8geW91IGlmIEkgaGF2ZSBtb3JlIHF1ZXN0aW9ucy4KPiBTaW1wbHkg c3BlYWtpbmcsIEkgZmFpbGVkIHRvIGZpbmQgYSBjbGVhciBhbmQgZm9ybWFsIGNvbnNpc3RlbmN5 Cj4gbW9kZWwvZ3VhcmFudGVlIG9mIFJDVSBxdWlja2x5Lgo+IFRoZSBkaWZmaWN1bHR5IGZvciBt ZSBpcywgZm9yIG1vc3QgcGVvcGxlLCBhdCB0aGVpciBmaXJzdCBnbGFuY2UsIFJDVSBvbmx5Cj4g aGFzIHJlbGF4L3dlYWsgY29uc2lzdGVuY3kgZ3VhcmFudGVlLgo+IFNvIHdoZW5ldmVyIEkgc3Vn Z2VzdCB0byB1c2UgUkNVIHRvIG90aGVycywgdGhleSBhbHdheXMgYXNrIGxpa2UKPiBEb2VzIFJD VSBwcm92aWRlIFNlcmlhbGl6YWJpbGl0eT8gLi4gbWF5YmUgbm90Lgo+IERvZXMgUkNVIHByb3Zp ZGUgTGluZWFyaXphYmlsaXR5PyAuLi4gbWF5YmUgbm90Cj4gLi4uCgpBcmUgc2VyaWFsaXphYmls aXR5IGFuZCBsaW5lYXJpemFiaWxpdHkgYWx3YXlzIHVzZWZ1bD8gIERlZmluaXRlbHkgbm90LgpB cmUgc2VyaWFsaXphYmlsaXR5IGFuZCBsaW5lYXJpemFiaWxpdHkgZXhwZW5zaXZlPyAgWW91IGJl dCEhIQoKQnV0IGlmIHlvdXIgdXNlIGNhc2UgYWJzb2x1dGVseSByZXF1aXJlcyBlaXRoZXIgc2Vy aWFsaXphYmlsaXR5IG9yCmxpbmVhcml6YWJpbGl0eSwgeW91IHByb2JhYmx5IHNob3VsZCB0cnkg c29tZXRoaW5nIG90aGVyIHRoYW4gUkNVLgpCdXQgeW91IHNob3VsZCBhbHNvIHN0cm9uZ2x5IHF1 ZXN0aW9uIHNlcmlhbGl6YWJpbGl0eSBhbmQgbGluZWFyaWFiaWxpdHkKcmVxdWlyZW1lbnRzLCBn aXZlbiB0aGF0IHRoZXkgYXJlIG9mdGVuIGltcG9zZWQgb3V0IG9mIGZlYXIgYW5kIGZvcmNlCm9m IGhhYml0IHJhdGhlciB0aGFuIHJlYWwgbmVlZC4KCk5ldmVydGhlbGVzcywgdGhlcmUgYXJlIHVz ZSBjYXNlcyB3aGVyZSBSQ1UgaXMgcGFydCBvZiBhIGNvbmN1cnJlbmN5CmRlc2lnbiB0aGF0IHBy b3ZpZGVzIHNlcmlhbGl6YWJpbGl0eSBhbmQvb3IgbGluZWFyaXphYmlsaXR5LiAgRm9yIGJ1dApv bmUgZXhhbXBsZToKCkBDb25mZXJlbmNle0FyY2FuZ2VsaTAzLAogQXV0aG9yPSJBbmRyZWEgQXJj YW5nZWxpIGFuZCBNaW5nbWluZyBDYW8gYW5kIFBhdWwgRS4gTWNLZW5uZXkgYW5kCkRpcGFua2Fy IFNhcm1hIiwKIFRpdGxlPSJVc2luZyBSZWFkLUNvcHkgVXBkYXRlIFRlY2huaXF1ZXMgZm9yIHtT eXN0ZW0gViBJUEN9IGluIHRoZQp7TGludXh9IDIuNSBLZXJuZWwiLAogQm9va3RpdGxlPSJQcm9j ZWVkaW5ncyBvZiB0aGUgMjAwMyBVU0VOSVggQW5udWFsIFRlY2huaWNhbCBDb25mZXJlbmNlCihG UkVFTklYIFRyYWNrKSIsCiBQdWJsaXNoZXI9IlVTRU5JWCBBc3NvY2lhdGlvbiIsCiBsb2NhdGlv bj0iU2FuIEFudG9uaW8sIFRleGFzLCBVU0EiLAogeWVhcj0iMjAwMyIsCiBtb250aD0iSnVuZSIs CiBwYWdlcz0iMjk3LTMxMCIsCiB1cmw9Imh0dHBzOi8vd3d3LnVzZW5peC5vcmcvbGVnYWN5L3B1 YmxpY2F0aW9ucy9saWJyYXJ5L3Byb2NlZWRpbmdzL3VzZW5peDAzL3RlY2gvZnJlZW5peDAzL2Z1 bGxfcGFwZXJzL2FyY2FuZ2VsaS9hcmNhbmdlbGkucGRmIiwKIGxhc3RjaGVja2VkPSJGZWJydWFy eSAyNywgMjAxNyIsCn0KCj4gT0ssIHdoYXQgZG9lcyBSQ1UgcHJvdmlkZT8gIC4uIEkgYW0gbm90 IHN1cmUuIEkgZXZlbiBjYW5ub3QgZmluZCBhIHNpbmdsZQo+IGRvY3VtZW50IHdoaWNoIGhhcyBh biBhY2N1cmF0ZSBhbnN3ZXIuCj4gCj4gTGV0IG1lIGdpdmUgbW9yZSBleGFtcGxlcy4KPiAxLiBX aGVuIEkgcmVjb21tZW5kIHVzaW5nIHJlYWQtd3JpdGUgbG9jaywgcGVvcGxlIGFyZSBoYXBweSB0 byB1c2UgaXQuCj4gICAgIEhvd2V2ZXIsIHdoZW4gSSBmdXJ0aGVyIHN1Z2dlc3QgdG8gcmVwbGFj ZSBydyBsb2NrIHdpdGggUkNVLCB0aGVuIHRoZXkKPiBoZXNpdGF0ZSB0byBkbyBzby4gQmVjYXVz ZSB0aGV5IGZlZWwgUkNVIGlzIHdlYWtlciB0aGFuIHJ3IGxvY2ssIGFuZCBkbyBub3QKPiBrbm93 IHdoYXQgcHJlY2lzZWx5IFJDVSBndWFyYW50ZWUuCj4gMi4gSW4gY29udHJhc3QsIGluIHRoZSBk YXRhYmFzZSBhcmVhLCBpdCBoYXMgcmVsYXRpdmUgY2xlYXIgZGVmaW5pdGlvbiBvZgo+IGNvbnNp c3RlbmN5IGxldmVscyAoc3VjaCBhcyBodHRwczovL2plcHNlbi5pby9jb25zaXN0ZW5jeSksIGFu ZCB1c2VycyBhcmUKPiBlYXN5IHRvIGFzc2VydCBpZiB0aGV5IGNhbiB1c2UgY2VydGFpbiBkYXRh YmFzZSBjb25maWd1cmF0aW9uIGJhc2VkIG9uIGl0cwo+IGNvbnNpc3RlbmN5IG1vZGVsLgoKVGhl IFJDVSBndWFyYW50ZWVzIGFyZSBjbGVhcmx5IHN0YXRlZCBpbiBhIG51bWJlciBvZiBwbGFjZXMu ICBNb3N0CnJlY2VudGx5IGFuZCBtb3N0IGZvcm1hbGx5LCB0aGUgMjAxOCBBU1BMT1MgIkZyaWdo dGVuaW5nIHNtYWxsIGNoaWxkcmVuIgpwYXBlciBjYWxsZWQgb3V0IGJlbG93ICh0aG91Z2ggdGhl IDIwMTIgcGFwZXIgeW91IHJlZmVycmVkIHRvIGhhcyBhCmdvb2QgYW5kIHN1ZmZpY2llbnQgc3Rh dGVtZW50IG9mIGl0cyBndWFyYW50ZWVzKS4gIFRoaXMgZm9ybWFsaXNtIGlzCmV4ZWN1dGFibGUs IGFuZCBoYXMgYmVlbiBhdmFpbGFibGUgd2l0aGluIHRoZSBMaW51eC1rZXJuZWwgc291cmNlIHRy ZWUKaW4gZXhlY3V0YWJsZSBmb3JtIGZvciBtb3JlIHRoYW4gYSB5ZWFyLiAgSSBzdHJvbmdseSBy ZWNvbW1lbmQgdGhhdCB5b3UKdHJ5IHBsYXlpbmcgd2l0aCB0aGlzIGV4ZWN1dGFibGUgbW9kZWwu CgpCdXQgc3VjY2Vzc2Z1bGx5IHVzaW5nIFJDVSByZXF1aXJlcyB0aGF0IHlvdSB0aGluayBhYm91 dCB5b3VyIHByb2JsZW0KZGlmZmVyZW50bHkuICBUaGlzIHNob3VsZCBub3QgYmUgYSBzdXJwcmlz ZS4gIEFmdGVyIGFsbCwgbG9ja2luZwpwcm92aWRlcyBtdXR1YWwgZXhjbHVzaW9uIGFuZCBSQ1Ug ZG9lcyBub3QuICBTbyBvbmUgc2V0IG9mIHByb21pc2luZwp1c2UgY2FzZXMgZm9yIFJDVSBhcmUg d2hlcmUgc29tZSBxdWFudGl0eSBpcyBjYWxjdWxhdGVkIHJlYWQtaG9sZGluZyBhCnJlYWRlci13 cml0ZXIgbG9jaywgYnV0IHdoZXJlIHRoYXQgcXVhbnRpdHkgaXMgdXNlZCBhZnRlciByZWxlYXNp bmcgdGhhdApyZWFkZXItd3JpdGVyIGxvY2suICBUaGlzIGlzIGEgY29tbW9uIHVzZSBjYXNlLCBh bmQgaXQgaXMgYSBjYXNlIHdoZXJlIHRoZQp1c2VyIGhhcyBwdXJwb3NlZnVsbHkgc2V0IGFzaWRl IHRoZSBzZXJpYWxpemFiaWxpdHkgYW5kIGxpbmVhcml6YWJpbGl0eQpndWFyYW50ZWVzIHRoYXQg cmVhZGVyLXdyaXRlciBsb2NraW5nIHByb3ZpZGVzIC0tIGJ1dCBoYXMgbmV2ZXJ0aGVsZXNzCnBh aWQgdGhlIGZ1bGwgcHJpY2UgZm9yIHRoZXNlIGd1YXJhbnRlZXMuCgo+IEFnYWluLCBJIHJlYWxs eSBhcHByZWNpYXRlIHlvdXIgcGF0aWVuY2UgYW5kIGNvbW11bmljYXRpb24sIGhvcGUgdGhvc2Ug ZG9lcwo+IG5vdCBib3RoZXIgeW91IGEgbG90LgoKQXQgdGhpcyBwb2ludCwgSSBtdXN0IGNvbmZl c3MgdG8gYmVpbmcgbXVjaCBtb3JlIGFtdXNlZCB0aGFuIGJvdGhlcmVkLgoKRm9yIGV4YW1wbGUs IGRvIEkgdW5kZXJzdGFuZCBjb3JyZWN0bHkgdGhhdCB5b3UgaGF2ZW4ndCB5ZXQgcmVhZCAtYW55 LQpvZiB0aGUgcmVmZXJlbmNlcyBJIHNlbnQgeW91PyAgSXQgc3VyZSBzZWVtcyB0aGF0IHdheS4g IDstKQoKQWxzbywgYXJlIHlvdSBnZXR0aW5nIGFkZXF1YXRlIHNsZWVwLCBhcyBpbiBhdCBsZWFz dCBzZXZlbiBob3VycwpwZXIgbmlnaHQ/ICBBZnRlciBhbGwsIGF0dGVtcHRpbmcgdG8gbGVhcm4g UkNVIHdoaWxlIHNsZWVwLWRlcHJpdmVkCnF1aXRlIGZvb2xoYXJkeS4KCgkJCQkJCQlUaGFueCwg UGF1bAoKPiBUaGFua3MKPiBCZXN0LAo+IFl1eGluCj4gCj4gPgo+ID4gICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFRoYW54LCBQ YXVsCj4gPgo+ID4gPiBCVFcsIHRoZSBSTFUgcmVhZC1zaWRlIG9wcyBhcmUgc2ltaWxhciwgYnV0 IG5vdCBlZmZpY2llbnQsIGNvbXBhcmluZyB0bwo+ID4gb3VyCj4gPiA+IFBhcnNlYyB3b3JrIFsx LCAyXQo+ID4gPiBbMV0gWXV4aW4gUmVuLCBHLiBMaXUsIEcuIFBhcm1lciwgQi4gQnJhbmRlbmJ1 cmcsIFNjYWxhYmxlIE1lbW9yeQo+ID4gPiBSZWNsYW1hdGlvbiBmb3IgTXVsdGktQ29yZSwgUmVh bC1UaW1lIFN5c3RlbXMsIGluIFByb2NlZWRpbmdzIG9mIHRoZSAyNHRoCj4gPiA+IElFRUUgUmVh bC1UaW1lIGFuZCBFbWJlZGRlZCBUZWNobm9sb2d5IGFuZCBBcHBsaWNhdGlvbnMgU3ltcG9zaXVt IChSVEFTKSwKPiA+ID4gMjAxOAo+ID4gPiBbMl0gUS4gV2FuZywgVC4gU3RhbWxlciwgYW5kIEcu IFBhcm1lciwgUGFyYWxsZWwgU2VjdGlvbnM6IFNjYWxpbmcKPiA+ID4gU3lzdGVtLUxldmVsIERh dGEtU3RydWN0dXJlcywgaW4gUHJvY2VlZGluZ3Mgb2YgRXVyb3BlYW4gQ29uZmVyZW5jZSBvbgo+ ID4gPiBDb21wdXRlciBTeXN0ZW1zIChFdXJvU3lzKSwgMjAxNgo+ID4gPgo+ID4gPiBUaGFua3Ms Cj4gPiA+IEJlc3QsCj4gPiA+IFl1eGluCj4gPiA+Cj4gPiA+Cj4gPiA+ID4gPiBUaGlzIGlzIGJl Y2F1c2UgTVZDQyBhbmQgUkxVIHByb3ZpZGUgcmVhZGVycyBhIGNvbnNpc3RlbnQgdmlldyBvZgo+ ID4gPiA+ID4gPiB0aGUgdXBkYXRlcywgYW5kIFJDVSBkb2VzIG5vdC4gIE9mIGNvdXJzZSwgaXQg aXMgb2Z0ZW4gdGhlIGNhc2UKPiA+IHRoYXQgYQo+ID4gPiA+ID4gPiBjb25zaXN0ZW50IHZpZXcg aXMgbm90IG5lZWRlZCwgaW4gd2hpY2ggY2FzZSB0aGUgTVZDQyBhbmQgUkxVCj4gPiA+ID4gZ3Vh cmFudGVlcwo+ID4gPiA+ID4gPiBhcmUgaW5jdXJyaW5nIHJlYWQtc2lkZSBvdmVyaGVhZCBmb3Ig bm8gcmVhc29uLiAgQnV0IGlmIHRoZSB1c2UKPiA+IGNhc2UKPiA+ID4gPiA+ID4gcmVxdWlyZXMg Y29uc2lzdGVudCByZWFkZXJzLCBSQ1UgaXMgbm90IGFuIG9wdGlvbi4KPiA+ID4gPiA+ID4KPiA+ ID4gPiA+ID4gVGhlIHJlYXNvbiBhIGNvbnNpc3RlbnQgdmlldyBpcyBub3QgYWx3YXlzIG5lZWRl ZCBpcyB0aGF0Cj4gPiA+ID4gc3BlZWQtb2YtbGlnaHQKPiA+ID4gPiA+ID4gZGVsYXlzIG1ha2Ug aXQgaW1wb3NzaWJsZSB0byBwcm92aWRlIGEgY29uc2lzdGVudCB2aWV3IG9mIHRoZQo+ID4gb3V0 c2lkZQo+ID4gPiA+ID4gPiB3b3JsZC4gIEluIHRoZSBjb21tb24gY2FzZSB3aGVyZSB0aGUgdXNl IGNhc2UgaW50ZXJhY3RzIHdpdGggdGhlCj4gPiA+ID4gPiA+IG91dHNpZGUgd29ybGQsIHRoZSBh bGdvcml0aG1zIGFic29sdXRlbHkgbXVzdCBiZSBkZXNpZ25lZCB0bwo+ID4gdG9sZXJhdGUKPiA+ ID4gPiA+ID4gaW5jb25zaXN0ZW5jeSwgd2hpY2ggb3BlbnMgdGhlIGRvb3IgdG8gdGhpbmdzIGxp a2UgUkNVLgo+ID4gPiA+ID4KPiA+ID4gPiA+IEkgYW0gY29uZnVzZWQgaGVyZS4gSSB0aGluayBz cGVlZC1vZi1saWdodCBkZWxheXMgaGFwcGVuIGV2ZXJ5d2hlcmUsCj4gPiBub3QKPiA+ID4gPiA+ IG9ubHkgYm91bmQgdG8gUkNVLCBidXQgYWxzbyAgYW55IG90aGVyIHN5bmNocm9uaXphdGlvbiBh cHByb2FjaCAoc3VjaAo+ID4gPiA+IFJMVSkuCj4gPiA+ID4gPiBJZiBzbywgaG93IGRvIG90aGVy cyAoUkxVKSBwcm92aWRlIGNvbnNpc3RlbnQgdmlld3M/Cj4gPiA+ID4KPiA+ID4gPiBZb3UganVz dCBzdGF0ZWQgdGhlIGFuc3dlci4gIE5vdyBpdCBpcyBvbmx5IG5lY2Vzc2FyeSBmb3IgeW91IHRv IGludmVzdAo+ID4gPiA+IHRoZSB0aW1lLCBlZmZvcnQsIGFuZCB0aG91Z2h0IHRvIGZ1bGx5IHVu ZGVyc3RhbmQgaXQuICBUbyBoZWxwIHdpdGgKPiA+IHRoaXMsCj4gPiA+ID4gdGhlIGZvbGxvd2lu ZyBwYXJhZ3JhcGggcHJvdmlkZXMgYW5vdGhlciBoaW50Ogo+ID4gPiA+Cj4gPiA+ID4gICAgICAg ICBZZXMsIHlvdSBhcmUgcXVpdGUgcmlnaHQsIHNwZWVkLW9mLWxpZ2h0IGRlbGF5cyBiZXR3ZWVu IHRoZQo+ID4gPiA+ICAgICAgICAgb3V0c2lkZSB3b3JsZCBhbmQgdGhlIGNvbXB1dGVyIGFmZmVj dCBSTFUganVzdCBhcyBzdXJlbHkgYXMKPiA+IHRoZXkKPiA+ID4gPiAgICAgICAgIGRvIFJDVS4g IFRoaXMgbWVhbnMgdGhhdCB0aGUgYWRkaXRpb25hbCBjb25zaXN0ZW5jeSBndWFyYW50ZWVzCj4g PiA+ID4gICAgICAgICBwcm92aWRlZCBieSBSTFUgbXVzdCBuZWNlc3NhcmlseSBiZSBsaW1pdGVk IHRvIHRoZSBjb25maW5lcyBvZgo+ID4gdGhlCj4gPiA+ID4gICAgICAgICBjb21wdXRlciBzeXN0 ZW0gaW4gcXVlc3Rpb24uICBUYWtpbmcgdGhpcyBvbmUgc3RlcCBmdXJ0aGVyLAo+ID4gdGhlcmUK PiA+ID4gPiAgICAgICAgIGFyZSBhbHNvIHNwZWVkLW9mLWxpZ2h0IGRlbGF5cyB3aXRoaW4gdGhl IGNvbXB1dGVyLiAgVGhlcmVmb3JlLAo+ID4gPiA+ICAgICAgICAgaW4gdGhlIGdlbmVyYWwgY2Fz ZSwgUkxVIGNhbiBwcm92aWRlIGl0cyBjb25zaXN0ZW5jeQo+ID4gZ3VhcmFudGVlcywKPiA+ID4g PiAgICAgICAgIGV2ZW4gd2l0aGluIHRoZSBjb25maW5lcyBvZiB0aGUgY29tcHV0ZXIgc3lzdGVt LCBvbmx5IGF0IHRoZQo+ID4gPiA+ICAgICAgICAgZXhwZW5zZSBvZiBpbmN1cnJpbmcgZGVsYXlz LiAgQmVjYXVzZSBSQ1UgZG9lcyBub3QgcHJvdmlkZQo+ID4gUkxVJ3MKPiA+ID4gPiAgICAgICAg IGNvbnNpc3RlbmN5IGd1YXJhbnRlZXMsIGl0IG5lZWQgbm90IGluY3VyIFJMVSdzIGRlbGF5cy4K PiA+ID4gPgo+ID4gPiA+IFRoaXMgaXMgbm90IGEgbmV3IGxpbmUgb2YgcmVhc29uaW5nLCBzZWUg Zm9yIGV4YW1wbGU6Cj4gPiA+ID4KPiA+ID4gPiBAYXJ0aWNsZXtIZXJsaWh5OjE5OTY6TENOOjEw NjMzNjkuMTA2MzM3Mgo+ID4gPiA+ICxhdXRob3IgPSB7SGVybGloeSwgTWF1cmljZSBhbmQgU2hh dml0LCBOaXIgYW5kIFdhYXJ0cywgT3JsaX0KPiA+ID4gPiAsdGl0bGUgPSB7TGluZWFyaXphYmxl IGNvdW50aW5nIG5ldHdvcmtzfQo+ID4gPiA+ICxqb3VybmFsID0ge0Rpc3RyaWIuIENvbXB1dC59 Cj4gPiA+ID4gLHZvbHVtZSA9IHs5fQo+ID4gPiA+ICxpc3N1ZSA9IHs0fQo+ID4gPiA+ICxtb250 aCA9IHtGZWJydWFyeX0KPiA+ID4gPiAseWVhciA9IHsxOTk2fQo+ID4gPiA+ICxpc3NuID0gezAx NzgtMjc3MH0KPiA+ID4gPiAscGFnZXMgPSB7MTkzLS0yMDN9Cj4gPiA+ID4gLG51bXBhZ2VzID0g ezExfQo+ID4gPiA+ICx1cmwgPSB7aHR0cDovL3BvcnRhbC5hY20ub3JnL2NpdGF0aW9uLmNmbT9p ZD0xMDYzMzY5LjEwNjMzNzJ9Cj4gPiA+ID4gLGRvaSA9IHsxMC4xMDA3L3MwMDQ0NjAwNTAwMTl9 Cj4gPiA+ID4gLGFjbWlkID0gezEwNjMzNzJ9Cj4gPiA+ID4gLHB1Ymxpc2hlciA9IHtTcHJpbmdl ci1WZXJsYWd9Cj4gPiA+ID4gLGFkZHJlc3MgPSB7TG9uZG9uLCBVS30KPiA+ID4gPiAsa2V5d29y ZHMgPSB7Y29uY3VycmVuY3ksIGNvbnRlbnRpb24sIGNvdW50aW5nIG5ldHdvcmtzLCBkYXRhCj4g PiBzdHJ1Y3R1cmVzLAo+ID4gPiA+IGxpbmVhcml6YWJpbGl0eX0KPiA+ID4gPiB9Cj4gPiA+ID4K PiA+ID4gPiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIFRoYW54LCBQYXVsCj4gPiA+ID4KPiA+ID4gPiA+IFRoYW5rcyBmb3IgeW91ciBlZHVj YXRpb24uCj4gPiA+ID4gPiBZdXhpbgo+ID4gPiA+ID4KPiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4g ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBU aGFueCwgUGF1bAo+ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiA+IFRoYW5rcwo+ID4gPiA+ID4gPiA+ IFl1eGluCj4gPiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4gPiBPbiBGcmksIERlYyA2LCAyMDE5IGF0 IDExOjMwIEFNIFBhdWwgRS4gTWNLZW5uZXkgPAo+ID4gcGF1bG1ja0BrZXJuZWwub3JnCj4gPiA+ ID4gPgo+ID4gPiA+ID4gPiB3cm90ZToKPiA+ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiA+ID4gT24g RnJpLCBEZWMgMDYsIDIwMTkgYXQgMTA6NTk6MDVBTSAtMDUwMCwgTWF0aGlldSBEZXNub3llcnMK PiA+IHdyb3RlOgo+ID4gPiA+ID4gPiA+ID4gPiAtLS0tLSBPbiBEZWMgNiwgMjAxOSwgYXQgMzo1 MSBQTSwgWXV4aW4gUmVuIDwKPiA+IHJ5eEBnd21haWwuZ3d1LmVkdT4KPiA+ID4gPiA+ID4gd3Jv dGU6Cj4gPiA+ID4gPiA+ID4gPiA+Cj4gPiA+ID4gPiA+ID4gPiA+ID4gT24gRnJpLCBEZWMgNiwg MjAxOSBhdCA1OjQ5IEFNIE1hdGhpZXUgRGVzbm95ZXJzIDwgWwo+ID4gPiA+ID4gPiA+ID4gPiA+ IG1haWx0bzptYXRoaWV1LmRlc25veWVyc0BlZmZpY2lvcy5jb20gfAo+ID4gPiA+ID4gPiBtYXRo aWV1LmRlc25veWVyc0BlZmZpY2lvcy5jb20KPiA+ID4gPiA+ID4gPiA+IF0gPgo+ID4gPiA+ID4g PiA+ID4gPiA+IHdyb3RlOgo+ID4gPiA+ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiA+ID4gPiA+PiAt LS0tLSBPbiBEZWMgNSwgMjAxOSwgYXQgODoxNyBQTSwgWXV4aW4gUmVuIDwgWyBtYWlsdG86Cj4g PiA+ID4gPiA+ID4gPiByeXhAZ3dtYWlsLmd3dS5lZHUgfAo+ID4gPiA+ID4gPiA+ID4gPiA+PiBy eXhAZ3dtYWlsLmd3dS5lZHUgXSA+IHdyb3RlOgo+ID4gPiA+ID4gPiA+ID4gPgo+ID4gPiA+ID4g PiA+ID4gPiA+Pj4gSGksCj4gPiA+ID4gPiA+ID4gPiA+ID4+PiBJIGFtIGEgc3R1ZGVudCwgYW5k IGxlYXJuaW5nIFJDVSBub3csIGJ1dCBzdGlsbCBrbm93IHZlcnkKPiA+ID4gPiBsaXR0bGUKPiA+ ID4gPiA+ID4gPiA+IGFib3V0IGl0Lgo+ID4gPiA+ID4gPiA+ID4gPiA+Pj4gQXJlIHRoZXJlIGFu eSBkb2N1bWVudHMvcGFwZXJzL21hdGVyaWFscyB3aGljaAo+ID4gKGluKWZvcm1hbGx5Cj4gPiA+ ID4gPiA+IGRlZmluZQo+ID4gPiA+ID4gPiA+ID4gYW5kIGV4cGxhaW4KPiA+ID4gPiA+ID4gPiA+ ID4gPj4+IFJDVSBjb25zaXN0ZW5jeSBndWFyYW50ZWVzPwo+ID4gPiA+ID4gPiA+ID4gPgo+ID4g PiA+ID4gPiA+ID4gPiA+PiBZb3UgbWF5IHdhbnQgdG8gaGF2ZSBhIGxvb2sgYXQKPiA+ID4gPiA+ ID4gPiA+ID4KPiA+ID4gPiA+ID4gPiA+ID4gPj4gVXNlci1MZXZlbCBJbXBsZW1lbnRhdGlvbnMg b2YgUmVhZC1Db3B5IFVwZGF0ZQo+ID4gPiA+ID4gPiA+ID4gPiA+PiBBcnRpY2xlIGluIElFRUUg VHJhbnNhY3Rpb25zIG9uIFBhcmFsbGVsIGFuZCBEaXN0cmlidXRlZAo+ID4gPiA+IFN5c3RlbXMK PiA+ID4gPiA+ID4gPiA+IDIzKDIpOjM3NSAtIDM4Mgo+ID4gPiA+ID4gPiA+ID4gPiA+PiDCtyBN YXJjaCAyMDEyCj4gPiA+ID4gPiA+ID4gPiA+Cj4gPiA+ID4gPiA+ID4gPiA+ID4gVGhhbmtzIGZv ciB5b3VyIGluZm8uCj4gPiA+ID4gPiA+ID4gPiA+ID4gSG93ZXZlciwgSSBkbyBub3QgdGhpbmsg VVJDVSB0YWxrcyBhYm91dCBhbnkgY29uc2lzdGVuY3kKPiA+IG1vZGVsCj4gPiA+ID4gPiA+ID4g PiBmb3JtYWxseS4KPiA+ID4gPiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4gPiA+ID4gPiBGcm9tIHBy ZXZpb3VzIGNvbW11bmljYXRpb24gd2l0aCBQYXVsLCBoZSBzYWlkIFJDVSBpcyBub3QKPiA+ID4g PiBkZXNpZ25lZAo+ID4gPiA+ID4gPiBmb3IKPiA+ID4gPiA+ID4gPiA+ID4gPiBsaW5lYXJpemFi aWxpdHksIGFuZCBpdCBpcyB0b3RhbGx5IGFjY2VwdGFibGUgdGhhdCBSQ1UgaXMKPiA+IG5vdAo+ ID4gPiA+ID4gPiA+ID4gbGluZWFyaXphYmxlLgo+ID4gPiA+ID4gPiA+ID4gPiA+IEhvd2V2ZXIs IEkgYW0gY3VyaW91cyBob3cgdG8gYWNjdXJhdGVseS9mb3JtYWxseQo+ID4gQ2hhcmFjdGVyaXpl Cj4gPiA+ID4gUkNVCj4gPiA+ID4gPiA+ID4gPiBjb25zaXN0ZW5jeQo+ID4gPiA+ID4gPiA+ID4g PiA+IG1vZGVsL2d1YXJhbnRlZXMKPiA+ID4gPiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4gPiA+ID4g QWRkaW5nIFBhdWwgRS4gTWNLZW5uZXkgaW4gQ0MuCj4gPiA+ID4gPiA+ID4gPiA+Cj4gPiA+ID4g PiA+ID4gPiA+IEkgYW0gcmVmZXJyaW5nIHRvIHRoZSBzZWN0aW9uICJPdmVydmlldyBvZiBSQ1Ug c2VtYW50aWNzIiBpbgo+ID4gdGhlCj4gPiA+ID4gPiA+IHBhcGVyLgo+ID4gPiA+ID4gPiA+ID4g Tm90IHN1cmUgaXQgaGFzIHRoZSBsZXZlbCBvZgo+ID4gPiA+ID4gPiA+ID4gPiBmb3JtYWxpdHkg eW91IGFyZSBsb29raW5nIGZvciB0aG91Z2guIFBhdWwsIGRvIHlvdSBoYXZlCj4gPiBwb2ludGVy cwo+ID4gPiA+IHRvCj4gPiA+ID4gPiA+ID4gPiBhZGRpdGlvbmFsIG1hdGVyaWFsID8KPiA+ID4g PiA+ID4gPiA+Cj4gPiA+ID4gPiA+ID4gPiBJbmRlZWQgSSBkbyEgIFRoZSBMaW51eCBrZXJuZWwg bWVtb3J5IG1vZGVsIChMS01NKSBpbmNsdWRlcwo+ID4gUkNVLgo+ID4gPiA+IEl0IGlzCj4gPiA+ ID4gPiA+ID4gPiBpbiB0b29scy9tZW1vcnktbW9kZWwgaW4gcmVjZW50IGtlcm5lbCBzb3VyY2Ug dHJlZXMsIHdoaWNoCj4gPiBpbmNsdWRlcwo+ID4gPiA+ID4gPiA+ID4gZG9jdW1lbnRhdGlvbi4g IFRoaXMgaXMgYW4gZXhlY3V0YWJsZSBtb2RlbCwgd2hpY2ggbWVhbnMgdGhhdAo+ID4geW91Cj4g PiA+ID4gPiA+ID4gPiBjYW4gY3JlYXRlIGxpdG11cyB0ZXN0cyBhbmQgaGF2ZSB0aGUgbW9kZWwg Zm9ybWFsbHkgYW5kCj4gPiA+ID4gYXV0b21hdGljYWxseQo+ID4gPiA+ID4gPiA+ID4gZXZhbHVh dGUgdGhlbS4KPiA+ID4gPiA+ID4gPiA+Cj4gPiA+ID4gPiA+ID4gPiBUaGVyZSBhcmUgYWxzbyBh IG51bWJlciBvZiBwdWJsaWNhdGlvbnMgY292ZXJpbmcgTEtNTToKPiA+ID4gPiA+ID4gPiA+Cj4g PiA+ID4gPiA+ID4gPiBvICAgICAgIEEgZm9ybWFsIGtlcm5lbCBtZW1vcnktb3JkZXJpbmcgbW9k ZWwKPiA+ID4gPiA+ID4gPiA+ICAgICAgICAgaHR0cHM6Ly9sd24ubmV0L0FydGljbGVzLzcxODYy OC8KPiA+ID4gPiA+ID4gPiA+ICAgICAgICAgaHR0cHM6Ly9sd24ubmV0L0FydGljbGVzLzcyMDU1 MC8KPiA+ID4gPiA+ID4gPiA+Cj4gPiA+ID4gPiA+ID4gPiAgICAgICAgIFRoZXNlIGNvdmVyIHRo ZSByZWxlYXNlIHN0b3JlcyBhbmQgZGVwZW5kZW5jeSBvcmRlcmluZwo+ID4gdGhhdAo+ID4gPiA+ ID4gPiA+ID4gICAgICAgICBwcm92aWRlIFJDVSdzIHB1Ymxpc2gtc3Vic2NyaWJlIGd1YXJhbnRl ZXMuCj4gPiA+ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiA+ID4gICAgICAgICBCYWNrdXAgbWF0ZXJp YWwgaGVyZToKPiA+ID4gPiA+ID4gPiA+Cj4gPiA+ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiA+ID4K PiA+ID4gPiA+ID4KPiA+ID4gPgo+ID4gaHR0cHM6Ly9taXJyb3JzLmVkZ2Uua2VybmVsLm9yZy9w dWIvbGludXgva2VybmVsL3Blb3BsZS9wYXVsbWNrL0xXTkxpbnV4TU0vCj4gPiA+ID4gPiA+ID4g Pgo+ID4gPiA+ID4gPiA+ID4gICAgICAgICBXaXRoIHRoZXNlIHR3byBsaWtlbHkgYmVpbmcgb2Yg cGFydGljdWxhciBpbnRlcmVzdDoKPiA+ID4gPiA+ID4gPiA+Cj4gPiA+ID4gPiA+ID4gPgo+ID4g PiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4KPiA+ID4gPgo+ID4gaHR0cHM6Ly9taXJyb3JzLmVkZ2Uu a2VybmVsLm9yZy9wdWIvbGludXgva2VybmVsL3Blb3BsZS9wYXVsbWNrL0xXTkxpbnV4TU0vUkNV Z3VhcmFudGVlcy5odG1sCj4gPiA+ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiA+ID4KPiA+ID4gPiA+ ID4KPiA+ID4gPgo+ID4gaHR0cHM6Ly9taXJyb3JzLmVkZ2Uua2VybmVsLm9yZy9wdWIvbGludXgv a2VybmVsL3Blb3BsZS9wYXVsbWNrL0xXTkxpbnV4TU0vc3JjdS5odG1sCj4gPiA+ID4gPiA+ID4g Pgo+ID4gPiA+ID4gPiA+ID4gbyAgICAgICBGcmlnaHRlbmluZyBTbWFsbCBDaGlsZHJlbiBhbmQg RGlzY29uY2VydGluZyBHcm93bi11cHM6Cj4gPiA+ID4gPiA+ID4gPiBDb25jdXJyZW5jeSBpbiB0 aGUgTGludXggS2VybmVsCj4gPiA+ID4gPiA+ID4gPiAgICAgICAgIGh0dHBzOi8vZGwuYWNtLm9y Zy9jaXRhdGlvbi5jZm0/aWQ9MzE3NzE1Ngo+ID4gPiA+ID4gPiA+ID4KPiA+ID4gPiBodHRwOi8v d3d3MC5jcy51Y2wuYWMudWsvc3RhZmYvai5hbGdsYXZlL3BhcGVycy9hc3Bsb3MxOC5wZGYKPiA+ ID4gPiA+ID4gPiA+Cj4gPiA+ID4gPiA+ID4gPiAgICAgICAgIEJhY2t1cCBtYXRlcmlhbDoKPiA+ ID4gPiA+ID4gPiA+Cj4gPiA+ID4gPiA+ID4gPiAgICAgICAgIGh0dHA6Ly9kaXkuaW5yaWEuZnIv bGludXgvCj4gPiA+ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiA+ID4gbyAgICAgICBXaG8ncyBhZnJh aWQgb2YgYSBiaWcgYmFkIG9wdGltaXppbmcgY29tcGlsZXI/Cj4gPiA+ID4gPiA+ID4gPiAgICAg ICAgIGh0dHBzOi8vbHduLm5ldC9BcnRpY2xlcy83OTMyNTMvCj4gPiA+ID4gPiA+ID4gPgo+ID4g PiA+ID4gPiA+ID4gbyAgICAgICBDYWxpYnJhdGluZyB5b3VyIGZlYXIgb2YgYmlnIGJhZCBvcHRp bWl6aW5nIGNvbXBpbGVycwo+ID4gPiA+ID4gPiA+ID4gICAgICAgICBodHRwczovL2x3bi5uZXQv QXJ0aWNsZXMvNzk5MjE4Lwo+ID4gPiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4gPiA+ICAgICAgICAg VGhlc2UgbGFzdCB0d28ganVzdGlmeSB1c2Ugb2Ygbm9ybWFsIEMtbGFuZ3VhZ2UKPiA+IGFzc2ln bm1lbnQKPiA+ID4gPiA+ID4gPiA+ICAgICAgICAgc3RhdGVtZW50cyB0byBpbml0aWFsaXplIGFu ZCBhY2Nlc3MgZGF0YSByZWZlcmVuY2VkIGJ5Cj4gPiA+ID4gPiA+ID4gPiAgICAgICAgIFJDVS1w cm90ZWN0ZWQgcG9pbnRlcnMuCj4gPiA+ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiA+ID4gVGhlcmUg aXMgYSBsYXJnZSBib2R5IG9mIGxpdG11cyB0ZXN0cyAodGhvdXNhbmRzIG9mIHRoZW0pIGhlcmU6 Cj4gPiA+ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiA+ID4gICAgICAgICBodHRwczovL2dpdGh1Yi5j b20vcGF1bG1ja3JjdS9saXRtdXMKPiA+ID4gPiA+ID4gPiA+Cj4gPiA+ID4gPiA+ID4gPiBNYW55 IG9mIHRoZXNlIGxpdG11cyB0ZXN0cyBpbnZvbHZlIFJDVSwgYW5kIHRoZXNlIGNhbiBiZQo+ID4g bG9jYXRlZCBieQo+ID4gPiA+ID4gPiA+ID4gc2VhcmNoIGZvciBmaWxlcyBjb250YWluaW5nIHJj dV9yZWFkX2xvY2soKSwgcmN1X3JlYWRfdW5sb2NrKCksCj4gPiA+ID4gPiA+ID4gPiBzeW5jaHJv bml6ZV9yY3UoKSwgYW5kIHNvIG9uLgo+ID4gPiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4gPiA+IE9y IHdlcmUgeW91IGxvb2tpbmcgZm9yIHNvbWV0aGluZyBlbHNlPwo+ID4gPiA+ID4gPiA+ID4KPiA+ ID4gPiA+ID4gPiA+ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgVGhhbngsCj4gPiBQYXVsCj4gPiA+ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiA+ ID4gPiBUaGFua3MsCj4gPiA+ID4gPiA+ID4gPiA+Cj4gPiA+ID4gPiA+ID4gPiA+IE1hdGhpZXUK PiA+ID4gPiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4gPiA+ID4gPj4gYXMgYSBzdGFydGluZyBwb2lu dC4KPiA+ID4gPiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4gPiA+ID4gPj4gVGhhbmtzLAo+ID4gPiA+ ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiA+ID4gPiA+PiBNYXRoaWV1Cj4gPiA+ID4gPiA+ID4gPiA+ Cj4gPiA+ID4gPiA+ID4gPiA+ID4+PiBJIGtub3cgdGhlcmUgYXJlIHNvbWUgY29uc2lzdGVuY3kg bW9kZWxzIGluIHRoZSBkYXRhYmFzZQo+ID4gYXJlYQo+ID4gPiA+ID4gPiAoc3VjaAo+ID4gPiA+ ID4gPiA+ID4gYXMgUFJBTSwKPiA+ID4gPiA+ID4gPiA+ID4gPj4+IFJlYWQgVW5jb21taXR0ZWQs IGV0YykgZnJvbSBbCj4gPiBodHRwczovL2plcHNlbi5pby9jb25zaXN0ZW5jeQo+ID4gPiA+IHwK PiA+ID4gPiA+ID4gPiA+ID4gPj4+IGh0dHBzOi8vamVwc2VuLmlvL2NvbnNpc3RlbmN5IF0gYW5k IFsxXS4KPiA+ID4gPiA+ID4gPiA+ID4gPj4+IEhvdyBkb2VzIFJDVSByZWxhdGVkIHRvIHRob3Nl IGNvbnNpc3RlbmN5IG1vZGVscz8KPiA+ID4gPiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4gPiA+ID4g Pj4+IEkgYWxzbyBmb3VuZCBzb21lIGNvbW1lbnRzIG9ubGluZSAoT25lIGtleSBkaXN0aW5jdGlv biBpcwo+ID4gPiA+IHRoYXQKPiA+ID4gPiA+ID4gYm90aAo+ID4gPiA+ID4gPiA+ID4gTVZDQyBh bmQgUkxVCj4gPiA+ID4gPiA+ID4gPiA+ID4+PiBwcm92aWRlIG11Y2ggc3Ryb25nZXIgY29uc2lz dGVuY3kgZ3VhcmFudGVlcyB0byByZWFkZXJzCj4gPiB0aGFuCj4gPiA+ID4gZG9lcwo+ID4gPiA+ ID4gPiA+ID4gUkNVIC4uLikgKCBbCj4gPiA+ID4gPiA+ID4gPiA+ID4+PiBodHRwczovL2x3bi5u ZXQvQXJ0aWNsZXMvNzc3MDM2LyB8Cj4gPiA+ID4gPiA+IGh0dHBzOi8vbHduLm5ldC9BcnRpY2xl cy83NzcwMzYvCj4gPiA+ID4gPiA+ID4gPiBdICkuCj4gPiA+ID4gPiA+ID4gPiA+ID4+PiBJIGRv IG5vdCB1bmRlcnN0YW5kIGhvdyB3ZSByZWFzb24vZHJlc2liZS9jb21wYXJlIHRoZQo+ID4gPiA+ IGNvbnNpc3RlbmN5Cj4gPiA+ID4gPiA+ID4gPiBndWFyYW50ZWVzLiAoCj4gPiA+ID4gPiA+ID4g PiA+ID4+PiBJIGV2ZW4gZG8gbm90IGtub3cgd2hhdCBjb25zaXN0ZW5jeSBndWFyYW50ZWVzIHBy b3ZpZGVkCj4gPiBieQo+ID4gPiA+IFJDVQo+ID4gPiA+ID4gPiA+ID4gZm9ybWFsbHkpCj4gPiA+ ID4gPiA+ID4gPiA+ID4+PiBDb3VsZCBzb21lb25lIGV4cGxhaW4gdGhpcyB0byBtZT8KPiA+ID4g PiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4gPiA+ID4gPj4+IFsxXSBCYWlsaXMsIFAuLCBEYXZpZHNv biwgQS4sIEZla2V0ZSwgQS4sIEdob2RzaSwgQS4sCj4gPiA+ID4gPiA+IEhlbGxlcnN0ZWluLAo+ ID4gPiA+ID4gPiA+ID4gSi4gTS4sICYKPiA+ID4gPiA+ID4gPiA+ID4gPj4+IFN0b2ljYSwgSS4g KDIwMTMpLiBIaWdobHkgYXZhaWxhYmxlIHRyYW5zYWN0aW9uczoKPiA+IFZpcnR1ZXMgYW5kCj4g PiA+ID4gPiA+ID4gPiBsaW1pdGF0aW9ucy4KPiA+ID4gPiA+ID4gPiA+ID4gPj4+IFByb2NlZWRp bmdzIG9mIHRoZSBWTERCIEVuZG93bWVudCwgNygzKSwgMTgxLTE5Mi4KPiA+ID4gPiA+ID4gPiA+ ID4KPiA+ID4gPiA+ID4gPiA+ID4gPj4+IFRoYW5rcwo+ID4gPiA+ID4gPiA+ID4gPiA+Pj4gWXV4 aW4KPiA+ID4gPiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4gPiA+ID4gPj4+IF9fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCj4gPiA+ID4gPiA+ID4gPiA+ID4+PiBs dHRuZy1kZXYgbWFpbGluZyBsaXN0Cj4gPiA+ID4gPiA+ID4gPiA+ID4+PiBbIG1haWx0bzpsdHRu Zy1kZXZAbGlzdHMubHR0bmcub3JnIHwKPiA+ID4gPiBsdHRuZy1kZXZAbGlzdHMubHR0bmcub3Jn IF0KPiA+ID4gPiA+ID4gPiA+ID4gPj4+IFsKPiA+IGh0dHBzOi8vbGlzdHMubHR0bmcub3JnL2Nn aS1iaW4vbWFpbG1hbi9saXN0aW5mby9sdHRuZy1kZXYKPiA+ID4gPiB8Cj4gPiA+ID4gPiA+ID4g PiA+ID4+Pgo+ID4gaHR0cHM6Ly9saXN0cy5sdHRuZy5vcmcvY2dpLWJpbi9tYWlsbWFuL2xpc3Rp bmZvL2x0dG5nLWRldiBdCj4gPiA+ID4gPiA+ID4gPiA+Cj4gPiA+ID4gPiA+ID4gPiA+ID4+IC0t Cj4gPiA+ID4gPiA+ID4gPiA+ID4+IE1hdGhpZXUgRGVzbm95ZXJzCj4gPiA+ID4gPiA+ID4gPiA+ ID4+IEVmZmljaU9TIEluYy4KPiA+ID4gPiA+ID4gPiA+ID4gPj4gWyBodHRwOi8vd3d3LmVmZmlj aW9zLmNvbS8gfCBodHRwOi8vd3d3LmVmZmljaW9zLmNvbSBdCj4gPiA+ID4gPiA+ID4gPiA+Cj4g PiA+ID4gPiA+ID4gPiA+IC0tCj4gPiA+ID4gPiA+ID4gPiA+IE1hdGhpZXUgRGVzbm95ZXJzCj4g PiA+ID4gPiA+ID4gPiA+IEVmZmljaU9TIEluYy4KPiA+ID4gPiA+ID4gPiA+ID4gaHR0cDovL3d3 dy5lZmZpY2lvcy5jb20KPiA+ID4gPiA+ID4gPiA+Cj4gPiA+ID4gPiA+Cj4gPiA+ID4KPiA+Cl9f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCmx0dG5nLWRldiBt YWlsaW5nIGxpc3QKbHR0bmctZGV2QGxpc3RzLmx0dG5nLm9yZwpodHRwczovL2xpc3RzLmx0dG5n Lm9yZy9jZ2ktYmluL21haWxtYW4vbGlzdGluZm8vbHR0bmctZGV2Cg==