From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 03F23C433F5 for ; Fri, 28 Jan 2022 23:21:21 +0000 (UTC) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-352-lLhLGkNGPtSY7qL2xJ3hIw-1; Fri, 28 Jan 2022 18:21:16 -0500 X-MC-Unique: lLhLGkNGPtSY7qL2xJ3hIw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id DDE2F835B4C; Fri, 28 Jan 2022 23:21:11 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 1B28A6C18C; Fri, 28 Jan 2022 23:21:11 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 87C711809CB8; Fri, 28 Jan 2022 23:21:08 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 20SNL6iu026081 for ; Fri, 28 Jan 2022 18:21:06 -0500 Received: by smtp.corp.redhat.com (Postfix) id 1D792C23DD3; Fri, 28 Jan 2022 23:21:06 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast05.extmail.prod.ext.rdu2.redhat.com [10.11.55.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 18ECFC23DCE for ; Fri, 28 Jan 2022 23:21:06 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-2.mimecast.com [207.211.31.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id EFA0F802A5E for ; Fri, 28 Jan 2022 23:21:05 +0000 (UTC) Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-103-5N8sjuzEOHmaBAPw4wPy9g-1; Fri, 28 Jan 2022 18:21:04 -0500 X-MC-Unique: 5N8sjuzEOHmaBAPw4wPy9g-1 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 12D611F37B; Fri, 28 Jan 2022 23:21:02 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id A4D1613AAD; Fri, 28 Jan 2022 23:21:01 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id voj3JV169GFHEAAAMHmgww (envelope-from ); Fri, 28 Jan 2022 23:21:01 +0000 Message-ID: <92de9eff521e2702e364f7aa3cce6927d9d9c03c.camel@suse.com> From: Martin Wilck To: Zdenek Kabelac , Zdenek Kabelac , David Teigland , Peter Rajnoha Date: Sat, 29 Jan 2022 00:21:00 +0100 In-Reply-To: References: <20220128134229.1783-1-mwilck@suse.com> <10ad6fc0-6c24-c98b-4a02-2140883af72d@gmail.com> <0a55dd1393df2c125f8cb443daaeb7d1b7162bcc.camel@suse.com> User-Agent: Evolution 3.42.3 MIME-Version: 1.0 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.85 on 10.11.54.8 X-MIME-Autoconverted: from quoted-printable to 8bit by lists01.pubmisc.prod.ext.phx2.redhat.com id 20SNL6iu026081 X-loop: dm-devel@redhat.com Cc: dm-devel@redhat.com, Heming Zhao , Franck Bui , lvm-devel@redhat.com Subject: Re: [dm-devel] [PATCH] udev: create symlinks and watch even in suspended state X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 T24gRnJpLCAyMDIyLTAxLTI4IGF0IDIyOjA2ICswMTAwLCBaZGVuZWsgS2FiZWxhYyB3cm90ZToK PiBEbmUgMjguIDAxLiAyMiB2IDE5OjQ2IE1hcnRpbiBXaWxjayBuYXBzYWwoYSk6Cj4gPiBPbiBG cmksIDIwMjItMDEtMjggYXQgMTg6NDcgKzAxMDAsIFpkZW5layBLYWJlbGFjIHdyb3RlOgo+ID4g PiBEbmUgMjguIDAxLiAyMiB2IDE3OjAyIE1hcnRpbiBXaWxjayBuYXBzYWwoYSk6Cj4gPiA+ID4g T24gRnJpLCAyMDIyLTAxLTI4IGF0IDE2OjU3ICswMTAwLCBNYXJ0aW4gV2lsY2sgd3JvdGU6Cj4g PiA+ID4gPiBJdCdzIGEgcmFjZSBjb25kaXRpb24uIEl0IHByb2JhYmx5IGhhcHBlbnMgd2hpbGUg bXVsdGlwYXRoZAo+ID4gPiA+ID4gaXMKPiA+ID4gPiA+IHJlbG9hZGluZyBhIG1hcCAoKiksIHN1 c3BlbmRpbmcgaXQgZHVyaW5nIHRoZSB0YWJsZSByZWxvYWQuCj4gPiA+ID4gPiBUaGUKPiA+ID4g PiA+IGRldmljZQo+ID4gPiA+ID4gd2lsbCBiZSByZXN1bWVkIGEgZmV3IGZyYWN0aW9ucyBvZiBh IHNlY29uZCBsYXRlciAoc28geWVzLAo+ID4gPiA+ID4gaXQncwo+ID4gPiA+ID4gInF1aWNrIiks IGJ1dCB0aGVuIGl0J3MgdG9vIGxhdGUKPiA+ID4gPiBNb3JlIHByZWNpc2VseTogVGhlIHN1c3Bl bmQgc3RhdGUgaXRzZWxmIG1heSBhY3R1YWxseSBub3QgbGFzdAo+ID4gPiA+IGxvbmdlcgo+ID4g PiA+IHRoYW4gYSBmZXcgbXMuIEJ1dCBvbmNlIHRoZSBzeW1saW5rIGlzIGJlbnQgdG8gcG9pbnQg dG8gdGhlCj4gPiA+ID4gd3JvbmcKPiA+ID4gPiBkZXZpY2UsIGl0IHdpbGwgcmVtYWluIHNvLCB1 bnRpbCB0aGUgQ0hBTkdFIGV2ZW50IGZvbGxvd2luZyB0aGUKPiA+ID4gPiBkZXZpY2UKPiA+ID4g PiByZXN1bWUgaXMgc3VjY2Vzc2Z1bGx5IHByb2Nlc3NlZCBieSB1ZGV2LCB3aGljaCBtYXkgaGFw cGVuCj4gPiA+ID4gc3Vic3RhbnRpYWxseSBsYXRlci4gQW5kIGl0J3MgdGhhdCBsb25nZXIgdGlt ZSBzcGFuIHdoaWNoCj4gPiA+ID4gbWF0dGVycwo+ID4gPiA+IGZvcgo+ID4gPiA+IHN5c3RlbWQn cyBtb3VudCBhdHRlbXB0IChvciBMVk0gZGV2aWNlIGFjdGl2YXRpb24sIGZvciB0aGF0Cj4gPiA+ ID4gbWF0dGVyKS4KPiA+ID4gPiAKPiA+ID4gPiAKPiA+ID4gCj4gPiA+IFRoaXMgbG9va3MgbGlr ZSB5b3UgYXJlIHRyeWluZyB0byBtYXNrLW91dCBkaWZmZXJlbnQKPiA+ID4gc3luY2hyb25pemF0 aW9uCj4gPiA+IGJ1Zy4KPiA+IFBsZWFzZSBleHBsYWluLiBXaGF0IGJ1ZyB3b3VsZCB0aGF0IGJl LCBpbiB3aGF0IHN1YnN5c3RlbT/CoEl0IHRha2VzCj4gPiB0aW1lIHVudGlsIGEgZG0tdHJpZ2dl cmVkIENIQU5HRSBldmVudCBtYWtlcyBpdHMgd2F5IHRocm91Z2ggdGhlCj4gPiB1ZGV2Cj4gPiBl dmVudCBxdWV1ZSBhbmQgY29tcGxldGVzIHJ1bGVzIHByb2Nlc3NpbmcuIFRoYXQncyBwZXJmZWN0 bHkKPiA+IG5vcm1hbC4KPiAKPiAKPiBXZWxsIGl0IGlzIGF0IHRoZSBwb2ludCB3ZSBuZWVkIHRv IGtub3cgZXhhY3RseSB3aGljaCBkZXZpY2UgaW4gd2hpY2gKPiBzdGF0ZSAKPiBjYXVzZXMgeW91 IHByb2JsZW0uIFRoZW4gd2UgbmVlZCB0byBzZWUgd2hhdCBpcyB3cm9uZyBpbiB0aGUgd2hvbGUK PiBwcm9jZXNzLgoKVGhpcyBpcyBob3cgbXVsdGlwYXRoZCBsb2FkcyB0aGUgbXVsdGlwYXRoIG1h cCBiZWZvcmUgdGhlIGVycm9yIG9jY3VyczoKClsgIDEyNy4zNTI2MTRdIGxvY2FsaG9zdCBtdWx0 aXBhdGhkWzEwNTldOiAzNjAwYTA5ODAwMGFhZDFlMzAwMDAwYjRiNWEyNzVkNDU6IHJlbG9hZCBb MCAxMzQyMTc3MjggbXVsdGlwYXRoIDMgcGdfaW5pdF9yZXRyaWVzIDUwIHF1ZXVlX2lmX25vX3Bh dGggMSBhbHVhIDIgMSBzZXJ2aWNlLXRpbWUgMCAxIDEgODo4MCAxIHNlcnZpY2UtdGltZSAwIDIg MSA4OjE2IDEgODoxNDQgMV0KCgpUaGUgbWFwcGluZyBvZiB0aGUgcGFydGl0aW9uIGRldmljZSBp cyB0aGlzIChhZnRlciBib290KToKCiMgZG1zZXR1cCB0YWJsZSAvZGV2L2RtLTEzCjAgMTMzOTUz NTAzIGxpbmVhciAyNTQ6MTAgMjY0MTkyCgojIGRtc2V0dXAgbHMgLS10cmVlICMgZXhjZXJwdAoz NjAwYTA5ODAwMGFhZDFlMzAwMDAwYjRiNWEyNzVkNDUtcGFydDIgKDI1NDoxMykKIOKUlOKUgDM2 MDBhMDk4MDAwYWFkMWUzMDAwMDBiNGI1YTI3NWQ0NSAoMjU0OjEwKQogICAg4pSc4pSAICg4OjE0 NCkKICAgIOKUnOKUgCAoODoxNikKICAgIOKUlOKUgCAoODo4MCkKCkhlcmUgeW91IGNhbiBzZWUg dGhlIGxpbmtzIHRoYXQgdWRldiBjcmVhdGVzIGZvciB0aGUgZGV2aWNlIHdoZW4gaXQncwpzZXQg dXAgZHVyaW5nIGluaXRyZCBwcm9jZXNzaW5nIChub3RlIHRoZSBieS11dWlkIGxpbmspOgoKWyAg IDQwLjYxMTk4OV0gbG9jYWxob3N0IHN5c3RlbWQtdWRldmRbNTU1XTogZG0tMTM6IEhhbmRsaW5n IGRldmljZSBub2RlICcvZGV2L2RtLTEzJywgZGV2bnVtPWIyNTQ6MTMKWyAgIDQwLjYxMjA3M10g bG9jYWxob3N0IHN5c3RlbWQtdWRldmRbNTU1XTogZG0tMTM6IENyZWF0aW5nIHN5bWxpbmsgJy9k ZXYvbWFwcGVyLzM2MDBhMDk4MDAwYWFkMWUzMDAwMDBiNGI1YTI3NWQ0NS1wYXJ0MicgdG8gJy4u L2RtLTEzJwpbICAgNDAuNjEyMzc0XSBsb2NhbGhvc3Qgc3lzdGVtZC11ZGV2ZFs1NTVdOiBkbS0x MzogQXRvbWljYWxseSByZXBsYWNlICcvZGV2L2Rpc2svYnktaWQvd3duLTB4NjAwYTA5ODAwMGFh ZDFlMzAwMDAwYjRiNWEyNzVkNDUtcGFydDInClsgICA0MC42MTI0NzZdIGxvY2FsaG9zdCBzeXN0 ZW1kLXVkZXZkWzU1NV06IGRtLTEzOiBDcmVhdGluZyBzeW1saW5rICcvZGV2L2Rpc2svYnktaWQv ZG0tdXVpZC1wYXJ0Mi1tcGF0aC0zNjAwYTA5ODAwMGFhZDFlMzAwMDAwYjRiNWEyNzVkNDUnIHRv ICcuLi8uLi9kbS0xMycKWyAgIDQwLjYxMjgyN10gbG9jYWxob3N0IHN5c3RlbWQtdWRldmRbNTU1 XTogZG0tMTM6IEF0b21pY2FsbHkgcmVwbGFjZSAnL2Rldi9kaXNrL2J5LXBhcnR1dWlkLzFjMmY3 MGUwLWZiOTEtNDlmNS04MjYwLTM4ZWFjYWY3OTkyYicKWyAgIDQwLjYxMjk4MF0gbG9jYWxob3N0 IHN5c3RlbWQtdWRldmRbNTU1XTogZG0tMTM6IENyZWF0aW5nIHN5bWxpbmsgJy9kZXYvZGlzay9i eS1pZC9kbS1uYW1lLTM2MDBhMDk4MDAwYWFkMWUzMDAwMDBiNGI1YTI3NWQ0NS1wYXJ0MicgdG8g Jy4uLy4uL2RtLTEzJwpbICAgNDAuNjEzMjIyXSBsb2NhbGhvc3Qgc3lzdGVtZC11ZGV2ZFs1NTVd OiBkbS0xMzogQXRvbWljYWxseSByZXBsYWNlICcvZGV2L2Rpc2svYnktaWQvc2NzaS0zNjAwYTA5 ODAwMGFhZDFlMzAwMDAwYjRiNWEyNzVkNDUtcGFydDInClsgICA0MC42MTM1MDJdIGxvY2FsaG9z dCBzeXN0ZW1kLXVkZXZkWzU1NV06IGRtLTEzOiBBdG9taWNhbGx5IHJlcGxhY2UgJy9kZXYvZGlz ay9ieS11dWlkL2U0MGQzMDA1LWFiMmYtNDg0NS1iZDgzLWJlNWZkMDllNjJhMCcKWyAgIDQwLjYx MzUzMV0gbG9jYWxob3N0IHN5c3RlbWQtdWRldmRbNTU1XTogZG0tMTM6IFByZXNlcnZlIGFscmVh ZHkgZXhpc3Rpbmcgc3ltbGluayAnL2Rldi9ibG9jay8yNTQ6MTMnIHRvICcuLi9kbS0xMycKClRo ZSBieS11dWlkIGxpbmsgaXMgdGhlbiByZW1vdmVkIGFmdGVyIHN3aXRjaGluZyByb290IGFzIEkg c2hvd2VkIAppbiBteSBwcmV2aW91cyBwb3N0LiBUaGUgcmVhc29uIGlzIHRoYXQgdGhlIHJlc3Bl Y3RpdmUgU1lNTElOSyBsaW5lIGluCjEzLWRtLWRpc2sucnVsZXMgaXMgc2tpcHBlZCBmb3Igc3Vz cGVuZGVkIGRldmljZXMsIGFuZCBteSBwYXRjaCBmaXhlcwp0aGF0LgoKQXQgdGhlIHRpbWUgdGhp cyBoYXBwZW5zLCB0aGUgc2FtZSBwYXJ0aXRvbiBfaXMgYWxyZWFkeSBtb3VudGVkXyBhcwpidHJm cyByb290IGZpbGUgc3lzdGVtLiBUaGlzIGZpbGUgc3lzdGVtIGhhcyBtdWx0aXBsZSBzdWJ2b2x1 bWVzIGZvcgovdG1wLCAvdmFyLCAvdXNyL2xvY2FsLCBldGMuLCB3aGljaCBuZWVkIHRvIGJlIG1v dW50ZWQgYWZ0ZXIgc3dpdGNoaW5nCnJvb3QuIFRoZSBwcm9ibGVtIG9jY3VycyB3aGVuIHRoZXNl IG1vdW50cyBhcmUgYXR0ZW1wdGVkLiBUeXBpY2FsbHkKc29tZSBzdWNjZWVkLCBhbmQgc29tZSBk b24ndCwgZGVwZW5kaW5nIG9uIHRoZSB0aW1pbmcuCgo+IEFsbCBJJ20gc2F5aW5nIGhlcmUgaXMg LSB0aGF0IHRoZSBwcm9wb3NlZCBwYXRjaCBpcyBub3QgZml4aW5nIGJ1ZyAtCj4gYnV0IAo+IHJh dGhlciBtYXNraW5nL21pbmltaXppbmcgd2luZG93IGZvciBlcnJvci4KCkFGQUlDUyBteSBwYXRj aCBlbGltaW5hdGVzIHRoZSB3aW5kb3cgZm9yIHRoaXMgZXJyb3IgZW50aXJlbHkuCgo+ID4gSSBz YXkgdGhlIGJ1ZyBpcyBpbiB0aGlzIHVkZXYgcnVsZSBmaWxlLiBUaGUgc2V0IG9mIHByb3BlcnRp ZXMgYW5kCj4gPiBzeW1saW5rcyBvZiBhIHN5c3RlbWQtbWFuYWdlZCBkZXZpY2UgbXVzdCBub3Qg Y2hhbmdlIHdoZW4gbmV3Cj4gPiB1ZXZlbnRzCj4gPiBmb3IgdGhpcyBkZXZpY2UgYXJlIHByb2Nl c3NlZCwgdW5sZXNzIGFic29sdXRlbHkgbmVjZXNzYXJ5LiBUaGUKPiA+IGN1cnJlbnQKPiAKPiAK PiBUaGUgcXVlc3Rpb24gaXMgLSB3aGV0aGVyIHRoZSBldmVudCBpcyBtZWFudCB0byBiZSB0aGVy ZSBpZiB0aGVyZSBpcwo+IG5vIAo+IGNoYW5nZSwgdGhhdCBzaG91bGQgYmUgcmVmbGVjdGVkIGlu IHRoZSBzeXN0ZW0uIEl0IHJlYWxseSBnb2VzIHRvICMxCj4gcXVlc3Rpb24gCj4gYWJvdXQgZGV0 YWlsZWQgc3RhdGUgb2YgRE0gdGFibGVzIGFuZCB5b3VyIG9ic2VydmVkIGJsb2NrZWQgc3lzdGVt LsKgCj4gV2l0aCBsdm0yIAo+IHdlIGFyZSBjYXJlZnVsbCBhYm91dCBzZW5kaW5nIGV2ZW50cy4K ClRoZSBjb2xkcGx1ZyAiYWRkIiBldmVudCBhbHdheXMgaGFwcGVucy4gSXQgY2FuJ3QgYmUgYXZv aWRlZC4gSXQgaXMKbmVjZXNzYXJ5IHRvIG1ha2UgdXNlci1zcGFjZSBhd2FyZSBvZiB0aGUgZGV2 aWNlIGFmdGVyIHN3aXRjaGluZyByb290LgpBbmQgZGV2aWNlLW1hcHBlciBydWxlcyBnbyBpbnRv IGdyZWF0IGRldGFpbCBob3cgdGhleSBkZWFsIHdpdGggaXQsIGFzCnlvdSBhcmUgY2VydGFpbmx5 IGF3YXJlLgoKPiAKPiA+IHJ1bGUgdmlvbGF0ZXMgdGhpcyBwcmluY2lwbGU6IElmIGEgZGV2aWNl IHRoYXQgY29udGFpbnMgYSBmaWxlCj4gPiBzeXN0ZW0KPiA+IGlzIHN1c3BlbmRlZCwgaXQgY29u dGludWVzIHRvIGNvbnRhaW4gdGhpcyBmaWxlIHN5c3RlbSwgYW5kIHRoZSBieS0KPiA+IHV1aWQK PiA+IHN5bWxpbmsgZm9yIHRoZSBmaWxlIHN5c3RlbSBzaG91bGQgYmUgcHJlc2VydmVkLgo+ID4g WW91IGNhbiBlYXNpbHkgdGVzdCB0aGlzLiBDaGVjayB0aGUgc2V0IG9mIHN5bWxpbmtzIGZvciBh IHBhcnRpdGlvbgo+ID4gb24KPiA+IGEgbXVsdGlwYXRoIGRldmljZSwgc3VzcGVuZCB0aGUgZGV2 aWNlLCBydW4gInVkZXZhZG0gdHJpZ2dlciAtYwo+ID4gYWRkIiBvbgo+ID4gdGhlIGRldmljZSAo c2ltdWxhdGluZyBjb2xkcGx1ZyksIGFuZCBjaGVjayB0aGUgc2V0IG9mIHN5bWxpbmtzCj4gPiBh Z2Fpbi4KPiAKPiBXaG9ldmVyIHJlYWRzIHN1c3BlbmRlZCBkZXZpY2UgaGFzIHRvIHdhaXQgZm9y ICdyZXN1bWUnCgpJIHN1cHBvc2UgaWYgbW91bnQoOCkgb3BlbmVkIC9kZXYvZG0tMTMgZGlyZWN0 bHksIGl0IHdvdWxkIGp1c3QgaGFuZwp3aGlsZSB0aGUgZGV2aWNlIHdhcyBzdXNwZW5kZWQsIGFu ZCBzdGFydCBJTyBhZnRlcndhcmRzLiBCdXQgdGhpcyBpc24ndApob3cgdGhpcyB3b3Jrcy4gbW91 bnQoOCkgb3BlbnMgdGhlIC9kZXYvZGlzay9ieS11dWlkIHN5bWxpbmsgd2hpY2ggbm93CnBvaW50 cyB0byB0aGUgdW5kZXJseWluZyBkZXZpY2UgKHNkYjIgLSB1ZGV2IGJlbnQgdGhlIHN5bWxpbmsg dG8gcG9pbnQKdG8gc2RiMiBhZnRlciBpdCB3YXMgZHJvcHBlZCBmcm9tIGRtLTEzKS4gT3Blbmlu ZyBzZGIyIGRvZXNuJ3QgaGFuZywgaXQKZmFpbHMgd2l0aCAtRUJVU1kgYmVjYXVzZSB0aGUgZGV2 aWNlIGlzIGxvY2tlZCBieSBkbS4gVGh1cyB0aGUgbW91bnQKZmFpbHMuIG1vdW50KDgpIGhhcyBu byB3YXkgdG8gZGV0ZWN0IHdoeSB0aGlzIGlzIHNvLiBzeXN0ZW1kIG1pZ2h0LCBhbmQKbWlnaHQg YXR0ZW1wdCBhIHJldHJ5LCBidXQgaXQgIGxhY2tzIHRoZSBsb2dpYyBmb3IgZG9pbmcgdGhhdC4K Cj4gPiAtIHNvIHRoZSB0ZXN0IGNhc2UgCj4gd2hlcmUgeW91IHN1c3BlbmQgZGV2aWNlIGZvciBs b25nIHBlcmlvZCBvZiB0aW1lIGFuZCB5b3Ugb2JzZXJ2ZQo+IG90aGVyIAo+IHByb2Nlc3NlZCB3 YWl0aW5nIGZvciBkZXZpY2UgdG8gYmUgcmVzdW1lZCBpcyB3YW50ZWQgYmVoYXZpb3IgLSBpdCdz Cj4gbm90IHZhbGlkIAo+IHRvIGhhdmXCoCBzdXNwZW5kZWQgZGV2aWNlcyBpbiB0aGUgc3lzdGVt LsKgwqAKClVuZm9ydHVuYXRlbHkgZG0gZGV2aWNlcyBhcmUgb2Z0ZW4gc3VzcGVuZGVkIGZvciBh IHNob3J0IHBlcmlvZCBvZgp0aW1lLCB3aGVuIHRoZWlyIHRhYmxlIGlzIHJlbG9hZGVkLiBBcyBJ IHBvaW50ZWQgb3V0IGJlZm9yZSwgaXQncwpub3JtYWwgZm9yIHRoaXMgdG8gaGFwcGVuIGR1cmlu ZyBtdWx0aXBhdGggc3RhcnR1cCwgd2hlbiBwYXRocyBhcmUKYWRkZWQgb3IgcmVtb3ZlZCwgcHJp b3JpdHkgZ3JvdXBzIHN3aXRjaGVkLCBldGMuCgo+IFlvdSBoYXZlIHN0YXRlICdiZWZvcmUnIHN1 c3BlbmTCoCAKPiBhbmQgc3RhdGUgYWZ0ZXIgJ3Jlc3VtZScuIMKgIFRoZXJlIGlzIG5vIHN0YXRl IGZvciAnc3VzcGVuZGVkJyB0eXBlIG9mCj4gZGV2aWNlIAo+IChhcyB0aGUgZGV2aWNlIGNvdWxk IGJlIHJlc3VtZWQgZWl0aGVyIHdpdGggZXhpdGluZyB0YWJsZSBvciBhIG5ldwo+IHRhYmxlLgoK U286IHRoZSBtYXBwaW5nIGNvdWxkIGJlIGVudGlyZWx5IGRpZmZlcmVudCB3aGVuIHRoZSBkZXZp Y2UgaXMgcmVzdW1lZCwKYW5kIGlmIHN5c3RlbWQgdHJpZWQgdG8gbW91bnQgaXQgaW4gdGhlIHRp bWUgc3BhbiBiZXR3ZWVuIHRoZSByZXN1bWUKYW5kIHRoZSBmaW5pc2ggb2YgdGhlIHVldmVudCB0 aGF0IGZvbGxvd3MsIGl0IG1pZ2h0IG1vdW50IGEgdG90YWxseQpkaWZmZXJlbnQgZmlsZSBzeXN0 ZW0uIElzIHRoYXQgd2hhdCB5b3UgbWVhbj8gSWYgeWVzLCBJIHN1cHBvc2UgeW91J3JlCnJpZ2h0 LiBCdXQgaXNuJ3QgdGhhdCBwb3NzaWJsZSB0b2RheSBhbHJlYWR5PyBDb3VsZG4ndCBJIHJlbG9h ZCBhCm1hcHBpbmcgd2l0aCB0b3RhbGx5IGRpZmZlcmVudCBkZXZpY2VzIGFuZCBvZmZzZXRzLCB3 aGlsZSBpdCBpcwptb3VudGVkPwoKQXQgbGVhc3QgZm9yIG11bHRpcGF0aCwgdGhhdCdzIGV4dHJl bWVseSB1bmxpa2VseSB0byBoYXBwZW4sIHdoZXJlYXMKcXVpY2sgcmVsb2FkcyB3aXRoIGEgc2lt aWxhciBtYXBwaW5nIHRoYXQgZG9lc24ndCBhZmZlY3QgdXBwZXIgbGF5ZXJzCmFyZSB2ZXJ5IGNv bW1vbi4KCklmIHlvdSB0YWtlIHRoZSBhYm92ZSBzZXJpb3VzbHksIHlvdSBuZWVkIHRvIHJld3Jp dGUgeW91ciBjb2RlIGZvcgpjb21tdW5pY2F0aW5nIHdpdGggdWRldiBhbmQgc3lzdGVtZC4gSSBn dWVzcyBzeXN0ZW1kIGl0c2VsZiB3b3VsZCBuZWVkCnRvIGNoYW5nZSB0aGUgd2F5IGl0IGhhbmRs ZXMgZG0gZGV2aWNlcywgZnVuZGFtZW50YWxseS4gQmFzaWNhbGx5IHlvdQp3b3VsZCBuZWVkIHRv IHRlbGwgc3lzdGVtZCB0byBjb25zaWRlciBhIHN1c3BlbmRlZCBkZXZpY2UgYXMgdW51c2FibGUs CmFuZCBub3QgbWFrZSBpdCBhdmFpbGFibGUgdG8gdGhlIHN5c3RlbSBhZ2FpbiBiZWZvcmUgaXQn cyByZXN1bWVkIGluIGEKc3RhdGUgdGhhdCdzIHZlcmlmaWVkIHRvIGJlIGNvbXBhdGlibGUgd2l0 aCB0aGUgcHJldmlvdXMgc3RhdGUuIEFuZAp3ZSdkIG5lZWQgZGVhbCB3aXRoIHRoZSBzcGVjaWFs IGNhdmVhdHMgZm9yIG11bHRpcGF0aCwgdG9vLgoKCj4gU28gYW55IHNpbXVsYXRpb24gcmVsYXlp bmcgb24gc3VzcGVuZGVkIGRldmljZSBpcyBiYXNpY2FsbHkgdGVzdGluZwo+IGludmFsaWQgCj4g c3RhdGUgb2Ygc3lzdGVtLgo+IAo+IEFuZCBJJ20gYXNzdW1pbmcgeW91ciBwcm9ibGVtIGlzIHNv bWV0aGluZyBlbHNlIC0gYXMgSSdkIHNheSB5b3UKPiBkb24ndCBsZWFrIAo+IGZvcmdvdHRlbiAn c3VzcGVuZGVkJyBkZXZpY2UgaW4gcmFtZGlzayA/CgpObywgZXZlcnl0aGluZyBpcyBwZXJmZWN0 bHkgZmluZSBpbiB0aGUgaW5pdHJkLiBBbmQgYWZ0ZXJ3YXJkcywgYWxsIGlzCmZpbmUsIHRvbyAt IGV4Y2VwdCBpZiwgYnkgdmVyeSBiYWQgbHVjaywgdGhlIHRpbWluZyBpcyBzdWNoIHRoYXTCoAoK IGEpIHRoZSBjb2xkcGx1ZyAiYWRkIiBldmVudCBoYXBwZW5zwqB3aGlsZSB0aGUgZGV2aWNlIGlz IHN1c3BlbmRlZAogICAgKGFzIEkgc2FpZCwgcHJvYmFibHkgZHVlIHRvIGEgcmVsb2FkIG9wZXJh dGlvbikKIGIpIHN5c3RlbWQgdHJpZXMgdG8gbW91bnQgdGhlIGZpbGUgc3lzdGVtIGR1cmluZyB0 aGUgc2hvcnQgdGltZSBzcGFuCmluIHdoaWNoIHRoZSBzeW1saW5rcyBhcmUgbWlzc2luZyAoaS5l LiBiZXR3ZWVuIHRoZSByZXN1bWUgYW5kIHRoZQpmaW5hbCBwcm9jZXNzaW5nIG9mIHRoZSBDSEFO R0UgZXZlbnQpLgoKSW4gbXkgdGVzdGluZywgdGhpcyBoYXBwZW5zIG9ubHkgb24gb25lIHN5c3Rl bSwgcm91Z2hseSBvbmNlIGV2ZXJ5IDUwCnJlYm9vdHMuCgo+IAo+ID4gWW91J2xsIHNlZSB0aGF0 IGJ5LXV1aWQgb3IgYnktbGFiZWwgbGlua3Mgd2lsbCBiZSBsb3N0IGFuZCB3aWxsIG5vdwo+ID4g cG9pbnQgdG8gb25lIG9mIHRoZSBtYXAncyBwYXRocy4gVGhlc2Ugc3ltbGlua3MgYXJlIGNydWNp YWwgZm9yCj4gPiBtb3VudGluZyBmaWxlIHN5c3RlbXMuIFdpdGggbXkgcGF0Y2gsIHRoZSBzeW1s aW5rcyBhcmUgcHJlc2VydmVkLgo+IAo+IAo+IEknbSBzdGlsbCBtaXNzaW5nIHdoeSAnVVVJRCcg c2hvdWxkIGJlIGxvc3QgPwoKVGhlIGxpbmsgaXMgYm91bmQgdG8gdWRldidzIElEX1BBUlRfRU5U UllfVVVJRCBwcm9wZXJ0eS4gVGhpcyBwcm9wZXJ0eQppcyBzZXQgYnkgSU1QT1JUe2J1aWRsdGlu fT0iYmxraWQiLCBvciB1c2luZwpJTVBPUlR7ZGJ9PSJJRF9QQVJUX0VOVFJZX1VVSUQiLiBUaGUg bGF0dGVyIGlzIGRvbmUgYnkgMTEtZG0tCm1wYXRoLnJ1bGVzIGFuZCAxMS1kbS1wYXJ0cy5ydWxl cywgZm9yIGV4YW1wbGUsIGV4YWN0bHkgZm9yIHRoZSBzYW1lCnJlYXNvbiAtIHdlIGNhbid0IHJp c2sgbG9vc2luZyB0aGUgc3ltbGluayBiZWNhdXNlIGJsa2lkIGZhaWxzLiBUaGlzCndvcmtzIG5p Y2VseSwgdW5sZXNzIERNX1NVU1BFTkRFRCBvciBETV9OT1NDQU4gaXMgc2V0LCBiZWNhdXNlIGlu IHRoaXMKY2FzZSAxMy1kbS1kaXNrLnJ1bGVzIHdvbid0IGNhbGwgU1lNTElOSy4KCj4gV2hlcmUg dGhleSBtaXNzaW5nIGJlZm9yZSBzdXNwZW5kID8KCk5vLgoKPiBBcmUgdGhleSBsb3N0IGFmdGVy IHJlc3VtZSA/CgpObywgYWZ0ZXIgcmVzdW1lIHVkZXYgcmVpbnN0YXRlcyB0aGVtLCBidXQgdGhl biBpdCdzIHRvbyBsYXRlLgoKPiBXaGF0IGlzIGNhdXNpbmcgeW91ciBVVUlEIHRvIGJlIGxvc3Qg PwoKVGhlIFVVSUQgaXMgbmV2ZXIgbG9zdC4gT25seSB0aGUgc3ltbGluayB0aGF0IHJlZmxlY3Rz IGl0LgoKPiAKPiBUbyBjb25uZWN0IHRoaXMgd2l0aCBsdm0yIGxvZ2ljIC3CoCB3aGVuIGx2bTIg Y3JlYXRlcyBvciBkZWxldGVzCj4gZGV2aWNlIC0gaXQgCj4gYWx3YXlzIHdhaXRzIGZvciB1ZGV2 IGNvbmZpcm1hdGlvbsKgIChBVE0gdmlhIGNvb2tpZSBzZW1hcGhvcmUpLiBTbwo+IGkuZS4gd2Ug Cj4gY2FuJ3QgY3JlYXRlIGEgc2l0dWF0aW9uIHdoZXJlIHdlIHdvdWxkIGJlIHRyaWdnZXJpbmfC oCAndWRldicgZGlzawo+IGNyZWF0aW9uIAo+IGV2ZW50IHRoYXQgd291bGQgYmUgJ3JhY2luZycg d2l0aMKgIHVkZXYgZGlzayByZW1vdmFsIHByb2Nlc3NpbmcgLSBzbwo+IHRoZXJlIGlzIAo+IGNs b3NlIHN5bmNocm9uaXphdGlvbiB0byBhdm9pZCB0aGVzZSBraW5kIG9mIHJhY2VzIHdoZXJlIHVk ZXYgd291bGQKPiBiZSAKPiBhY2Nlc3NpbmfCoCBvdXIgTFYgZGV2aWNlcyBpbiBzb21lwqAgYXN5 bmMgcGFyYWxsZWwgbG9naWMgYW5kIHBvc3NpYmx5Cj4gd291bGQgCj4gaGF2ZSAnbW9kaWZlZCcg aXRzIHN5bWxpbmsgZGF0YWJhc2Ugd2l0aCBzb21lIG9ic29sZXRlIGluZm8uCgptdWx0aXBhdGhk IGRvZXMgdGhhdCwgdG9vLiBCdXQgaXQgZG9lc24ndCBhcHBseSBoZXJlLiBCZWNhdXNlIHRoZQpk ZXZpY2UgaXMgbGl2ZSAoc3lzdGVtIGhhcyBtb3VudGVkIGl0IGluIHRoZSBpbml0cmFtZnMgYWxy ZWFkeSBhbmQgaXQncwphY3R1YWxseSBtb3VudGVkIGF0IHRoZSB0aW1lIGFsbCB0aGlzIGhhcHBl bnMsIHNlZSBhYm92ZSkuIFlldCB0aGUKY29sZHBsdWcgZXZlbnQgZG9lcyBhcnJpdmUsIHdlIGNh bid0IGF2b2lkIGl0LiAKCihXZWxsIHdlIGNvdWxkLCBhdCBsZWFzdCBwYXJ0bHksIGJ5IGRlbGF5 aW5nIG11bHRpcGF0aGQgc3RhcnR1cCB1bnRpbAphZnRlciAidWRldiBzZXR0bGUiIGFnYWluLCBi dXQgSSdtIHN1cmUgQmVuIGFncmVlcyB3aXRoIG1lIHRoYXQgd2UnZApyYXRoZXIgbm90IGRvIHRo YXQpLgo+IAo+IAo+ID4gPiBBbHNvIGl0J3Mgd29ydGggdG8gbm90ZSAtIHVzaW5nIHN5bWxpbmtz IGlzIHNvbWV3aGF0IGRvb21lZCBvbgo+ID4gPiBpdHMKPiA+ID4gb3duLgo+ID4gVGhhdCdzIGhv dyBkZXZpY2UgaWRlbnRpZmljYXRpb24gYW5kIGFjdGl2YXRpb24gd2l0aCB1ZGV2IGFuZAo+ID4g c3lzdGVtZAo+ID4gd29ya3MuIEl0IHdvbid0IGNoYW5nZSBhbnkgdGltZSBzb29uLiBFeHBsYWlu IG9uIHdoaWNoIGdyb3VuZHMKPiA+IHlvdSdyZQo+ID4gY2FsbGluZyBpdCAic29tZXdoYXQgZG9v bWVkIi4KPiAKPiBZb3UgY291bGQgZmluZCBteSBkaXNjdXNzaW9uIHdpdGggTGVubmFydMKgIC0g b25lIG9mIG1ham9yIGRlc2lnbgo+IHByb2JsZW0gb2YgCj4gc3ltbGlua3MgaXMgaXQgY2FuJ3Qg aGFuZGxlIGR1cGxpY2F0ZSBkZXZpY2VzIHdpdGggc2FtZSBJRHMuwqAgU28gaWYKPiB5b3UgaGF2 ZSAKPiBtdWx0aXBsZSBkaXNrcyB3aXRoIHNhbWUgaWRlbnRpZmljYXRpb24gLSBzeW1saW5rcyB3 aWxsIGJlIHJhbmRvbWx5Cj4gc3dpdGNoaW5nIAo+IGJldHdlZW4gdGhlbSAtwqAgc28gcGxhaW4g c2ltcGxlICdkZMKgIGlmPWRpc2tBIG9mPWRpc2tCJyBydWlucwo+IHN5bWxpbmtzIGluIHlvdXIg Cj4gc3lzdGVtLgoKdWRldiBoYXMgaXQncyBsaW5rIHByaW9yaXR5IGNvbmNlcHQgZm9yIHRoaXMg cmVhc29uLiBJdCBtYXkgbm90IGJlCnBlcmZlY3QgYnV0IGl0IHdvcmtzIHF1aXRlIHdlbGwsIGFu ZCB0aGUgc3ltbGluayBoYW5kbGluZyBoYXMgaW1wcm92ZWQKYSBsb3QgbGF0ZWx5LiBJIGFtIGN1 cmlvdXMgd2hhdCB5b3VyIGFsdGVybmF0aXZlIGNvbmNlcHQgaXMuIERvIHlvdQpoYXZlIGEgbGlu az8KCj4gWy4uLl0KPiAKPiBIb3cgY291bGQgeW91IGdldCB3cm9uZyBVVUlEIGZvciBhbHJlYWR5 IG1vdW50ZWQgKHNvIGxpa2VseSBhbHNvCj4gb3BlbmVkKSBkZXZpY2U/PwoKV2h5IGRvIHlvdSBz YXkgIndyb25nIFVVSUQiID8gVGhlIFVVSUQgd2FzIGNvcnJlY3QgYWxsIHRoZSB0aW1lLgpJIGV4 cGxhaW5lZCBhYm92ZSB3aGF0IGhhcHBlbmVkLiBZZXMsIHRoZSBkZXZpY2Ugd2FzIG1vdW50ZWQg KHRvIG90aGVyCmJ0cmZzIHN1YnZvbHVtZXMpIHdoZW4gdGhlIGVycm9yIG9jY3VyZWQuCgo+IFdo YXQgaXMgaGFwcGVuaW5nIHdpdGggeW91ciBkZXZpY2UgdGhhdCBleGlzdGluZyB1ZGV2IHJ1bGUg aXMKPiBnZW5lcmF0aW5nIAo+IGluY29ycmVjdCBzeW1saW5rID8KPiAKPiBUaGF0IHJlYWxseSBu ZWVkcyBjbG9zZSBsb2cgZXZpZGVuY2UgYWJvdXQgc3RhdGVzIG9mIHlvdXIgc3lzdGVtLgoKQmVs aWV2ZSBtZSwgSSBoYXZlIHNjcnV0aW5pemVkIHRoZXNlIGxvZ3MgdmVyeSBjYXJlZnVsbHkuIEkn dmUgYWxyZWFkeQp0b2xkIHlvdSB0aGUgcmVsZXZhbnQgcGFydHMgb2YgaXQuwqBCdXQgYXMgeW91 IGFza2VkIGZvciBpdDoKaHR0cHM6Ly93d3cuZHJvcGJveC5jb20vc2gva3M5NDZhOHdqb3M5N2pp L0FBQVZ0T0szWjNYcGJRN0RGekUxeWthWWE/ZGw9MAoKPiA+ID4gSXQgbG9va3MgbGlrZSB0aGVy ZSBpcyBzb21lIHJhY2UgaW4gdWRldiBydWxlcyBwcm9jZXNzaW5nIC0ganVzdAo+ID4gPiBzb21l d2hlcmUgZWxzZS4KPiA+IEkndmUgcGVyc29uYWxseSBoYWQgYSBwYXJ0IGluIGZpeGluZyBtdWx0 aXBsZSByYWNlIGNvbmRpdGlvbnMgYm90aAo+ID4gbXVsdGlwYXRoZCBhbmQgdWRldi7CoEJlbiBN YXJ6aW5za2kgYW5kIEkgcmVjZW50bHkgd29ya2VkIG9uCj4gPiBkcm9wcGluZwo+ID4gdGhlIGRl cGVuZGVuY3kgb2YgbXVsdGlwYXRoZC5zZXJ2aWNlIG9uIHVkZXYtc2V0dGxlLnNlcnZpY2UuIFRo YXQKPiA+IGFjY2VsbGVyYXRlcyBib290aW5nIGFuZCBpcyBjb25jZXB0dWFsbHkgY2xlYW5lciB0 aGFuIGJlZm9yZSwgYnV0Cj4gPiBpdAo+ID4gcmV2ZWFscyBidWdzIGluIG90aGVyIHBhcnRzIG9m IHRoZSBzeXN0ZW0sIGxpa2UgdGhpcyBvbmUuCj4gCj4gQW5kIHdlIG5lZWQgdG8gZmluZCB0aGUg Zml4IGZvciB0aGUgYnVnIC0gbm90IGp1c3QgbWFzayBpdCBvdXQuCgo+IEFzIHRlY2huaWNhbGx5 IHVkZXYgaXMganVzdCBvbmUgb2YgbWFueSBwb3NzaWJsZSAncmVhZGVycycgb2YgeW91cgo+IGRl dmljZS4gU28gCj4gd2hpbGUgeW91IHdvdWxkIG1hc2sgYnVnIGZvciB1ZGV2IC0gb3RoZXIgZGV2 aWNlIHVzZXJzIHdvdWxkIHN0aWxsCj4gZ2V0IAo+IG1pc2xlYWRpbmcgaW5mbyBhYm91dCB5b3Vy IGRldmljZS4KClRoZSBvbmx5IHRoaW5nIHRoYXQncyBtaXNsZWFkaW5nIGhlcmUgaXMgdGhlIGZh Y3QgdGhhdCB1ZGV2IHJlbW92ZXMgdGhlCmRldmljZSBzeW1saW5rcyBiZWNhdXNlIHRoZSBydWxl cyBmaWxlIGRvbid0IGV4ZWN1dGUgdGhlIFNZTUxJTksKZGlyZWN0aXZlcy4gQWxsIGVsc2UgaXMg IGZpbmUuICJNeSBkZXZpY2UiIGlzIGluIGEgc2FuZSBzdGF0ZS4KCj4gPiBJIHdvdWxkIGFwcHJl Y2lhdGUgaWYgeW91IHNhaWQgd2hhdCdzIGFjdHVhbGx5IHdyb25nIG15IHBhdGNoLiBTbwo+ID4g ZmFyCj4gPiB5b3UgaGF2ZW4ndC4gWW91IGp1c3Qgc3BlY3VsYXRlZCBhYm91dCBlcnJvcnMgaW4g b3RoZXIgcGFydHMgb2YgdGhlCj4gCj4gCj4gQXMgc2FpZCB0aGUgYXV0b21hdGljwqAgJ3NraXAn wqAgb2YgcmVhZCBvZiBzdXNwZW5kIGRldmljZSBjYW4ndCBiZSB0aGUKPiByaWdodCB0aGluZy4K CkJ1dCB0aGF0IGlzIG5vdCB3aGF0IG15IHBhdGNoIGRvZXMsIGFzIEkgYWxyZWFkeSBleHBsYWlu ZWQuIFRoZSByZWFkaW5nCndhcyBza2lwcGVkIGFueXdheSwgYW5kIHRoaXMgaGFzIGJlZW4gc28g c2luY2UgdGhlIGZpbGUgY2FtZSBpbnRvCmV4aXN0ZW5jZS4gUGxlYXNlIGxvb2sgYXQgdGhlIGhp c3Rvcnkgb2YgdGhlIGZpbGUuIEFsbCBJIGFtIGNoYW5naW5nIGlzCnRvIHByZXZlbnQgdGhlIHN5 bWxpbmtzIGZyb20gYmVpbmcgZGVsZXRlZCBwcmVtYXR1cmVseS4KCkFsbCBJIHdhbnQgdG8gYWNo aWV2ZSBoZXJlIGlzIGZpeCBhIG5hc3R5IG11bHRpcGF0aCBib290IGZhaWx1cmUuCkkgdGhpbmsg d2UgYWdyZWUgdGhhdCB0aGUgaGFuZGxpbmcgb2YgZG0gZGV2aWNlcyBieSB1ZGV2IGFuZCBzeXN0 ZW1kIGlzCm5vdCBvcHRpbWFsLiBCdXQgbXkgZ29hbCB3aXRoIHRoaXMgcGF0Y2ggaXMgbm90IHRv IHNhdmUgdGhhdCBnZW5lcmljCnByb2JsZW0uIFRoYXQncyBzb21ldGhpbmcgeW91IG5lZWQgdG8g d29yayBvdXQgd2l0aCBMZW5uYXJ0IGFuZCBoaXMgY28tCndvcmtlcnMuIE15IGVzdGltYXRlIGlz IHRoYXQgYSBwcm9wZXIgc29sdXRpb24gd291bGQgdGFrZSBhIHllYXIgb3IKbW9yZSBvZiBpbnRl bnNpdmUgd29yay4KCj4gCj4gU3VzcGVuZCBpcyBwYXJhbGxlbCZpbnZpc2libGUgb3BlcmF0aW9u IGFuZCBhcyBzdWNowqAgYW55IHJlYWRlciBvZgo+IGRldmljZSBpcyAKPiBtZWFudCB0byB3YWl0 IHRpbGwgZGV2aWNlIGlzIHJlc3VtZWQgYW5kIGZpbmlzaCBpdCdzIHJlYWRpbmcuIFRoYXQncwo+ IHRoZSBjb3JlIAo+IGlkZWEgb2Ygd2hvbGUgRE0gdXNhZ2UgLcKgIGFsbCB1c2VycyBvZiBETSBk ZXZpY2VzIHVzZSBpdCBsaWtlIG5vcm1hbAo+IGJsb2NrIAo+IGRldmljZS7CoCBTbyBpZiB5b3Ug bmVlZCB0byBjaGVjayBzdGF0ZSBvZiBETSBkZXZpY2XCoCAoYW5kIHlvdSBhcmUgbm90Cj4gYSBE TSAKPiBtYWludGFpbmluZyB0b29swqAgYW5kIGJsa2lkIHRvb2wgY2VydGFpbmx5IGlzIG5vdCBz dWNoIHRvb2wpwqAgdGhlbgo+IHRoZXJlIGlzIAo+IHNvbWV0aGluZyB3cm9uZyBpbiBkZXNpZ27C oCAoYWthIHdlIGRvIG5vdCB3YW50IHRvIHN1cHBvcnQgJ3NraXAnIG9mCj4gdWRldiBydWxlcyAK PiBpZiB1c2VyIGlzIHJ1bm5pbmfCoCByYW5kb20gc3RyZWFtwqAgb2YgJ2Rtc2V0dXAgc3VzcGVu ZCAmIGRtc2V0dXAKPiByZXN1bWUnIGNvbW1hbmRzKQoKV2hhdCdzICJ3cm9uZyBpbiBkZXNpZ24i IGhlcmUgaXMgdGhhdCB1ZGV2IGNhbid0IGFmZm9yZCB0byB3YWl0IGZvciBhIApkZXZpY2UgdG8g YmUgcmVzdW1lZC4gSWYgdGhlIGRldmljZSBpcyBzdXNwZW5kZWQsIGJsa2lkIGNhbid0IGJlCmNh bGxlZCwgcGVyaW9kLiBJbiB0aGlzIGNhc2Ugd2UgYXJlIGluIGplb3BhcmR5LCBieSBkZWZpbml0 aW9uLiBXZSBoYXZlCnRvIG1ha2Ugc29tZSBhc3N1bXB0aW9uLiBXZSBjYW4gYXNzdW1lIHRoYXQg dGhlIGJ5LXV1aWQgbGluayBpcyBzdGlsbAp2YWxpZCwgbGlrZSBteSBwYXRjaCBkb2VzLCBhbmQg cmlzayB0aGF0IHRoZSBhc3N1bXB0aW9uIGlzIHdyb25nLiBPcgpzaW1wbHkgZHJvcCB0aGUgc3lt bGluaywgYXMgdGhlIGN1cnJlbnQgY29kZSBkb2VzLCBhbmQgYnJlYWsgc3lzdGVtZCdzCmFzc3Vt cHRpb25zLgoKRGlzY3VzcyB0aGlzIHdpdGggTGVubmFydCwgcGxlYXNlLiBJdCBtZWFucyB0aGF0 IHN5c3RlbWQgbmVlZHMgdG8KY2hhbmdlIGl0J3MgZGV2aWNlIGhhbmRsaW5nIGZ1bmRhbWVudGFs bHkuIEJ1dCByZWFsbHksIHRoaXMgZ29lcyBtdWNoCmZ1cnRoZXIgdGhhbiB3aGF0IEknbSBjdXJy ZW50bHkgaW50ZXJlc3RlZCBpbi4gSSBqdXN0IHdhbnQgdG8gbWFrZSBzdXJlCnRoZXNlIHN5c3Rl bXMgYm9vdCByZWxpYWJseS4KCklmIHlvdSBjYW4ndCB0YWtlIHRoaXMgZm9yIGdlbmVyaWMgZG0s IHdlJ2xsIGhhdmUgdG8gZmluZCBhIHdheSB0byBkbwppdCBpbiB0aGUgbXVsdGlwYXRoIHJ1bGVz LiBUaGlzIGlzIHdoZXJlIHdoZXJlIHdlIHJ1biBJTVBPUlR7ZGJ9IGFueXdheQp0byBhdm9pZCBm YWlsdXJlcy4gSXQgd2lsbCBiZSBmYXIgbGVzcyBlbGVnYW50IGJlY2F1c2UgaXQgbWVhbnMgY29k ZQpkdXBsaWNhdGlvbiAoMTMtZG0tZGlzay5ydWxlcyBpcyB3aGVyZSB0aGlzIGxvZ2ljIGJlbG9u Z3MpLCBidXQgd2VsbCwKSSBndWVzcyB0aGlzIGlzIHRoZSBwcmljZSB3ZSBoYXZlIHRvIHBheSBp ZiB3ZSBjYW4ndCBhZ3JlZS4KClJlZ2FyZHMKTWFydGluCgoKLS0KZG0tZGV2ZWwgbWFpbGluZyBs aXN0CmRtLWRldmVsQHJlZGhhdC5jb20KaHR0cHM6Ly9saXN0bWFuLnJlZGhhdC5jb20vbWFpbG1h bi9saXN0aW5mby9kbS1kZXZlbA== From mboxrd@z Thu Jan 1 00:00:00 1970 From: Martin Wilck Date: Sat, 29 Jan 2022 00:21:00 +0100 Subject: [PATCH] udev: create symlinks and watch even in suspended state In-Reply-To: References: <20220128134229.1783-1-mwilck@suse.com> <10ad6fc0-6c24-c98b-4a02-2140883af72d@gmail.com> <0a55dd1393df2c125f8cb443daaeb7d1b7162bcc.camel@suse.com> Message-ID: <92de9eff521e2702e364f7aa3cce6927d9d9c03c.camel@suse.com> List-Id: To: lvm-devel@redhat.com MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit On Fri, 2022-01-28 at 22:06 +0100, Zdenek Kabelac wrote: > Dne 28. 01. 22 v 19:46 Martin Wilck napsal(a): > > On Fri, 2022-01-28 at 18:47 +0100, Zdenek Kabelac wrote: > > > Dne 28. 01. 22 v 17:02 Martin Wilck napsal(a): > > > > On Fri, 2022-01-28 at 16:57 +0100, Martin Wilck wrote: > > > > > It's a race condition. It probably happens while multipathd > > > > > is > > > > > reloading a map (*), suspending it during the table reload. > > > > > The > > > > > device > > > > > will be resumed a few fractions of a second later (so yes, > > > > > it's > > > > > "quick"), but then it's too late > > > > More precisely: The suspend state itself may actually not last > > > > longer > > > > than a few ms. But once the symlink is bent to point to the > > > > wrong > > > > device, it will remain so, until the CHANGE event following the > > > > device > > > > resume is successfully processed by udev, which may happen > > > > substantially later. And it's that longer time span which > > > > matters > > > > for > > > > systemd's mount attempt (or LVM device activation, for that > > > > matter). > > > > > > > > > > > > > > This looks like you are trying to mask-out different > > > synchronization > > > bug. > > Please explain. What bug would that be, in what subsystem??It takes > > time until a dm-triggered CHANGE event makes its way through the > > udev > > event queue and completes rules processing. That's perfectly > > normal. > > > Well it is at the point we need to know exactly which device in which > state > causes you problem. Then we need to see what is wrong in the whole > process. This is how multipathd loads the multipath map before the error occurs: [ 127.352614] localhost multipathd[1059]: 3600a098000aad1e300000b4b5a275d45: reload [0 134217728 multipath 3 pg_init_retries 50 queue_if_no_path 1 alua 2 1 service-time 0 1 1 8:80 1 service-time 0 2 1 8:16 1 8:144 1] The mapping of the partition device is this (after boot): # dmsetup table /dev/dm-13 0 133953503 linear 254:10 264192 # dmsetup ls --tree # excerpt 3600a098000aad1e300000b4b5a275d45-part2 (254:13) ??3600a098000aad1e300000b4b5a275d45 (254:10) ?? (8:144) ?? (8:16) ?? (8:80) Here you can see the links that udev creates for the device when it's set up during initrd processing (note the by-uuid link): [ 40.611989] localhost systemd-udevd[555]: dm-13: Handling device node '/dev/dm-13', devnum=b254:13 [ 40.612073] localhost systemd-udevd[555]: dm-13: Creating symlink '/dev/mapper/3600a098000aad1e300000b4b5a275d45-part2' to '../dm-13' [ 40.612374] localhost systemd-udevd[555]: dm-13: Atomically replace '/dev/disk/by-id/wwn-0x600a098000aad1e300000b4b5a275d45-part2' [ 40.612476] localhost systemd-udevd[555]: dm-13: Creating symlink '/dev/disk/by-id/dm-uuid-part2-mpath-3600a098000aad1e300000b4b5a275d45' to '../../dm-13' [ 40.612827] localhost systemd-udevd[555]: dm-13: Atomically replace '/dev/disk/by-partuuid/1c2f70e0-fb91-49f5-8260-38eacaf7992b' [ 40.612980] localhost systemd-udevd[555]: dm-13: Creating symlink '/dev/disk/by-id/dm-name-3600a098000aad1e300000b4b5a275d45-part2' to '../../dm-13' [ 40.613222] localhost systemd-udevd[555]: dm-13: Atomically replace '/dev/disk/by-id/scsi-3600a098000aad1e300000b4b5a275d45-part2' [ 40.613502] localhost systemd-udevd[555]: dm-13: Atomically replace '/dev/disk/by-uuid/e40d3005-ab2f-4845-bd83-be5fd09e62a0' [ 40.613531] localhost systemd-udevd[555]: dm-13: Preserve already existing symlink '/dev/block/254:13' to '../dm-13' The by-uuid link is then removed after switching root as I showed in my previous post. The reason is that the respective SYMLINK line in 13-dm-disk.rules is skipped for suspended devices, and my patch fixes that. At the time this happens, the same partiton _is already mounted_ as btrfs root file system. This file system has multiple subvolumes for /tmp, /var, /usr/local, etc., which need to be mounted after switching root. The problem occurs when these mounts are attempted. Typically some succeed, and some don't, depending on the timing. > All I'm saying here is - that the proposed patch is not fixing bug - > but > rather masking/minimizing window for error. AFAICS my patch eliminates the window for this error entirely. > > I say the bug is in this udev rule file. The set of properties and > > symlinks of a systemd-managed device must not change when new > > uevents > > for this device are processed, unless absolutely necessary. The > > current > > > The question is - whether the event is meant to be there if there is > no > change, that should be reflected in the system. It really goes to #1 > question > about detailed state of DM tables and your observed blocked system.? > With lvm2 > we are carefull about sending events. The coldplug "add" event always happens. It can't be avoided. It is necessary to make user-space aware of the device after switching root. And device-mapper rules go into great detail how they deal with it, as you are certainly aware. > > > rule violates this principle: If a device that contains a file > > system > > is suspended, it continues to contain this file system, and the by- > > uuid > > symlink for the file system should be preserved. > > You can easily test this. Check the set of symlinks for a partition > > on > > a multipath device, suspend the device, run "udevadm trigger -c > > add" on > > the device (simulating coldplug), and check the set of symlinks > > again. > > Whoever reads suspended device has to wait for 'resume' I suppose if mount(8) opened /dev/dm-13 directly, it would just hang while the device was suspended, and start IO afterwards. But this isn't how this works. mount(8) opens the /dev/disk/by-uuid symlink which now points to the underlying device (sdb2 - udev bent the symlink to point to sdb2 after it was dropped from dm-13). Opening sdb2 doesn't hang, it fails with -EBUSY because the device is locked by dm. Thus the mount fails. mount(8) has no way to detect why this is so. systemd might, and might attempt a retry, but it lacks the logic for doing that. > > - so the test case > where you suspend device for long period of time and you observe > other > processed waiting for device to be resumed is wanted behavior - it's > not valid > to have? suspended devices in the system.?? Unfortunately dm devices are often suspended for a short period of time, when their table is reloaded. As I pointed out before, it's normal for this to happen during multipath startup, when paths are added or removed, priority groups switched, etc. > You have state 'before' suspend? > and state after 'resume'. ? There is no state for 'suspended' type of > device > (as the device could be resumed either with exiting table or a new > table. So: the mapping could be entirely different when the device is resumed, and if systemd tried to mount it in the time span between the resume and the finish of the uevent that follows, it might mount a totally different file system. Is that what you mean? If yes, I suppose you're right. But isn't that possible today already? Couldn't I reload a mapping with totally different devices and offsets, while it is mounted? At least for multipath, that's extremely unlikely to happen, whereas quick reloads with a similar mapping that doesn't affect upper layers are very common. If you take the above seriously, you need to rewrite your code for communicating with udev and systemd. I guess systemd itself would need to change the way it handles dm devices, fundamentally. Basically you would need to tell systemd to consider a suspended device as unusable, and not make it available to the system again before it's resumed in a state that's verified to be compatible with the previous state. And we'd need deal with the special caveats for multipath, too. > So any simulation relaying on suspended device is basically testing > invalid > state of system. > > And I'm assuming your problem is something else - as I'd say you > don't leak > forgotten 'suspended' device in ramdisk ? No, everything is perfectly fine in the initrd. And afterwards, all is fine, too - except if, by very bad luck, the timing is such that? a) the coldplug "add" event happens?while the device is suspended (as I said, probably due to a reload operation) b) systemd tries to mount the file system during the short time span in which the symlinks are missing (i.e. between the resume and the final processing of the CHANGE event). In my testing, this happens only on one system, roughly once every 50 reboots. > > > You'll see that by-uuid or by-label links will be lost and will now > > point to one of the map's paths. These symlinks are crucial for > > mounting file systems. With my patch, the symlinks are preserved. > > > I'm still missing why 'UUID' should be lost ? The link is bound to udev's ID_PART_ENTRY_UUID property. This property is set by IMPORT{buidltin}="blkid", or using IMPORT{db}="ID_PART_ENTRY_UUID". The latter is done by 11-dm- mpath.rules and 11-dm-parts.rules, for example, exactly for the same reason - we can't risk loosing the symlink because blkid fails. This works nicely, unless DM_SUSPENDED or DM_NOSCAN is set, because in this case 13-dm-disk.rules won't call SYMLINK. > Where they missing before suspend ? No. > Are they lost after resume ? No, after resume udev reinstates them, but then it's too late. > What is causing your UUID to be lost ? The UUID is never lost. Only the symlink that reflects it. > > To connect this with lvm2 logic -? when lvm2 creates or deletes > device - it > always waits for udev confirmation? (ATM via cookie semaphore). So > i.e. we > can't create a situation where we would be triggering? 'udev' disk > creation > event that would be 'racing' with? udev disk removal processing - so > there is > close synchronization to avoid these kind of races where udev would > be > accessing? our LV devices in some? async parallel logic and possibly > would > have 'modifed' its symlink database with some obsolete info. multipathd does that, too. But it doesn't apply here. Because the device is live (system has mounted it in the initramfs already and it's actually mounted at the time all this happens, see above). Yet the coldplug event does arrive, we can't avoid it. (Well we could, at least partly, by delaying multipathd startup until after "udev settle" again, but I'm sure Ben agrees with me that we'd rather not do that). > > > > > Also it's worth to note - using symlinks is somewhat doomed on > > > its > > > own. > > That's how device identification and activation with udev and > > systemd > > works. It won't change any time soon. Explain on which grounds > > you're > > calling it "somewhat doomed". > > You could find my discussion with Lennart? - one of major design > problem of > symlinks is it can't handle duplicate devices with same IDs.? So if > you have > multiple disks with same identification - symlinks will be randomly > switching > between them -? so plain simple 'dd? if=diskA of=diskB' ruins > symlinks in your > system. udev has it's link priority concept for this reason. It may not be perfect but it works quite well, and the symlink handling has improved a lot lately. I am curious what your alternative concept is. Do you have a link? > [...] > > How could you get wrong UUID for already mounted (so likely also > opened) device?? Why do you say "wrong UUID" ? The UUID was correct all the time. I explained above what happened. Yes, the device was mounted (to other btrfs subvolumes) when the error occured. > What is happening with your device that existing udev rule is > generating > incorrect symlink ? > > That really needs close log evidence about states of your system. Believe me, I have scrutinized these logs very carefully. I've already told you the relevant parts of it.?But as you asked for it: https://www.dropbox.com/sh/ks946a8wjos97ji/AAAVtOK3Z3XpbQ7DFzE1ykaYa?dl=0 > > > It looks like there is some race in udev rules processing - just > > > somewhere else. > > I've personally had a part in fixing multiple race conditions both > > multipathd and udev.?Ben Marzinski and I recently worked on > > dropping > > the dependency of multipathd.service on udev-settle.service. That > > accellerates booting and is conceptually cleaner than before, but > > it > > reveals bugs in other parts of the system, like this one. > > And we need to find the fix for the bug - not just mask it out. > As technically udev is just one of many possible 'readers' of your > device. So > while you would mask bug for udev - other device users would still > get > misleading info about your device. The only thing that's misleading here is the fact that udev removes the device symlinks because the rules file don't execute the SYMLINK directives. All else is fine. "My device" is in a sane state. > > I would appreciate if you said what's actually wrong my patch. So > > far > > you haven't. You just speculated about errors in other parts of the > > > As said the automatic? 'skip'? of read of suspend device can't be the > right thing. But that is not what my patch does, as I already explained. The reading was skipped anyway, and this has been so since the file came into existence. Please look at the history of the file. All I am changing is to prevent the symlinks from being deleted prematurely. All I want to achieve here is fix a nasty multipath boot failure. I think we agree that the handling of dm devices by udev and systemd is not optimal. But my goal with this patch is not to save that generic problem. That's something you need to work out with Lennart and his co- workers. My estimate is that a proper solution would take a year or more of intensive work. > > Suspend is parallel&invisible operation and as such? any reader of > device is > meant to wait till device is resumed and finish it's reading. That's > the core > idea of whole DM usage -? all users of DM devices use it like normal > block > device.? So if you need to check state of DM device? (and you are not > a DM > maintaining tool? and blkid tool certainly is not such tool)? then > there is > something wrong in design? (aka we do not want to support 'skip' of > udev rules > if user is running? random stream? of 'dmsetup suspend & dmsetup > resume' commands) What's "wrong in design" here is that udev can't afford to wait for a device to be resumed. If the device is suspended, blkid can't be called, period. In this case we are in jeopardy, by definition. We have to make some assumption. We can assume that the by-uuid link is still valid, like my patch does, and risk that the assumption is wrong. Or simply drop the symlink, as the current code does, and break systemd's assumptions. Discuss this with Lennart, please. It means that systemd needs to change it's device handling fundamentally. But really, this goes much further than what I'm currently interested in. I just want to make sure these systems boot reliably. If you can't take this for generic dm, we'll have to find a way to do it in the multipath rules. This is where where we run IMPORT{db} anyway to avoid failures. It will be far less elegant because it means code duplication (13-dm-disk.rules is where this logic belongs), but well, I guess this is the price we have to pay if we can't agree. Regards Martin