From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43E26C43387 for ; Thu, 27 Dec 2018 10:00:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id E92DF214C6 for ; Thu, 27 Dec 2018 10:00:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730699AbeL0KAw (ORCPT ); Thu, 27 Dec 2018 05:00:52 -0500 Received: from mx1.redhat.com ([209.132.183.28]:37536 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727926AbeL0KAw (ORCPT ); Thu, 27 Dec 2018 05:00:52 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 7615BB0824; Thu, 27 Dec 2018 10:00:51 +0000 (UTC) Received: from [10.72.12.191] (ovpn-12-191.pek2.redhat.com [10.72.12.191]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 1E81A60C66; Thu, 27 Dec 2018 10:00:38 +0000 (UTC) Subject: Re: [PATCH RFC 1/2] virtio-net: bql support To: "Michael S. Tsirkin" Cc: linux-kernel@vger.kernel.org, maxime.coquelin@redhat.com, tiwei.bie@intel.com, wexu@redhat.com, jfreimann@redhat.com, "David S. Miller" , virtualization@lists.linux-foundation.org, netdev@vger.kernel.org References: <20181205225323.12555-1-mst@redhat.com> <20181205225323.12555-2-mst@redhat.com> <21384cb5-99a6-7431-1039-b356521e1bc3@redhat.com> <20181226101528-mutt-send-email-mst@kernel.org> From: Jason Wang Message-ID: <0fa99d9b-e510-d7eb-db1b-831bd7610ce9@redhat.com> Date: Thu, 27 Dec 2018 18:00:36 +0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: <20181226101528-mutt-send-email-mst@kernel.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Thu, 27 Dec 2018 10:00:51 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2018/12/26 下午11:19, Michael S. Tsirkin wrote: > On Thu, Dec 06, 2018 at 04:17:36PM +0800, Jason Wang wrote: >> On 2018/12/6 上午6:54, Michael S. Tsirkin wrote: >>> When use_napi is set, let's enable BQLs. Note: some of the issues are >>> similar to wifi. It's worth considering whether something similar to >>> commit 36148c2bbfbe ("mac80211: Adjust TSQ pacing shift") might be >>> benefitial. >> >> I've played a similar patch several days before. The tricky part is the mode >> switching between napi and no napi. We should make sure when the packet is >> sent and trakced by BQL,  it should be consumed by BQL as well. I did it by >> tracking it through skb->cb.  And deal with the freeze by reset the BQL >> status. Patch attached. >> >> But when testing with vhost-net, I don't very a stable performance, > So how about increasing TSQ pacing shift then? I can test this. But changing default TCP value is much more than a virtio-net specific thing. > >> it was >> probably because we batch the used ring updating so tx interrupt may come >> randomly. We probably need to implement time bounded coalescing mechanism >> which could be configured from userspace. > I don't think it's reasonable to expect userspace to be that smart ... > Why do we need time bounded? used ring is always updated when ring > becomes empty. We don't add used when means BQL may not see the consumed packet in time. And the delay varies based on the workload since we count packets not bytes or time before doing the batched updating. Thanks > >> Btw, maybe it's time just enable napi TX by default. I get ~10% TCP_RR >> regression on machine without APICv, (haven't found time to test APICv >> machine). But consider it was for correctness, I think it's acceptable? Then >> we can do optimization on top? >> >> >> Thanks >> >> >>> Signed-off-by: Michael S. Tsirkin >>> --- >>> drivers/net/virtio_net.c | 27 +++++++++++++++++++-------- >>> 1 file changed, 19 insertions(+), 8 deletions(-) >>> >>> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c >>> index cecfd77c9f3c..b657bde6b94b 100644 >>> --- a/drivers/net/virtio_net.c >>> +++ b/drivers/net/virtio_net.c >>> @@ -1325,7 +1325,8 @@ static int virtnet_receive(struct receive_queue *rq, int budget, >>> return stats.packets; >>> } >>> -static void free_old_xmit_skbs(struct send_queue *sq) >>> +static void free_old_xmit_skbs(struct send_queue *sq, struct netdev_queue *txq, >>> + bool use_napi) >>> { >>> struct sk_buff *skb; >>> unsigned int len; >>> @@ -1347,6 +1348,9 @@ static void free_old_xmit_skbs(struct send_queue *sq) >>> if (!packets) >>> return; >>> + if (use_napi) >>> + netdev_tx_completed_queue(txq, packets, bytes); >>> + >>> u64_stats_update_begin(&sq->stats.syncp); >>> sq->stats.bytes += bytes; >>> sq->stats.packets += packets; >>> @@ -1364,7 +1368,7 @@ static void virtnet_poll_cleantx(struct receive_queue *rq) >>> return; >>> if (__netif_tx_trylock(txq)) { >>> - free_old_xmit_skbs(sq); >>> + free_old_xmit_skbs(sq, txq, true); >>> __netif_tx_unlock(txq); >>> } >>> @@ -1440,7 +1444,7 @@ static int virtnet_poll_tx(struct napi_struct *napi, int budget) >>> struct netdev_queue *txq = netdev_get_tx_queue(vi->dev, vq2txq(sq->vq)); >>> __netif_tx_lock(txq, raw_smp_processor_id()); >>> - free_old_xmit_skbs(sq); >>> + free_old_xmit_skbs(sq, txq, true); >>> __netif_tx_unlock(txq); >>> virtqueue_napi_complete(napi, sq->vq, 0); >>> @@ -1505,13 +1509,15 @@ static netdev_tx_t start_xmit(struct sk_buff *skb, struct net_device *dev) >>> struct send_queue *sq = &vi->sq[qnum]; >>> int err; >>> struct netdev_queue *txq = netdev_get_tx_queue(dev, qnum); >>> - bool kick = !skb->xmit_more; >>> + bool more = skb->xmit_more; >>> bool use_napi = sq->napi.weight; >>> + unsigned int bytes = skb->len; >>> + bool kick; >>> /* Free up any pending old buffers before queueing new ones. */ >>> - free_old_xmit_skbs(sq); >>> + free_old_xmit_skbs(sq, txq, use_napi); >>> - if (use_napi && kick) >>> + if (use_napi && !more) >>> virtqueue_enable_cb_delayed(sq->vq); >>> /* timestamp packet in software */ >>> @@ -1552,7 +1558,7 @@ static netdev_tx_t start_xmit(struct sk_buff *skb, struct net_device *dev) >>> if (!use_napi && >>> unlikely(!virtqueue_enable_cb_delayed(sq->vq))) { >>> /* More just got used, free them then recheck. */ >>> - free_old_xmit_skbs(sq); >>> + free_old_xmit_skbs(sq, txq, false); >>> if (sq->vq->num_free >= 2+MAX_SKB_FRAGS) { >>> netif_start_subqueue(dev, qnum); >>> virtqueue_disable_cb(sq->vq); >>> @@ -1560,7 +1566,12 @@ static netdev_tx_t start_xmit(struct sk_buff *skb, struct net_device *dev) >>> } >>> } >>> - if (kick || netif_xmit_stopped(txq)) { >>> + if (use_napi) >>> + kick = __netdev_tx_sent_queue(txq, bytes, more); >>> + else >>> + kick = !more || netif_xmit_stopped(txq); >>> + >>> + if (kick) { >>> if (virtqueue_kick_prepare(sq->vq) && virtqueue_notify(sq->vq)) { >>> u64_stats_update_begin(&sq->stats.syncp); >>> sq->stats.kicks++; From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jason Wang Subject: Re: [PATCH RFC 1/2] virtio-net: bql support Date: Thu, 27 Dec 2018 18:00:36 +0800 Message-ID: <0fa99d9b-e510-d7eb-db1b-831bd7610ce9@redhat.com> References: <20181205225323.12555-1-mst@redhat.com> <20181205225323.12555-2-mst@redhat.com> <21384cb5-99a6-7431-1039-b356521e1bc3@redhat.com> <20181226101528-mutt-send-email-mst@kernel.org> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8"; Format="flowed" Content-Transfer-Encoding: base64 Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, maxime.coquelin@redhat.com, wexu@redhat.com, "David S. Miller" To: "Michael S. Tsirkin" Return-path: In-Reply-To: <20181226101528-mutt-send-email-mst@kernel.org> Content-Language: en-US List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: virtualization-bounces@lists.linux-foundation.org Errors-To: virtualization-bounces@lists.linux-foundation.org List-Id: netdev.vger.kernel.org Ck9uIDIwMTgvMTIvMjYg5LiL5Y2IMTE6MTksIE1pY2hhZWwgUy4gVHNpcmtpbiB3cm90ZToKPiBP biBUaHUsIERlYyAwNiwgMjAxOCBhdCAwNDoxNzozNlBNICswODAwLCBKYXNvbiBXYW5nIHdyb3Rl Ogo+PiBPbiAyMDE4LzEyLzYg5LiK5Y2INjo1NCwgTWljaGFlbCBTLiBUc2lya2luIHdyb3RlOgo+ Pj4gV2hlbiB1c2VfbmFwaSBpcyBzZXQsIGxldCdzIGVuYWJsZSBCUUxzLiAgTm90ZTogc29tZSBv ZiB0aGUgaXNzdWVzIGFyZQo+Pj4gc2ltaWxhciB0byB3aWZpLiAgSXQncyB3b3J0aCBjb25zaWRl cmluZyB3aGV0aGVyIHNvbWV0aGluZyBzaW1pbGFyIHRvCj4+PiBjb21taXQgMzYxNDhjMmJiZmJl ICgibWFjODAyMTE6IEFkanVzdCBUU1EgcGFjaW5nIHNoaWZ0IikgbWlnaHQgYmUKPj4+IGJlbmVm aXRpYWwuCj4+Cj4+IEkndmUgcGxheWVkIGEgc2ltaWxhciBwYXRjaCBzZXZlcmFsIGRheXMgYmVm b3JlLiBUaGUgdHJpY2t5IHBhcnQgaXMgdGhlIG1vZGUKPj4gc3dpdGNoaW5nIGJldHdlZW4gbmFw aSBhbmQgbm8gbmFwaS4gV2Ugc2hvdWxkIG1ha2Ugc3VyZSB3aGVuIHRoZSBwYWNrZXQgaXMKPj4g c2VudCBhbmQgdHJha2NlZCBieSBCUUwswqAgaXQgc2hvdWxkIGJlIGNvbnN1bWVkIGJ5IEJRTCBh cyB3ZWxsLiBJIGRpZCBpdCBieQo+PiB0cmFja2luZyBpdCB0aHJvdWdoIHNrYi0+Y2IuwqAgQW5k IGRlYWwgd2l0aCB0aGUgZnJlZXplIGJ5IHJlc2V0IHRoZSBCUUwKPj4gc3RhdHVzLiBQYXRjaCBh dHRhY2hlZC4KPj4KPj4gQnV0IHdoZW4gdGVzdGluZyB3aXRoIHZob3N0LW5ldCwgSSBkb24ndCB2 ZXJ5IGEgc3RhYmxlIHBlcmZvcm1hbmNlLAo+IFNvIGhvdyBhYm91dCBpbmNyZWFzaW5nIFRTUSBw YWNpbmcgc2hpZnQgdGhlbj8KCgpJIGNhbiB0ZXN0IHRoaXMuIEJ1dCBjaGFuZ2luZyBkZWZhdWx0 IFRDUCB2YWx1ZSBpcyBtdWNoIG1vcmUgdGhhbiBhIAp2aXJ0aW8tbmV0IHNwZWNpZmljIHRoaW5n LgoKCj4KPj4gaXQgd2FzCj4+IHByb2JhYmx5IGJlY2F1c2Ugd2UgYmF0Y2ggdGhlIHVzZWQgcmlu ZyB1cGRhdGluZyBzbyB0eCBpbnRlcnJ1cHQgbWF5IGNvbWUKPj4gcmFuZG9tbHkuIFdlIHByb2Jh Ymx5IG5lZWQgdG8gaW1wbGVtZW50IHRpbWUgYm91bmRlZCBjb2FsZXNjaW5nIG1lY2hhbmlzbQo+ PiB3aGljaCBjb3VsZCBiZSBjb25maWd1cmVkIGZyb20gdXNlcnNwYWNlLgo+IEkgZG9uJ3QgdGhp bmsgaXQncyByZWFzb25hYmxlIHRvIGV4cGVjdCB1c2Vyc3BhY2UgdG8gYmUgdGhhdCBzbWFydCAu Li4KPiBXaHkgZG8gd2UgbmVlZCB0aW1lIGJvdW5kZWQ/IHVzZWQgcmluZyBpcyBhbHdheXMgdXBk YXRlZCB3aGVuIHJpbmcKPiBiZWNvbWVzIGVtcHR5LgoKCldlIGRvbid0IGFkZCB1c2VkIHdoZW4g bWVhbnMgQlFMIG1heSBub3Qgc2VlIHRoZSBjb25zdW1lZCBwYWNrZXQgaW4gCnRpbWUuIEFuZCB0 aGUgZGVsYXkgdmFyaWVzIGJhc2VkIG9uIHRoZSB3b3JrbG9hZCBzaW5jZSB3ZSBjb3VudCBwYWNr ZXRzIApub3QgYnl0ZXMgb3IgdGltZSBiZWZvcmUgZG9pbmcgdGhlIGJhdGNoZWQgdXBkYXRpbmcu CgpUaGFua3MKCgo+Cj4+IEJ0dywgbWF5YmUgaXQncyB0aW1lIGp1c3QgZW5hYmxlIG5hcGkgVFgg YnkgZGVmYXVsdC4gSSBnZXQgfjEwJSBUQ1BfUlIKPj4gcmVncmVzc2lvbiBvbiBtYWNoaW5lIHdp dGhvdXQgQVBJQ3YsIChoYXZlbid0IGZvdW5kIHRpbWUgdG8gdGVzdCBBUElDdgo+PiBtYWNoaW5l KS4gQnV0IGNvbnNpZGVyIGl0IHdhcyBmb3IgY29ycmVjdG5lc3MsIEkgdGhpbmsgaXQncyBhY2Nl cHRhYmxlPyBUaGVuCj4+IHdlIGNhbiBkbyBvcHRpbWl6YXRpb24gb24gdG9wPwo+Pgo+Pgo+PiBU aGFua3MKPj4KPj4KPj4+IFNpZ25lZC1vZmYtYnk6IE1pY2hhZWwgUy4gVHNpcmtpbiA8bXN0QHJl ZGhhdC5jb20+Cj4+PiAtLS0KPj4+ICAgIGRyaXZlcnMvbmV0L3ZpcnRpb19uZXQuYyB8IDI3ICsr KysrKysrKysrKysrKysrKystLS0tLS0tLQo+Pj4gICAgMSBmaWxlIGNoYW5nZWQsIDE5IGluc2Vy dGlvbnMoKyksIDggZGVsZXRpb25zKC0pCj4+Pgo+Pj4gZGlmZiAtLWdpdCBhL2RyaXZlcnMvbmV0 L3ZpcnRpb19uZXQuYyBiL2RyaXZlcnMvbmV0L3ZpcnRpb19uZXQuYwo+Pj4gaW5kZXggY2VjZmQ3 N2M5ZjNjLi5iNjU3YmRlNmI5NGIgMTAwNjQ0Cj4+PiAtLS0gYS9kcml2ZXJzL25ldC92aXJ0aW9f bmV0LmMKPj4+ICsrKyBiL2RyaXZlcnMvbmV0L3ZpcnRpb19uZXQuYwo+Pj4gQEAgLTEzMjUsNyAr MTMyNSw4IEBAIHN0YXRpYyBpbnQgdmlydG5ldF9yZWNlaXZlKHN0cnVjdCByZWNlaXZlX3F1ZXVl ICpycSwgaW50IGJ1ZGdldCwKPj4+ICAgIAlyZXR1cm4gc3RhdHMucGFja2V0czsKPj4+ICAgIH0K Pj4+IC1zdGF0aWMgdm9pZCBmcmVlX29sZF94bWl0X3NrYnMoc3RydWN0IHNlbmRfcXVldWUgKnNx KQo+Pj4gK3N0YXRpYyB2b2lkIGZyZWVfb2xkX3htaXRfc2ticyhzdHJ1Y3Qgc2VuZF9xdWV1ZSAq c3EsIHN0cnVjdCBuZXRkZXZfcXVldWUgKnR4cSwKPj4+ICsJCQkgICAgICAgYm9vbCB1c2VfbmFw aSkKPj4+ICAgIHsKPj4+ICAgIAlzdHJ1Y3Qgc2tfYnVmZiAqc2tiOwo+Pj4gICAgCXVuc2lnbmVk IGludCBsZW47Cj4+PiBAQCAtMTM0Nyw2ICsxMzQ4LDkgQEAgc3RhdGljIHZvaWQgZnJlZV9vbGRf eG1pdF9za2JzKHN0cnVjdCBzZW5kX3F1ZXVlICpzcSkKPj4+ICAgIAlpZiAoIXBhY2tldHMpCj4+ PiAgICAJCXJldHVybjsKPj4+ICsJaWYgKHVzZV9uYXBpKQo+Pj4gKwkJbmV0ZGV2X3R4X2NvbXBs ZXRlZF9xdWV1ZSh0eHEsIHBhY2tldHMsIGJ5dGVzKTsKPj4+ICsKPj4+ICAgIAl1NjRfc3RhdHNf dXBkYXRlX2JlZ2luKCZzcS0+c3RhdHMuc3luY3ApOwo+Pj4gICAgCXNxLT5zdGF0cy5ieXRlcyAr PSBieXRlczsKPj4+ICAgIAlzcS0+c3RhdHMucGFja2V0cyArPSBwYWNrZXRzOwo+Pj4gQEAgLTEz NjQsNyArMTM2OCw3IEBAIHN0YXRpYyB2b2lkIHZpcnRuZXRfcG9sbF9jbGVhbnR4KHN0cnVjdCBy ZWNlaXZlX3F1ZXVlICpycSkKPj4+ICAgIAkJcmV0dXJuOwo+Pj4gICAgCWlmIChfX25ldGlmX3R4 X3RyeWxvY2sodHhxKSkgewo+Pj4gLQkJZnJlZV9vbGRfeG1pdF9za2JzKHNxKTsKPj4+ICsJCWZy ZWVfb2xkX3htaXRfc2ticyhzcSwgdHhxLCB0cnVlKTsKPj4+ICAgIAkJX19uZXRpZl90eF91bmxv Y2sodHhxKTsKPj4+ICAgIAl9Cj4+PiBAQCAtMTQ0MCw3ICsxNDQ0LDcgQEAgc3RhdGljIGludCB2 aXJ0bmV0X3BvbGxfdHgoc3RydWN0IG5hcGlfc3RydWN0ICpuYXBpLCBpbnQgYnVkZ2V0KQo+Pj4g ICAgCXN0cnVjdCBuZXRkZXZfcXVldWUgKnR4cSA9IG5ldGRldl9nZXRfdHhfcXVldWUodmktPmRl diwgdnEydHhxKHNxLT52cSkpOwo+Pj4gICAgCV9fbmV0aWZfdHhfbG9jayh0eHEsIHJhd19zbXBf cHJvY2Vzc29yX2lkKCkpOwo+Pj4gLQlmcmVlX29sZF94bWl0X3NrYnMoc3EpOwo+Pj4gKwlmcmVl X29sZF94bWl0X3NrYnMoc3EsIHR4cSwgdHJ1ZSk7Cj4+PiAgICAJX19uZXRpZl90eF91bmxvY2so dHhxKTsKPj4+ICAgIAl2aXJ0cXVldWVfbmFwaV9jb21wbGV0ZShuYXBpLCBzcS0+dnEsIDApOwo+ Pj4gQEAgLTE1MDUsMTMgKzE1MDksMTUgQEAgc3RhdGljIG5ldGRldl90eF90IHN0YXJ0X3htaXQo c3RydWN0IHNrX2J1ZmYgKnNrYiwgc3RydWN0IG5ldF9kZXZpY2UgKmRldikKPj4+ICAgIAlzdHJ1 Y3Qgc2VuZF9xdWV1ZSAqc3EgPSAmdmktPnNxW3FudW1dOwo+Pj4gICAgCWludCBlcnI7Cj4+PiAg ICAJc3RydWN0IG5ldGRldl9xdWV1ZSAqdHhxID0gbmV0ZGV2X2dldF90eF9xdWV1ZShkZXYsIHFu dW0pOwo+Pj4gLQlib29sIGtpY2sgPSAhc2tiLT54bWl0X21vcmU7Cj4+PiArCWJvb2wgbW9yZSA9 IHNrYi0+eG1pdF9tb3JlOwo+Pj4gICAgCWJvb2wgdXNlX25hcGkgPSBzcS0+bmFwaS53ZWlnaHQ7 Cj4+PiArCXVuc2lnbmVkIGludCBieXRlcyA9IHNrYi0+bGVuOwo+Pj4gKwlib29sIGtpY2s7Cj4+ PiAgICAJLyogRnJlZSB1cCBhbnkgcGVuZGluZyBvbGQgYnVmZmVycyBiZWZvcmUgcXVldWVpbmcg bmV3IG9uZXMuICovCj4+PiAtCWZyZWVfb2xkX3htaXRfc2ticyhzcSk7Cj4+PiArCWZyZWVfb2xk X3htaXRfc2ticyhzcSwgdHhxLCB1c2VfbmFwaSk7Cj4+PiAtCWlmICh1c2VfbmFwaSAmJiBraWNr KQo+Pj4gKwlpZiAodXNlX25hcGkgJiYgIW1vcmUpCj4+PiAgICAJCXZpcnRxdWV1ZV9lbmFibGVf Y2JfZGVsYXllZChzcS0+dnEpOwo+Pj4gICAgCS8qIHRpbWVzdGFtcCBwYWNrZXQgaW4gc29mdHdh cmUgKi8KPj4+IEBAIC0xNTUyLDcgKzE1NTgsNyBAQCBzdGF0aWMgbmV0ZGV2X3R4X3Qgc3RhcnRf eG1pdChzdHJ1Y3Qgc2tfYnVmZiAqc2tiLCBzdHJ1Y3QgbmV0X2RldmljZSAqZGV2KQo+Pj4gICAg CQlpZiAoIXVzZV9uYXBpICYmCj4+PiAgICAJCSAgICB1bmxpa2VseSghdmlydHF1ZXVlX2VuYWJs ZV9jYl9kZWxheWVkKHNxLT52cSkpKSB7Cj4+PiAgICAJCQkvKiBNb3JlIGp1c3QgZ290IHVzZWQs IGZyZWUgdGhlbSB0aGVuIHJlY2hlY2suICovCj4+PiAtCQkJZnJlZV9vbGRfeG1pdF9za2JzKHNx KTsKPj4+ICsJCQlmcmVlX29sZF94bWl0X3NrYnMoc3EsIHR4cSwgZmFsc2UpOwo+Pj4gICAgCQkJ aWYgKHNxLT52cS0+bnVtX2ZyZWUgPj0gMitNQVhfU0tCX0ZSQUdTKSB7Cj4+PiAgICAJCQkJbmV0 aWZfc3RhcnRfc3VicXVldWUoZGV2LCBxbnVtKTsKPj4+ICAgIAkJCQl2aXJ0cXVldWVfZGlzYWJs ZV9jYihzcS0+dnEpOwo+Pj4gQEAgLTE1NjAsNyArMTU2NiwxMiBAQCBzdGF0aWMgbmV0ZGV2X3R4 X3Qgc3RhcnRfeG1pdChzdHJ1Y3Qgc2tfYnVmZiAqc2tiLCBzdHJ1Y3QgbmV0X2RldmljZSAqZGV2 KQo+Pj4gICAgCQl9Cj4+PiAgICAJfQo+Pj4gLQlpZiAoa2ljayB8fCBuZXRpZl94bWl0X3N0b3Bw ZWQodHhxKSkgewo+Pj4gKwlpZiAodXNlX25hcGkpCj4+PiArCQlraWNrID0gX19uZXRkZXZfdHhf c2VudF9xdWV1ZSh0eHEsIGJ5dGVzLCBtb3JlKTsKPj4+ICsJZWxzZQo+Pj4gKwkJa2ljayA9ICFt b3JlIHx8IG5ldGlmX3htaXRfc3RvcHBlZCh0eHEpOwo+Pj4gKwo+Pj4gKwlpZiAoa2ljaykgewo+ Pj4gICAgCQlpZiAodmlydHF1ZXVlX2tpY2tfcHJlcGFyZShzcS0+dnEpICYmIHZpcnRxdWV1ZV9u b3RpZnkoc3EtPnZxKSkgewo+Pj4gICAgCQkJdTY0X3N0YXRzX3VwZGF0ZV9iZWdpbigmc3EtPnN0 YXRzLnN5bmNwKTsKPj4+ICAgIAkJCXNxLT5zdGF0cy5raWNrcysrOwpfX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpWaXJ0dWFsaXphdGlvbiBtYWlsaW5nIGxp c3QKVmlydHVhbGl6YXRpb25AbGlzdHMubGludXgtZm91bmRhdGlvbi5vcmcKaHR0cHM6Ly9saXN0 cy5saW51eGZvdW5kYXRpb24ub3JnL21haWxtYW4vbGlzdGluZm8vdmlydHVhbGl6YXRpb24=