From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45808C43387 for ; Thu, 27 Dec 2018 09:57:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 133D2214AE for ; Thu, 27 Dec 2018 09:57:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730682AbeL0J5F (ORCPT ); Thu, 27 Dec 2018 04:57:05 -0500 Received: from mx1.redhat.com ([209.132.183.28]:37442 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730555AbeL0J5E (ORCPT ); Thu, 27 Dec 2018 04:57:04 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id C85B8C050E00; Thu, 27 Dec 2018 09:57:03 +0000 (UTC) Received: from [10.72.12.191] (ovpn-12-191.pek2.redhat.com [10.72.12.191]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 3C160620CA; Thu, 27 Dec 2018 09:56:53 +0000 (UTC) Subject: Re: [PATCH RFC 1/2] virtio-net: bql support To: "Michael S. Tsirkin" Cc: linux-kernel@vger.kernel.org, maxime.coquelin@redhat.com, tiwei.bie@intel.com, wexu@redhat.com, jfreimann@redhat.com, "David S. Miller" , virtualization@lists.linux-foundation.org, netdev@vger.kernel.org References: <20181205225323.12555-1-mst@redhat.com> <20181205225323.12555-2-mst@redhat.com> <21384cb5-99a6-7431-1039-b356521e1bc3@redhat.com> <20181226101411-mutt-send-email-mst@kernel.org> From: Jason Wang Message-ID: <0520efac-049d-d1c0-e085-4e65247316ff@redhat.com> Date: Thu, 27 Dec 2018 17:56:51 +0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: <20181226101411-mutt-send-email-mst@kernel.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Thu, 27 Dec 2018 09:57:04 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2018/12/26 下午11:15, Michael S. Tsirkin wrote: > On Thu, Dec 06, 2018 at 04:17:36PM +0800, Jason Wang wrote: >> On 2018/12/6 上午6:54, Michael S. Tsirkin wrote: >>> When use_napi is set, let's enable BQLs. Note: some of the issues are >>> similar to wifi. It's worth considering whether something similar to >>> commit 36148c2bbfbe ("mac80211: Adjust TSQ pacing shift") might be >>> benefitial. >> >> I've played a similar patch several days before. The tricky part is the mode >> switching between napi and no napi. We should make sure when the packet is >> sent and trakced by BQL,  it should be consumed by BQL as well. I did it by >> tracking it through skb->cb.  And deal with the freeze by reset the BQL >> status. Patch attached. >> >> But when testing with vhost-net, I don't very a stable performance, it was >> probably because we batch the used ring updating so tx interrupt may come >> randomly. We probably need to implement time bounded coalescing mechanism >> which could be configured from userspace. >> >> Btw, maybe it's time just enable napi TX by default. I get ~10% TCP_RR >> regression on machine without APICv, (haven't found time to test APICv >> machine). But consider it was for correctness, I think it's acceptable? Then >> we can do optimization on top? >> >> >> Thanks > I don't see how it's for correctness to be frank. Socket accounting is wrong in the case. This should be a bug in fact. > What if we just do the bulk free? Does that fix the regression? I can test it. > > >>> Signed-off-by: Michael S. Tsirkin >>> --- >>> drivers/net/virtio_net.c | 27 +++++++++++++++++++-------- >>> 1 file changed, 19 insertions(+), 8 deletions(-) >>> >>> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c >>> index cecfd77c9f3c..b657bde6b94b 100644 >>> --- a/drivers/net/virtio_net.c >>> +++ b/drivers/net/virtio_net.c >>> @@ -1325,7 +1325,8 @@ static int virtnet_receive(struct receive_queue *rq, int budget, >>> return stats.packets; >>> } >>> -static void free_old_xmit_skbs(struct send_queue *sq) >>> +static void free_old_xmit_skbs(struct send_queue *sq, struct netdev_queue *txq, >>> + bool use_napi) >>> { >>> struct sk_buff *skb; >>> unsigned int len; >>> @@ -1347,6 +1348,9 @@ static void free_old_xmit_skbs(struct send_queue *sq) >>> if (!packets) >>> return; >>> + if (use_napi) >>> + netdev_tx_completed_queue(txq, packets, bytes); >>> + >>> u64_stats_update_begin(&sq->stats.syncp); >>> sq->stats.bytes += bytes; >>> sq->stats.packets += packets; >>> @@ -1364,7 +1368,7 @@ static void virtnet_poll_cleantx(struct receive_queue *rq) >>> return; >>> if (__netif_tx_trylock(txq)) { >>> - free_old_xmit_skbs(sq); >>> + free_old_xmit_skbs(sq, txq, true); >>> __netif_tx_unlock(txq); >>> } >>> @@ -1440,7 +1444,7 @@ static int virtnet_poll_tx(struct napi_struct *napi, int budget) >>> struct netdev_queue *txq = netdev_get_tx_queue(vi->dev, vq2txq(sq->vq)); >>> __netif_tx_lock(txq, raw_smp_processor_id()); >>> - free_old_xmit_skbs(sq); >>> + free_old_xmit_skbs(sq, txq, true); >>> __netif_tx_unlock(txq); >>> virtqueue_napi_complete(napi, sq->vq, 0); >>> @@ -1505,13 +1509,15 @@ static netdev_tx_t start_xmit(struct sk_buff *skb, struct net_device *dev) >>> struct send_queue *sq = &vi->sq[qnum]; >>> int err; >>> struct netdev_queue *txq = netdev_get_tx_queue(dev, qnum); >>> - bool kick = !skb->xmit_more; >>> + bool more = skb->xmit_more; >>> bool use_napi = sq->napi.weight; >>> + unsigned int bytes = skb->len; >>> + bool kick; >>> /* Free up any pending old buffers before queueing new ones. */ >>> - free_old_xmit_skbs(sq); >>> + free_old_xmit_skbs(sq, txq, use_napi); >>> - if (use_napi && kick) >>> + if (use_napi && !more) >>> virtqueue_enable_cb_delayed(sq->vq); >>> /* timestamp packet in software */ >>> @@ -1552,7 +1558,7 @@ static netdev_tx_t start_xmit(struct sk_buff *skb, struct net_device *dev) >>> if (!use_napi && >>> unlikely(!virtqueue_enable_cb_delayed(sq->vq))) { >>> /* More just got used, free them then recheck. */ >>> - free_old_xmit_skbs(sq); >>> + free_old_xmit_skbs(sq, txq, false); >>> if (sq->vq->num_free >= 2+MAX_SKB_FRAGS) { >>> netif_start_subqueue(dev, qnum); >>> virtqueue_disable_cb(sq->vq); >>> @@ -1560,7 +1566,12 @@ static netdev_tx_t start_xmit(struct sk_buff *skb, struct net_device *dev) >>> } >>> } >>> - if (kick || netif_xmit_stopped(txq)) { >>> + if (use_napi) >>> + kick = __netdev_tx_sent_queue(txq, bytes, more); >>> + else >>> + kick = !more || netif_xmit_stopped(txq); >>> + >>> + if (kick) { >>> if (virtqueue_kick_prepare(sq->vq) && virtqueue_notify(sq->vq)) { >>> u64_stats_update_begin(&sq->stats.syncp); >>> sq->stats.kicks++; From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jason Wang Subject: Re: [PATCH RFC 1/2] virtio-net: bql support Date: Thu, 27 Dec 2018 17:56:51 +0800 Message-ID: <0520efac-049d-d1c0-e085-4e65247316ff@redhat.com> References: <20181205225323.12555-1-mst@redhat.com> <20181205225323.12555-2-mst@redhat.com> <21384cb5-99a6-7431-1039-b356521e1bc3@redhat.com> <20181226101411-mutt-send-email-mst@kernel.org> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8"; Format="flowed" Content-Transfer-Encoding: base64 Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, maxime.coquelin@redhat.com, wexu@redhat.com, "David S. Miller" To: "Michael S. Tsirkin" Return-path: In-Reply-To: <20181226101411-mutt-send-email-mst@kernel.org> Content-Language: en-US List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: virtualization-bounces@lists.linux-foundation.org Errors-To: virtualization-bounces@lists.linux-foundation.org List-Id: netdev.vger.kernel.org Ck9uIDIwMTgvMTIvMjYg5LiL5Y2IMTE6MTUsIE1pY2hhZWwgUy4gVHNpcmtpbiB3cm90ZToKPiBP biBUaHUsIERlYyAwNiwgMjAxOCBhdCAwNDoxNzozNlBNICswODAwLCBKYXNvbiBXYW5nIHdyb3Rl Ogo+PiBPbiAyMDE4LzEyLzYg5LiK5Y2INjo1NCwgTWljaGFlbCBTLiBUc2lya2luIHdyb3RlOgo+ Pj4gV2hlbiB1c2VfbmFwaSBpcyBzZXQsIGxldCdzIGVuYWJsZSBCUUxzLiAgTm90ZTogc29tZSBv ZiB0aGUgaXNzdWVzIGFyZQo+Pj4gc2ltaWxhciB0byB3aWZpLiAgSXQncyB3b3J0aCBjb25zaWRl cmluZyB3aGV0aGVyIHNvbWV0aGluZyBzaW1pbGFyIHRvCj4+PiBjb21taXQgMzYxNDhjMmJiZmJl ICgibWFjODAyMTE6IEFkanVzdCBUU1EgcGFjaW5nIHNoaWZ0IikgbWlnaHQgYmUKPj4+IGJlbmVm aXRpYWwuCj4+Cj4+IEkndmUgcGxheWVkIGEgc2ltaWxhciBwYXRjaCBzZXZlcmFsIGRheXMgYmVm b3JlLiBUaGUgdHJpY2t5IHBhcnQgaXMgdGhlIG1vZGUKPj4gc3dpdGNoaW5nIGJldHdlZW4gbmFw aSBhbmQgbm8gbmFwaS4gV2Ugc2hvdWxkIG1ha2Ugc3VyZSB3aGVuIHRoZSBwYWNrZXQgaXMKPj4g c2VudCBhbmQgdHJha2NlZCBieSBCUUwswqAgaXQgc2hvdWxkIGJlIGNvbnN1bWVkIGJ5IEJRTCBh cyB3ZWxsLiBJIGRpZCBpdCBieQo+PiB0cmFja2luZyBpdCB0aHJvdWdoIHNrYi0+Y2IuwqAgQW5k IGRlYWwgd2l0aCB0aGUgZnJlZXplIGJ5IHJlc2V0IHRoZSBCUUwKPj4gc3RhdHVzLiBQYXRjaCBh dHRhY2hlZC4KPj4KPj4gQnV0IHdoZW4gdGVzdGluZyB3aXRoIHZob3N0LW5ldCwgSSBkb24ndCB2 ZXJ5IGEgc3RhYmxlIHBlcmZvcm1hbmNlLCBpdCB3YXMKPj4gcHJvYmFibHkgYmVjYXVzZSB3ZSBi YXRjaCB0aGUgdXNlZCByaW5nIHVwZGF0aW5nIHNvIHR4IGludGVycnVwdCBtYXkgY29tZQo+PiBy YW5kb21seS4gV2UgcHJvYmFibHkgbmVlZCB0byBpbXBsZW1lbnQgdGltZSBib3VuZGVkIGNvYWxl c2NpbmcgbWVjaGFuaXNtCj4+IHdoaWNoIGNvdWxkIGJlIGNvbmZpZ3VyZWQgZnJvbSB1c2Vyc3Bh Y2UuCj4+Cj4+IEJ0dywgbWF5YmUgaXQncyB0aW1lIGp1c3QgZW5hYmxlIG5hcGkgVFggYnkgZGVm YXVsdC4gSSBnZXQgfjEwJSBUQ1BfUlIKPj4gcmVncmVzc2lvbiBvbiBtYWNoaW5lIHdpdGhvdXQg QVBJQ3YsIChoYXZlbid0IGZvdW5kIHRpbWUgdG8gdGVzdCBBUElDdgo+PiBtYWNoaW5lKS4gQnV0 IGNvbnNpZGVyIGl0IHdhcyBmb3IgY29ycmVjdG5lc3MsIEkgdGhpbmsgaXQncyBhY2NlcHRhYmxl PyBUaGVuCj4+IHdlIGNhbiBkbyBvcHRpbWl6YXRpb24gb24gdG9wPwo+Pgo+Pgo+PiBUaGFua3MK PiBJIGRvbid0IHNlZSBob3cgaXQncyBmb3IgY29ycmVjdG5lc3MgdG8gYmUgZnJhbmsuCgoKU29j a2V0IGFjY291bnRpbmcgaXMgd3JvbmcgaW4gdGhlIGNhc2UuIFRoaXMgc2hvdWxkIGJlIGEgYnVn IGluIGZhY3QuCgoKPiBXaGF0IGlmIHdlIGp1c3QgZG8gdGhlIGJ1bGsgZnJlZT8gRG9lcyB0aGF0 IGZpeCB0aGUgcmVncmVzc2lvbj8KCgpJIGNhbiB0ZXN0IGl0LgoKCj4KPgo+Pj4gU2lnbmVkLW9m Zi1ieTogTWljaGFlbCBTLiBUc2lya2luIDxtc3RAcmVkaGF0LmNvbT4KPj4+IC0tLQo+Pj4gICAg ZHJpdmVycy9uZXQvdmlydGlvX25ldC5jIHwgMjcgKysrKysrKysrKysrKysrKysrKy0tLS0tLS0t Cj4+PiAgICAxIGZpbGUgY2hhbmdlZCwgMTkgaW5zZXJ0aW9ucygrKSwgOCBkZWxldGlvbnMoLSkK Pj4+Cj4+PiBkaWZmIC0tZ2l0IGEvZHJpdmVycy9uZXQvdmlydGlvX25ldC5jIGIvZHJpdmVycy9u ZXQvdmlydGlvX25ldC5jCj4+PiBpbmRleCBjZWNmZDc3YzlmM2MuLmI2NTdiZGU2Yjk0YiAxMDA2 NDQKPj4+IC0tLSBhL2RyaXZlcnMvbmV0L3ZpcnRpb19uZXQuYwo+Pj4gKysrIGIvZHJpdmVycy9u ZXQvdmlydGlvX25ldC5jCj4+PiBAQCAtMTMyNSw3ICsxMzI1LDggQEAgc3RhdGljIGludCB2aXJ0 bmV0X3JlY2VpdmUoc3RydWN0IHJlY2VpdmVfcXVldWUgKnJxLCBpbnQgYnVkZ2V0LAo+Pj4gICAg CXJldHVybiBzdGF0cy5wYWNrZXRzOwo+Pj4gICAgfQo+Pj4gLXN0YXRpYyB2b2lkIGZyZWVfb2xk X3htaXRfc2ticyhzdHJ1Y3Qgc2VuZF9xdWV1ZSAqc3EpCj4+PiArc3RhdGljIHZvaWQgZnJlZV9v bGRfeG1pdF9za2JzKHN0cnVjdCBzZW5kX3F1ZXVlICpzcSwgc3RydWN0IG5ldGRldl9xdWV1ZSAq dHhxLAo+Pj4gKwkJCSAgICAgICBib29sIHVzZV9uYXBpKQo+Pj4gICAgewo+Pj4gICAgCXN0cnVj dCBza19idWZmICpza2I7Cj4+PiAgICAJdW5zaWduZWQgaW50IGxlbjsKPj4+IEBAIC0xMzQ3LDYg KzEzNDgsOSBAQCBzdGF0aWMgdm9pZCBmcmVlX29sZF94bWl0X3NrYnMoc3RydWN0IHNlbmRfcXVl dWUgKnNxKQo+Pj4gICAgCWlmICghcGFja2V0cykKPj4+ICAgIAkJcmV0dXJuOwo+Pj4gKwlpZiAo dXNlX25hcGkpCj4+PiArCQluZXRkZXZfdHhfY29tcGxldGVkX3F1ZXVlKHR4cSwgcGFja2V0cywg Ynl0ZXMpOwo+Pj4gKwo+Pj4gICAgCXU2NF9zdGF0c191cGRhdGVfYmVnaW4oJnNxLT5zdGF0cy5z eW5jcCk7Cj4+PiAgICAJc3EtPnN0YXRzLmJ5dGVzICs9IGJ5dGVzOwo+Pj4gICAgCXNxLT5zdGF0 cy5wYWNrZXRzICs9IHBhY2tldHM7Cj4+PiBAQCAtMTM2NCw3ICsxMzY4LDcgQEAgc3RhdGljIHZv aWQgdmlydG5ldF9wb2xsX2NsZWFudHgoc3RydWN0IHJlY2VpdmVfcXVldWUgKnJxKQo+Pj4gICAg CQlyZXR1cm47Cj4+PiAgICAJaWYgKF9fbmV0aWZfdHhfdHJ5bG9jayh0eHEpKSB7Cj4+PiAtCQlm cmVlX29sZF94bWl0X3NrYnMoc3EpOwo+Pj4gKwkJZnJlZV9vbGRfeG1pdF9za2JzKHNxLCB0eHEs IHRydWUpOwo+Pj4gICAgCQlfX25ldGlmX3R4X3VubG9jayh0eHEpOwo+Pj4gICAgCX0KPj4+IEBA IC0xNDQwLDcgKzE0NDQsNyBAQCBzdGF0aWMgaW50IHZpcnRuZXRfcG9sbF90eChzdHJ1Y3QgbmFw aV9zdHJ1Y3QgKm5hcGksIGludCBidWRnZXQpCj4+PiAgICAJc3RydWN0IG5ldGRldl9xdWV1ZSAq dHhxID0gbmV0ZGV2X2dldF90eF9xdWV1ZSh2aS0+ZGV2LCB2cTJ0eHEoc3EtPnZxKSk7Cj4+PiAg ICAJX19uZXRpZl90eF9sb2NrKHR4cSwgcmF3X3NtcF9wcm9jZXNzb3JfaWQoKSk7Cj4+PiAtCWZy ZWVfb2xkX3htaXRfc2ticyhzcSk7Cj4+PiArCWZyZWVfb2xkX3htaXRfc2ticyhzcSwgdHhxLCB0 cnVlKTsKPj4+ICAgIAlfX25ldGlmX3R4X3VubG9jayh0eHEpOwo+Pj4gICAgCXZpcnRxdWV1ZV9u YXBpX2NvbXBsZXRlKG5hcGksIHNxLT52cSwgMCk7Cj4+PiBAQCAtMTUwNSwxMyArMTUwOSwxNSBA QCBzdGF0aWMgbmV0ZGV2X3R4X3Qgc3RhcnRfeG1pdChzdHJ1Y3Qgc2tfYnVmZiAqc2tiLCBzdHJ1 Y3QgbmV0X2RldmljZSAqZGV2KQo+Pj4gICAgCXN0cnVjdCBzZW5kX3F1ZXVlICpzcSA9ICZ2aS0+ c3FbcW51bV07Cj4+PiAgICAJaW50IGVycjsKPj4+ICAgIAlzdHJ1Y3QgbmV0ZGV2X3F1ZXVlICp0 eHEgPSBuZXRkZXZfZ2V0X3R4X3F1ZXVlKGRldiwgcW51bSk7Cj4+PiAtCWJvb2wga2ljayA9ICFz a2ItPnhtaXRfbW9yZTsKPj4+ICsJYm9vbCBtb3JlID0gc2tiLT54bWl0X21vcmU7Cj4+PiAgICAJ Ym9vbCB1c2VfbmFwaSA9IHNxLT5uYXBpLndlaWdodDsKPj4+ICsJdW5zaWduZWQgaW50IGJ5dGVz ID0gc2tiLT5sZW47Cj4+PiArCWJvb2wga2ljazsKPj4+ICAgIAkvKiBGcmVlIHVwIGFueSBwZW5k aW5nIG9sZCBidWZmZXJzIGJlZm9yZSBxdWV1ZWluZyBuZXcgb25lcy4gKi8KPj4+IC0JZnJlZV9v bGRfeG1pdF9za2JzKHNxKTsKPj4+ICsJZnJlZV9vbGRfeG1pdF9za2JzKHNxLCB0eHEsIHVzZV9u YXBpKTsKPj4+IC0JaWYgKHVzZV9uYXBpICYmIGtpY2spCj4+PiArCWlmICh1c2VfbmFwaSAmJiAh bW9yZSkKPj4+ICAgIAkJdmlydHF1ZXVlX2VuYWJsZV9jYl9kZWxheWVkKHNxLT52cSk7Cj4+PiAg ICAJLyogdGltZXN0YW1wIHBhY2tldCBpbiBzb2Z0d2FyZSAqLwo+Pj4gQEAgLTE1NTIsNyArMTU1 OCw3IEBAIHN0YXRpYyBuZXRkZXZfdHhfdCBzdGFydF94bWl0KHN0cnVjdCBza19idWZmICpza2Is IHN0cnVjdCBuZXRfZGV2aWNlICpkZXYpCj4+PiAgICAJCWlmICghdXNlX25hcGkgJiYKPj4+ICAg IAkJICAgIHVubGlrZWx5KCF2aXJ0cXVldWVfZW5hYmxlX2NiX2RlbGF5ZWQoc3EtPnZxKSkpIHsK Pj4+ICAgIAkJCS8qIE1vcmUganVzdCBnb3QgdXNlZCwgZnJlZSB0aGVtIHRoZW4gcmVjaGVjay4g Ki8KPj4+IC0JCQlmcmVlX29sZF94bWl0X3NrYnMoc3EpOwo+Pj4gKwkJCWZyZWVfb2xkX3htaXRf c2ticyhzcSwgdHhxLCBmYWxzZSk7Cj4+PiAgICAJCQlpZiAoc3EtPnZxLT5udW1fZnJlZSA+PSAy K01BWF9TS0JfRlJBR1MpIHsKPj4+ICAgIAkJCQluZXRpZl9zdGFydF9zdWJxdWV1ZShkZXYsIHFu dW0pOwo+Pj4gICAgCQkJCXZpcnRxdWV1ZV9kaXNhYmxlX2NiKHNxLT52cSk7Cj4+PiBAQCAtMTU2 MCw3ICsxNTY2LDEyIEBAIHN0YXRpYyBuZXRkZXZfdHhfdCBzdGFydF94bWl0KHN0cnVjdCBza19i dWZmICpza2IsIHN0cnVjdCBuZXRfZGV2aWNlICpkZXYpCj4+PiAgICAJCX0KPj4+ICAgIAl9Cj4+ PiAtCWlmIChraWNrIHx8IG5ldGlmX3htaXRfc3RvcHBlZCh0eHEpKSB7Cj4+PiArCWlmICh1c2Vf bmFwaSkKPj4+ICsJCWtpY2sgPSBfX25ldGRldl90eF9zZW50X3F1ZXVlKHR4cSwgYnl0ZXMsIG1v cmUpOwo+Pj4gKwllbHNlCj4+PiArCQlraWNrID0gIW1vcmUgfHwgbmV0aWZfeG1pdF9zdG9wcGVk KHR4cSk7Cj4+PiArCj4+PiArCWlmIChraWNrKSB7Cj4+PiAgICAJCWlmICh2aXJ0cXVldWVfa2lj a19wcmVwYXJlKHNxLT52cSkgJiYgdmlydHF1ZXVlX25vdGlmeShzcS0+dnEpKSB7Cj4+PiAgICAJ CQl1NjRfc3RhdHNfdXBkYXRlX2JlZ2luKCZzcS0+c3RhdHMuc3luY3ApOwo+Pj4gICAgCQkJc3Et PnN0YXRzLmtpY2tzKys7Cl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fClZpcnR1YWxpemF0aW9uIG1haWxpbmcgbGlzdApWaXJ0dWFsaXphdGlvbkBsaXN0cy5s aW51eC1mb3VuZGF0aW9uLm9yZwpodHRwczovL2xpc3RzLmxpbnV4Zm91bmRhdGlvbi5vcmcvbWFp bG1hbi9saXN0aW5mby92aXJ0dWFsaXphdGlvbg==