From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.3 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6AC54C48BDF for ; Sun, 20 Jun 2021 16:42:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3CD946113C for ; Sun, 20 Jun 2021 16:42:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229650AbhFTQjR (ORCPT ); Sun, 20 Jun 2021 12:39:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51066 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229605AbhFTQjH (ORCPT ); Sun, 20 Jun 2021 12:39:07 -0400 Received: from mail-pf1-x431.google.com (mail-pf1-x431.google.com [IPv6:2607:f8b0:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0CF71C061574 for ; Sun, 20 Jun 2021 09:36:53 -0700 (PDT) Received: by mail-pf1-x431.google.com with SMTP id y4so4820393pfi.9 for ; Sun, 20 Jun 2021 09:36:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language:content-transfer-encoding; bh=40fJho7zFxUORfAU0PgoiB9XNKSR1aFNaisbGsGTME4=; b=fa039sxkxMwhDT+fC80Yz6MdjfczZw4ak/wSFBDwMgmpSyA8QWq5v4pwq1/DJVY9SH ioaISrjNWG1IV1HEM2LfCLw5vI+4W7VXnm/C5Qa1Gt8JoX/7ITadgkFhg10a3eHDvO9S /iyq83bNXpdbaTTka/ad8rjt0lyMlRo2d7kHb7AoeuStWEkbIvUcKSrmue7kRoNQyHI8 5/XEkf9g/lDx5gckvo8+79U94f+niI8rSNjwNQRIrAbM/woh1rFWAqzkM04VhX3Hm72Q FMBLTQuapfuAe3s+iR1Gi4YnsI82sUgXcmJ0eNU/xeRwvI0U9OfPNRgF50aLVvqY+xOG TWFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=40fJho7zFxUORfAU0PgoiB9XNKSR1aFNaisbGsGTME4=; b=VPdFVE30HgpY+2l6y1tRTbDmIqQqmGrnFd7/h5UfbXFdC+PwvMoElbK3x9X6TTPgrf 9knsqyQSjRYz1IS9j8RFJBVDiI96RlhnXNJTb+LLlGC9cbw65f6UrwdfAB956vWEMWvs cqIB96b1Luxm5F4kV5wEouuiRpap62vqIstCwR6fMIW7VEIjHU2P1mv/CRvJWL/jKwUv MfAL2vE2ODXZE1jIpGFvqZ2VDhJFPIkSu6pOjCxoEYiQJ0dq+oAPdEWQJ7ga0bdFh6iB aCXeG63io5RFgTVTcrB3OZyvM5Yy5dlMcz0Bdr50FRwuMLLphML++HgzGRpdxLACDhZs qE/g== X-Gm-Message-State: AOAM531xQfrgdabnAYpwVA1rytdhb+VJvmr7xKt6hsqGQCGpilIWkzoS WUB/5YtxgoGXjkoYiNZqKIg= X-Google-Smtp-Source: ABdhPJwAWt0Q9dppEFDIYFvVOXfk56VB5bN5oTHiQS/DJ1Lv4QvzmoXvV8uKggnuFht7CDLBjbBb6Q== X-Received: by 2002:a62:3606:0:b029:302:3c87:4f37 with SMTP id d6-20020a6236060000b02903023c874f37mr5042058pfa.53.1624207012263; Sun, 20 Jun 2021 09:36:52 -0700 (PDT) Received: from [192.168.1.153] (87.19.178.217.shared.user.transix.jp. [217.178.19.87]) by smtp.gmail.com with ESMTPSA id v9sm12913263pfn.22.2021.06.20.09.36.50 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 20 Jun 2021 09:36:51 -0700 (PDT) Subject: Re: [PATCH v2 0/5] riscv: improving uaccess with logs from network bench To: Ben Dooks , Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org References: <5a5c07ac-8c11-79d3-46a3-a255d4148f76@gmail.com> <542310bc-840d-d5c9-a7b3-40f58504e7b5@codethink.co.uk> From: Akira Tsukamoto Message-ID: <88ba6862-b43b-bcab-9485-1339cc765f47@gmail.com> Date: Mon, 21 Jun 2021 01:36:49 +0900 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <542310bc-840d-d5c9-a7b3-40f58504e7b5@codethink.co.uk> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 6/20/21 19:02, Ben Dooks wrote: > On 19/06/2021 12:21, Akira Tsukamoto wrote: >> Optimizing copy_to_user and copy_from_user. >> >> I rewrote the functions in v2, heavily influenced by Garry's memcpy >> function [1]. >> The functions must be written in assembler to handle page faults manually >> inside the function. >> >> With the changes, improves in the percentage usage and some performance >> of network speed in UDP packets. >> Only patching copy_user. Using the original memcpy. >> >> All results are from the same base kernel, same rootfs and same >> BeagleV beta board. > > Is there a git tree for these to try them out? Sure, please try. The kernel without the patch is starlight branch and the kernel with these patches is starlight-ua-new-up at https://github.com/mcd500/linux-jh7100 The starlight is maintained by Esmil where main development is happening. And the rootfs on beaglev is uploaded bellow. https://github.com/mcd500/Fedora_on_StarFive#custome-fedora-image To reproduce the results: (please customize with your ip addreses) The command I used for iperf3. --- TCP recv --- ** on PC side, using default mtu 1500 $ iperf3 -c 192.168.1.112 ** on riscv beaglev side, using default mtu 1500 [root@fedora-starfive ~]# iperf3 -s --- TCP send --- ** on PC side, using default mtu 1500 $ iperf3 -s ** on riscv beaglev side, using default mtu 1500 [root@fedora-starfive ~]# iperf3 -c 192.168.1.153 --- UDP send --- ** on PC side first, changing mtu size from 1500 to 9000 $ sudo ifconfig eth0 down $ sudo ifconfig eth0 mtu 9000 up $ iperf3 -s ** on riscv beaglev, No changing the mtu size on riscv beaglev [root@fedora-starfive ~]# iperf3 -u -b 1000M --length 50000 -c 192.168.1.153 --- UDP recv --- ** on PC side first, changing mtu size to 9000 $ sudo ifconfig eth0 down $ sudo ifconfig eth0 mtu 9000 up $ iperf3 -u -b 1000M --length 6500 -c 192.168.1.112 ** on riscv beaglev side, changing mtu size to 9000 too [root@fedora-starfive ~]# sudo ifconfig eth0 down [root@fedora-starfive ~]# sudo ifconfig eth0 mtu 9000 up [root@fedora-starfive ~]# iperf3 -s The perf: $ sudo perf top -Ue task-clock after login with ssh. > >> Comparison by "perf top -Ue task-clock" while running iperf3. >> >> --- TCP recv --- >>   * Before >>    40.40%  [kernel]  [k] memcpy >>    33.09%  [kernel]  [k] __asm_copy_to_user >>   * After >>    50.35%  [kernel]  [k] memcpy >>    13.76%  [kernel]  [k] __asm_copy_to_user >> >> --- TCP send --- >>   * Before >>    19.96%  [kernel]  [k] memcpy >>     9.84%  [kernel]  [k] __asm_copy_to_user >>   * After >>    14.27%  [kernel]  [k] memcpy >>     7.37%  [kernel]  [k] __asm_copy_to_user >> >> --- UDP send --- >>   * Before >>    25.18%  [kernel]  [k] memcpy >>    22.50%  [kernel]  [k] __asm_copy_to_user >>   * After >>    28.90%  [kernel]  [k] memcpy >>     9.49%  [kernel]  [k] __asm_copy_to_user >> >> --- UDP recv --- >>   * Before >>    44.45%  [kernel]  [k] memcpy >>    31.04%  [kernel]  [k] __asm_copy_to_user >>   * After >>    55.62%  [kernel]  [k] memcpy >>    11.22%  [kernel]  [k] __asm_copy_to_user > > What's the memcpy figure in the above? > Could you explain the figures please? It is the output of "perf top -Ue task-clock" while performing the iperf3 which I described above. Showing which functions are causing the most overhead inside the kernel during running iperf3 in cpu usage. The two biggest culprits were memcpy and __asm_copy_to_user showing high cpu usage, and this is the reason I listed the two. Initially this discussion started with Gary's memcpy patch on this list. I will write more details bellow. > >> Processing network packets require a lot of unaligned access for the packet >> header, which is not able to change the design of the header format to be >> aligned. > > Isn't there an option to allow padding of network packets > in the skbuff to make the fields aligned for architectures > which do not have efficient unaligned loads (looking at you > arm32). Has this been looked at? I am trying at 64bit risc-v beaglev beta board. My understanding of skbuff is that it is for aligning data when the handling inside the kernel. It would help if memcpy and __asm_copy_to_user were not causing such a huge percentage. This patch is against copy_to_user and copy_from_user purely used for copying between kernel space and user space. By looking the overhead on perf, the cpu usage is increasing on copy_to_user because the user app (iperf3 here) uses socket API with large packet size. I used to use maximum buffer size of mtu for reduce the number of calling recvform() sendto() for UDP programing too. And most of the network programmer probably do the same. > >> And user applications call system calls with a large buffer for send/recf() >> and sendto/recvfrom() to repeat less function calls for the optimization. >> >> v1 -> v2: >> - Added shift copy >> - Separated patches for readability of changes in assembler >> - Using perf results >> >> [1] https://lkml.org/lkml/2021/2/16/778 >> >> Akira Tsukamoto (5): >>    riscv: __asm_to/copy_from_user: delete existing code >>    riscv: __asm_to/copy_from_user: Adding byte copy first >>    riscv: __asm_to/copy_from_user: Copy until dst is aligned address >>    riscv: __asm_to/copy_from_user: Bulk copy while shifting misaligned >>      data >>    riscv: __asm_to/copy_from_user: Bulk copy when both src dst are >>      aligned >> >>   arch/riscv/lib/uaccess.S | 181 +++++++++++++++++++++++++++++++-------- >>   1 file changed, 146 insertions(+), 35 deletions(-) > > I'm concerned that delete and then re-add is either going to make > the series un-bisectable or leave a point where the kernel is very > broken? I completely agree and understand. The only reason I split the patches is because of the comments in the other thread. It definitely breaks the bisect. Once the content of this patch is understood and agreed, I will re-spin them in one patch, otherwise it will not boot when only individual patch is applied. The Gary's memcpy patch was posted a while ago, and even it has the best result in bench, it was not merged. When we were measuring network performance on beaglev beta board, the Gary's memcpy did have huge improvement. The patch was not easy to review and understand, but it really helps the network performance. By reading the discussion on his patch, I felt the first priority is able to be understood the cause of cpu usage and speed results. Please read the discussion by Gary, Palmer, Matteo, others and I in the list. Matteo is rewriting Gary's patch in C, which is better for maintainability and incorporate the wisdom of the optimization of the compiler. The user_copy are written in assembler or inline assembler, as I wrote. I just want to help making it better, so once the consensus are made, I will make them to one patch. Or I am fine somebody else comes out with better results. My attempt to do similar patches dates long time ago in 2002. https://linux-kernel.vger.kernel.narkive.com/zU6OFlI6/cft-patch-2-5-47-athlon-druon-much-faster-copy-user-function http://lkml.iu.edu/hypermail/linux/kernel/0211.2/0928.html Akira From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.2 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0669DC48BDF for ; Sun, 20 Jun 2021 16:37:31 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 84E516113C for ; Sun, 20 Jun 2021 16:37:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 84E516113C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date: Message-ID:From:References:To:Subject:Reply-To:Cc:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Ng33vKGGYg8Ji//3DQFHUIgzfJgnjKvnkIXoqJCBeww=; b=lyBaAwtUTwVK7k/1f0wKpvL7pQ 0jgQHdFVfltxRhT200bm3qwXc6osrkcZJXGg02Oxlp89lDD+oNDawmCP6nIV2leVe3lkxOU04ExHp dfxah/e25pyo8fYdKh3byLg/nYiZydAbWpRc9kZopGX5zJ4fAFYU1hJ2c8fwvk+PuJvqLT4tfJYdF ZXfpKt6/qs588KLL3vwGnBGz0aiORk0X3nhIWLVHWhbvkF9LgRv2WY24gK5pvZfG3KkeZI3RRrzhi rD3yONnWTC8ZLhteohAaxZfHZDzKaOUBJwQOd6YDEzE30VVcmhktX/J2PNDpl1YXehQfgUk1/tQ2g wRkpe3mw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lv0RQ-001KCP-48; Sun, 20 Jun 2021 16:37:00 +0000 Received: from mail-pg1-x52f.google.com ([2607:f8b0:4864:20::52f]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lv0RM-001KC0-8K for linux-riscv@lists.infradead.org; Sun, 20 Jun 2021 16:36:58 +0000 Received: by mail-pg1-x52f.google.com with SMTP id n12so3422614pgs.13 for ; Sun, 20 Jun 2021 09:36:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language:content-transfer-encoding; bh=40fJho7zFxUORfAU0PgoiB9XNKSR1aFNaisbGsGTME4=; b=fa039sxkxMwhDT+fC80Yz6MdjfczZw4ak/wSFBDwMgmpSyA8QWq5v4pwq1/DJVY9SH ioaISrjNWG1IV1HEM2LfCLw5vI+4W7VXnm/C5Qa1Gt8JoX/7ITadgkFhg10a3eHDvO9S /iyq83bNXpdbaTTka/ad8rjt0lyMlRo2d7kHb7AoeuStWEkbIvUcKSrmue7kRoNQyHI8 5/XEkf9g/lDx5gckvo8+79U94f+niI8rSNjwNQRIrAbM/woh1rFWAqzkM04VhX3Hm72Q FMBLTQuapfuAe3s+iR1Gi4YnsI82sUgXcmJ0eNU/xeRwvI0U9OfPNRgF50aLVvqY+xOG TWFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=40fJho7zFxUORfAU0PgoiB9XNKSR1aFNaisbGsGTME4=; b=G9QIW2D32gz+gcjdgLccUZxIbDdYSNz60CyY7c/UGY05/5b1PRLsbytQ4eN+f50v5Y 5Ad9HzZwdODc49bswLl9UWikGy+kt8wb6R0qyux/v/b/9MS8ZUUtkPbo/UbOZ3gcUOru 0u/jdy05b+0DyQN46JpHmOW4dNNTn+G4GjuHEG1NsKRQj9kkUGFXby+sMky20pWphAoH Nru39J/eiz9cVYowJeV0c2gIabZfUttaxq5ywukrmRqHI4WRBiLYlkynzGrr+7b/relV Wkmnom2ppfeFSbmJrRJKe5PrJ1PN6tH/QogdikK1f+MtSgsdPfotEmdlFqPM7/1JzoCV 2XxA== X-Gm-Message-State: AOAM531xZ+grhYsDSIl1jQBWWuxm/jNdOtbv3SF5N2cc1UfOgPO1jj8F Faox5hlAgzg8c5y2rqlYN+WOChhFYq+Lpg== X-Google-Smtp-Source: ABdhPJwAWt0Q9dppEFDIYFvVOXfk56VB5bN5oTHiQS/DJ1Lv4QvzmoXvV8uKggnuFht7CDLBjbBb6Q== X-Received: by 2002:a62:3606:0:b029:302:3c87:4f37 with SMTP id d6-20020a6236060000b02903023c874f37mr5042058pfa.53.1624207012263; Sun, 20 Jun 2021 09:36:52 -0700 (PDT) Received: from [192.168.1.153] (87.19.178.217.shared.user.transix.jp. [217.178.19.87]) by smtp.gmail.com with ESMTPSA id v9sm12913263pfn.22.2021.06.20.09.36.50 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 20 Jun 2021 09:36:51 -0700 (PDT) Subject: Re: [PATCH v2 0/5] riscv: improving uaccess with logs from network bench To: Ben Dooks , Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org References: <5a5c07ac-8c11-79d3-46a3-a255d4148f76@gmail.com> <542310bc-840d-d5c9-a7b3-40f58504e7b5@codethink.co.uk> From: Akira Tsukamoto Message-ID: <88ba6862-b43b-bcab-9485-1339cc765f47@gmail.com> Date: Mon, 21 Jun 2021 01:36:49 +0900 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <542310bc-840d-d5c9-a7b3-40f58504e7b5@codethink.co.uk> Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210620_093656_366511_F9F44556 X-CRM114-Status: GOOD ( 35.20 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Ck9uIDYvMjAvMjEgMTk6MDIsIEJlbiBEb29rcyB3cm90ZToKPiBPbiAxOS8wNi8yMDIxIDEyOjIx LCBBa2lyYSBUc3VrYW1vdG8gd3JvdGU6Cj4+IE9wdGltaXppbmcgY29weV90b191c2VyIGFuZCBj b3B5X2Zyb21fdXNlci4KPj4KPj4gSSByZXdyb3RlIHRoZSBmdW5jdGlvbnMgaW4gdjIsIGhlYXZp bHkgaW5mbHVlbmNlZCBieSBHYXJyeSdzIG1lbWNweQo+PiBmdW5jdGlvbiBbMV0uCj4+IFRoZSBm dW5jdGlvbnMgbXVzdCBiZSB3cml0dGVuIGluIGFzc2VtYmxlciB0byBoYW5kbGUgcGFnZSBmYXVs dHMgbWFudWFsbHkKPj4gaW5zaWRlIHRoZSBmdW5jdGlvbi4KPj4KPj4gV2l0aCB0aGUgY2hhbmdl cywgaW1wcm92ZXMgaW4gdGhlIHBlcmNlbnRhZ2UgdXNhZ2UgYW5kIHNvbWUgcGVyZm9ybWFuY2UK Pj4gb2YgbmV0d29yayBzcGVlZCBpbiBVRFAgcGFja2V0cy4KPj4gT25seSBwYXRjaGluZyBjb3B5 X3VzZXIuIFVzaW5nIHRoZSBvcmlnaW5hbCBtZW1jcHkuCj4+Cj4+IEFsbCByZXN1bHRzIGFyZSBm cm9tIHRoZSBzYW1lIGJhc2Uga2VybmVsLCBzYW1lIHJvb3RmcyBhbmQgc2FtZQo+PiBCZWFnbGVW IGJldGEgYm9hcmQuCj4gCj4gSXMgdGhlcmUgYSBnaXQgdHJlZSBmb3IgdGhlc2UgdG8gdHJ5IHRo ZW0gb3V0PwoKU3VyZSwgcGxlYXNlIHRyeS4KClRoZSBrZXJuZWwgd2l0aG91dCB0aGUgcGF0Y2gg aXMgc3RhcmxpZ2h0IGJyYW5jaCBhbmQKdGhlIGtlcm5lbCB3aXRoIHRoZXNlIHBhdGNoZXMgaXMg c3RhcmxpZ2h0LXVhLW5ldy11cCBhdCAKaHR0cHM6Ly9naXRodWIuY29tL21jZDUwMC9saW51eC1q aDcxMDAKClRoZSBzdGFybGlnaHQgaXMgbWFpbnRhaW5lZCBieSBFc21pbCB3aGVyZSBtYWluIGRl dmVsb3BtZW50CmlzIGhhcHBlbmluZy4KCkFuZCB0aGUgcm9vdGZzIG9uIGJlYWdsZXYgaXMgdXBs b2FkZWQgYmVsbG93LgpodHRwczovL2dpdGh1Yi5jb20vbWNkNTAwL0ZlZG9yYV9vbl9TdGFyRml2 ZSNjdXN0b21lLWZlZG9yYS1pbWFnZQoKVG8gcmVwcm9kdWNlIHRoZSByZXN1bHRzOgoocGxlYXNl IGN1c3RvbWl6ZSB3aXRoIHlvdXIgaXAgYWRkcmVzZXMpCgpUaGUgY29tbWFuZCBJIHVzZWQgZm9y IGlwZXJmMy4KCi0tLSBUQ1AgcmVjdiAtLS0KKiogb24gUEMgc2lkZSwgdXNpbmcgZGVmYXVsdCBt dHUgMTUwMAokIGlwZXJmMyAtYyAxOTIuMTY4LjEuMTEyCioqIG9uIHJpc2N2IGJlYWdsZXYgc2lk ZSwgdXNpbmcgZGVmYXVsdCBtdHUgMTUwMApbcm9vdEBmZWRvcmEtc3RhcmZpdmUgfl0jIGlwZXJm MyAtcwoKLS0tIFRDUCBzZW5kIC0tLQoqKiBvbiBQQyBzaWRlLCB1c2luZyBkZWZhdWx0IG10dSAx NTAwCiQgaXBlcmYzIC1zCioqIG9uIHJpc2N2IGJlYWdsZXYgc2lkZSwgdXNpbmcgZGVmYXVsdCBt dHUgMTUwMApbcm9vdEBmZWRvcmEtc3RhcmZpdmUgfl0jIGlwZXJmMyAtYyAxOTIuMTY4LjEuMTUz CgotLS0gVURQIHNlbmQgLS0tCioqIG9uIFBDIHNpZGUgZmlyc3QsIGNoYW5naW5nIG10dSBzaXpl IGZyb20gMTUwMCB0byA5MDAwCiQgc3VkbyBpZmNvbmZpZyBldGgwIGRvd24KJCBzdWRvIGlmY29u ZmlnIGV0aDAgbXR1IDkwMDAgdXAKJCBpcGVyZjMgLXMKKiogb24gcmlzY3YgYmVhZ2xldiwgTm8g Y2hhbmdpbmcgdGhlIG10dSBzaXplIG9uIHJpc2N2IGJlYWdsZXYKW3Jvb3RAZmVkb3JhLXN0YXJm aXZlIH5dIyBpcGVyZjMgLXUgLWIgMTAwME0gLS1sZW5ndGggNTAwMDAgLWMgMTkyLjE2OC4xLjE1 MwoKLS0tIFVEUCByZWN2IC0tLQoqKiBvbiBQQyBzaWRlIGZpcnN0LCBjaGFuZ2luZyBtdHUgc2l6 ZSB0byA5MDAwCiQgc3VkbyBpZmNvbmZpZyBldGgwIGRvd24KJCBzdWRvIGlmY29uZmlnIGV0aDAg bXR1IDkwMDAgdXAKJCBpcGVyZjMgLXUgLWIgMTAwME0gLS1sZW5ndGggNjUwMCAtYyAxOTIuMTY4 LjEuMTEyCioqIG9uIHJpc2N2IGJlYWdsZXYgc2lkZSwgY2hhbmdpbmcgbXR1IHNpemUgdG8gOTAw MCB0b28KW3Jvb3RAZmVkb3JhLXN0YXJmaXZlIH5dIyBzdWRvIGlmY29uZmlnIGV0aDAgZG93bgpb cm9vdEBmZWRvcmEtc3RhcmZpdmUgfl0jIHN1ZG8gaWZjb25maWcgZXRoMCBtdHUgOTAwMCB1cApb cm9vdEBmZWRvcmEtc3RhcmZpdmUgfl0jIGlwZXJmMyAtcwoKVGhlIHBlcmY6CiQgc3VkbyBwZXJm IHRvcCAtVWUgdGFzay1jbG9jawphZnRlciBsb2dpbiB3aXRoIHNzaC4KCj4gCj4+IENvbXBhcmlz b24gYnkgInBlcmYgdG9wIC1VZSB0YXNrLWNsb2NrIiB3aGlsZSBydW5uaW5nIGlwZXJmMy4KPj4K Pj4gLS0tIFRDUCByZWN2IC0tLQo+PiDCoMKgKiBCZWZvcmUKPj4gwqDCoCA0MC40MCXCoCBba2Vy bmVsXcKgIFtrXSBtZW1jcHkKPj4gwqDCoCAzMy4wOSXCoCBba2VybmVsXcKgIFtrXSBfX2FzbV9j b3B5X3RvX3VzZXIKPj4gwqDCoCogQWZ0ZXIKPj4gwqDCoCA1MC4zNSXCoCBba2VybmVsXcKgIFtr XSBtZW1jcHkKPj4gwqDCoCAxMy43NiXCoCBba2VybmVsXcKgIFtrXSBfX2FzbV9jb3B5X3RvX3Vz ZXIKPj4KPj4gLS0tIFRDUCBzZW5kIC0tLQo+PiDCoMKgKiBCZWZvcmUKPj4gwqDCoCAxOS45NiXC oCBba2VybmVsXcKgIFtrXSBtZW1jcHkKPj4gwqDCoMKgIDkuODQlwqAgW2tlcm5lbF3CoCBba10g X19hc21fY29weV90b191c2VyCj4+IMKgwqAqIEFmdGVyCj4+IMKgwqAgMTQuMjclwqAgW2tlcm5l bF3CoCBba10gbWVtY3B5Cj4+IMKgwqDCoCA3LjM3JcKgIFtrZXJuZWxdwqAgW2tdIF9fYXNtX2Nv cHlfdG9fdXNlcgo+Pgo+PiAtLS0gVURQIHNlbmQgLS0tCj4+IMKgwqAqIEJlZm9yZQo+PiDCoMKg IDI1LjE4JcKgIFtrZXJuZWxdwqAgW2tdIG1lbWNweQo+PiDCoMKgIDIyLjUwJcKgIFtrZXJuZWxd wqAgW2tdIF9fYXNtX2NvcHlfdG9fdXNlcgo+PiDCoMKgKiBBZnRlcgo+PiDCoMKgIDI4LjkwJcKg IFtrZXJuZWxdwqAgW2tdIG1lbWNweQo+PiDCoMKgwqAgOS40OSXCoCBba2VybmVsXcKgIFtrXSBf X2FzbV9jb3B5X3RvX3VzZXIKPj4KPj4gLS0tIFVEUCByZWN2IC0tLQo+PiDCoMKgKiBCZWZvcmUK Pj4gwqDCoCA0NC40NSXCoCBba2VybmVsXcKgIFtrXSBtZW1jcHkKPj4gwqDCoCAzMS4wNCXCoCBb a2VybmVsXcKgIFtrXSBfX2FzbV9jb3B5X3RvX3VzZXIKPj4gwqDCoCogQWZ0ZXIKPj4gwqDCoCA1 NS42MiXCoCBba2VybmVsXcKgIFtrXSBtZW1jcHkKPj4gwqDCoCAxMS4yMiXCoCBba2VybmVsXcKg IFtrXSBfX2FzbV9jb3B5X3RvX3VzZXIKPiAKPiBXaGF0J3MgdGhlIG1lbWNweSBmaWd1cmUgaW4g dGhlIGFib3ZlPwo+IENvdWxkIHlvdSBleHBsYWluIHRoZSBmaWd1cmVzIHBsZWFzZT8KCkl0IGlz IHRoZSBvdXRwdXQgb2YgInBlcmYgdG9wIC1VZSB0YXNrLWNsb2NrIiAKd2hpbGUgcGVyZm9ybWlu ZyB0aGUgaXBlcmYzIHdoaWNoIEkgZGVzY3JpYmVkIGFib3ZlLgpTaG93aW5nIHdoaWNoIGZ1bmN0 aW9ucyBhcmUgY2F1c2luZyB0aGUgbW9zdCBvdmVyaGVhZAppbnNpZGUgdGhlIGtlcm5lbCBkdXJp bmcgcnVubmluZyBpcGVyZjMgaW4gY3B1IHVzYWdlLgoKVGhlIHR3byBiaWdnZXN0IGN1bHByaXRz IHdlcmUgbWVtY3B5IGFuZCBfX2FzbV9jb3B5X3RvX3VzZXIKc2hvd2luZyBoaWdoIGNwdSB1c2Fn ZSwgYW5kIHRoaXMgaXMgdGhlIHJlYXNvbiBJIGxpc3RlZCB0aGUgdHdvLgoKSW5pdGlhbGx5IHRo aXMgZGlzY3Vzc2lvbiBzdGFydGVkIHdpdGggR2FyeSdzIG1lbWNweSBwYXRjaApvbiB0aGlzIGxp c3QuIEkgd2lsbCB3cml0ZSBtb3JlIGRldGFpbHMgYmVsbG93LgoKPiAKPj4gUHJvY2Vzc2luZyBu ZXR3b3JrIHBhY2tldHMgcmVxdWlyZSBhIGxvdCBvZiB1bmFsaWduZWQgYWNjZXNzIGZvciB0aGUg cGFja2V0Cj4+IGhlYWRlciwgd2hpY2ggaXMgbm90IGFibGUgdG8gY2hhbmdlIHRoZSBkZXNpZ24g b2YgdGhlIGhlYWRlciBmb3JtYXQgdG8gYmUKPj4gYWxpZ25lZC4KPiAKPiBJc24ndCB0aGVyZSBh biBvcHRpb24gdG8gYWxsb3cgcGFkZGluZyBvZiBuZXR3b3JrIHBhY2tldHMKPiBpbiB0aGUgc2ti dWZmIHRvIG1ha2UgdGhlIGZpZWxkcyBhbGlnbmVkIGZvciBhcmNoaXRlY3R1cmVzCj4gd2hpY2gg ZG8gbm90IGhhdmUgZWZmaWNpZW50IHVuYWxpZ25lZCBsb2FkcyAobG9va2luZyBhdCB5b3UKPiBh cm0zMikuIEhhcyB0aGlzIGJlZW4gbG9va2VkIGF0PwoKSSBhbSB0cnlpbmcgYXQgNjRiaXQgcmlz Yy12IGJlYWdsZXYgYmV0YSBib2FyZC4KTXkgdW5kZXJzdGFuZGluZyBvZiBza2J1ZmYgaXMgdGhh dCBpdCBpcyBmb3IgYWxpZ25pbmcgZGF0YSB3aGVuCnRoZSBoYW5kbGluZyBpbnNpZGUgdGhlIGtl cm5lbC4gSXQgd291bGQgaGVscCBpZiBtZW1jcHkgYW5kIApfX2FzbV9jb3B5X3RvX3VzZXIgd2Vy ZSBub3QgY2F1c2luZyBzdWNoIGEgaHVnZSBwZXJjZW50YWdlLgoKVGhpcyBwYXRjaCBpcyBhZ2Fp bnN0IGNvcHlfdG9fdXNlciBhbmQgY29weV9mcm9tX3VzZXIKcHVyZWx5IHVzZWQgZm9yIGNvcHlp bmcgYmV0d2VlbiBrZXJuZWwgc3BhY2UgYW5kIHVzZXIgc3BhY2UuCgpCeSBsb29raW5nIHRoZSBv dmVyaGVhZCBvbiBwZXJmLCB0aGUgY3B1IHVzYWdlIGlzIGluY3JlYXNpbmcKb24gY29weV90b191 c2VyIGJlY2F1c2UgdGhlIHVzZXIgYXBwIChpcGVyZjMgaGVyZSkgdXNlcyBzb2NrZXQgQVBJIAp3 aXRoIGxhcmdlIHBhY2tldCBzaXplLiBJIHVzZWQgdG8gdXNlIG1heGltdW0gYnVmZmVyIHNpemUg Cm9mIG10dSBmb3IgcmVkdWNlIHRoZSBudW1iZXIgb2YgY2FsbGluZyByZWN2Zm9ybSgpIHNlbmR0 bygpCmZvciBVRFAgcHJvZ3JhbWluZyB0b28uIEFuZCBtb3N0IG9mIHRoZSBuZXR3b3JrIHByb2dy YW1tZXIgcHJvYmFibHkKZG8gdGhlIHNhbWUuCgo+IAo+PiBBbmQgdXNlciBhcHBsaWNhdGlvbnMg Y2FsbCBzeXN0ZW0gY2FsbHMgd2l0aCBhIGxhcmdlIGJ1ZmZlciBmb3Igc2VuZC9yZWNmKCkKPj4g YW5kIHNlbmR0by9yZWN2ZnJvbSgpIHRvIHJlcGVhdCBsZXNzIGZ1bmN0aW9uIGNhbGxzIGZvciB0 aGUgb3B0aW1pemF0aW9uLgo+Pgo+PiB2MSAtPiB2MjoKPj4gLSBBZGRlZCBzaGlmdCBjb3B5Cj4+ IC0gU2VwYXJhdGVkIHBhdGNoZXMgZm9yIHJlYWRhYmlsaXR5IG9mIGNoYW5nZXMgaW4gYXNzZW1i bGVyCj4+IC0gVXNpbmcgcGVyZiByZXN1bHRzCj4+Cj4+IFsxXSBodHRwczovL2xrbWwub3JnL2xr bWwvMjAyMS8yLzE2Lzc3OAo+Pgo+PiBBa2lyYSBUc3VrYW1vdG8gKDUpOgo+PiDCoMKgIHJpc2N2 OiBfX2FzbV90by9jb3B5X2Zyb21fdXNlcjogZGVsZXRlIGV4aXN0aW5nIGNvZGUKPj4gwqDCoCBy aXNjdjogX19hc21fdG8vY29weV9mcm9tX3VzZXI6IEFkZGluZyBieXRlIGNvcHkgZmlyc3QKPj4g wqDCoCByaXNjdjogX19hc21fdG8vY29weV9mcm9tX3VzZXI6IENvcHkgdW50aWwgZHN0IGlzIGFs aWduZWQgYWRkcmVzcwo+PiDCoMKgIHJpc2N2OiBfX2FzbV90by9jb3B5X2Zyb21fdXNlcjogQnVs ayBjb3B5IHdoaWxlIHNoaWZ0aW5nIG1pc2FsaWduZWQKPj4gwqDCoMKgwqAgZGF0YQo+PiDCoMKg IHJpc2N2OiBfX2FzbV90by9jb3B5X2Zyb21fdXNlcjogQnVsayBjb3B5IHdoZW4gYm90aCBzcmMg ZHN0IGFyZQo+PiDCoMKgwqDCoCBhbGlnbmVkCj4+Cj4+IMKgwqBhcmNoL3Jpc2N2L2xpYi91YWNj ZXNzLlMgfCAxODEgKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKy0tLS0tLS0tCj4+IMKg wqAxIGZpbGUgY2hhbmdlZCwgMTQ2IGluc2VydGlvbnMoKyksIDM1IGRlbGV0aW9ucygtKQo+IAo+ IEknbSBjb25jZXJuZWQgdGhhdCBkZWxldGUgYW5kIHRoZW4gcmUtYWRkIGlzIGVpdGhlciBnb2lu ZyB0byBtYWtlCj4gdGhlIHNlcmllcyB1bi1iaXNlY3RhYmxlIG9yIGxlYXZlIGEgcG9pbnQgd2hl cmUgdGhlIGtlcm5lbCBpcyB2ZXJ5Cj4gYnJva2VuPwoKSSBjb21wbGV0ZWx5IGFncmVlIGFuZCB1 bmRlcnN0YW5kLiBUaGUgb25seSByZWFzb24gSSBzcGxpdCB0aGUgcGF0Y2hlcwppcyBiZWNhdXNl IG9mIHRoZSBjb21tZW50cyBpbiB0aGUgb3RoZXIgdGhyZWFkLiBJdCBkZWZpbml0ZWx5IGJyZWFr cwp0aGUgYmlzZWN0LiBPbmNlIHRoZSBjb250ZW50IG9mIHRoaXMgcGF0Y2ggaXMgdW5kZXJzdG9v ZCBhbmQgYWdyZWVkLApJIHdpbGwgcmUtc3BpbiB0aGVtIGluIG9uZSBwYXRjaCwgb3RoZXJ3aXNl IGl0IHdpbGwgbm90IGJvb3Qgd2hlbgpvbmx5IGluZGl2aWR1YWwgcGF0Y2ggaXMgYXBwbGllZC4K ClRoZSBHYXJ5J3MgbWVtY3B5IHBhdGNoIHdhcyBwb3N0ZWQgYSB3aGlsZSBhZ28sIGFuZCBldmVu IGl0IGhhcyB0aGUgCmJlc3QgcmVzdWx0IGluIGJlbmNoLCBpdCB3YXMgbm90IG1lcmdlZC4KCldo ZW4gd2Ugd2VyZSBtZWFzdXJpbmcgbmV0d29yayBwZXJmb3JtYW5jZSBvbiBiZWFnbGV2IGJldGEg Ym9hcmQsCnRoZSBHYXJ5J3MgbWVtY3B5IGRpZCBoYXZlIGh1Z2UgaW1wcm92ZW1lbnQuCgpUaGUg cGF0Y2ggd2FzIG5vdCBlYXN5IHRvIHJldmlldyBhbmQgdW5kZXJzdGFuZCwgYnV0IGl0IHJlYWxs eSBoZWxwcwp0aGUgbmV0d29yayBwZXJmb3JtYW5jZS4KCkJ5IHJlYWRpbmcgdGhlIGRpc2N1c3Np b24gb24gaGlzIHBhdGNoLCBJIGZlbHQgdGhlIGZpcnN0IHByaW9yaXR5CmlzIGFibGUgdG8gYmUg dW5kZXJzdG9vZCB0aGUgY2F1c2Ugb2YgY3B1IHVzYWdlIGFuZCBzcGVlZCByZXN1bHRzLgoKUGxl YXNlIHJlYWQgdGhlIGRpc2N1c3Npb24gYnkgR2FyeSwgUGFsbWVyLCBNYXR0ZW8sIG90aGVycyBh bmQgSSBpbiAKdGhlIGxpc3QuCgpNYXR0ZW8gaXMgcmV3cml0aW5nIEdhcnkncyBwYXRjaCBpbiBD LCB3aGljaCBpcyBiZXR0ZXIgZm9yIAptYWludGFpbmFiaWxpdHkgYW5kIGluY29ycG9yYXRlIHRo ZSB3aXNkb20gb2YgdGhlIG9wdGltaXphdGlvbgpvZiB0aGUgY29tcGlsZXIuClRoZSB1c2VyX2Nv cHkgYXJlIHdyaXR0ZW4gaW4gYXNzZW1ibGVyIG9yIGlubGluZSBhc3NlbWJsZXIsCmFzIEkgd3Jv dGUuCgpJIGp1c3Qgd2FudCB0byBoZWxwIG1ha2luZyBpdCBiZXR0ZXIsIHNvIG9uY2UgdGhlIGNv bnNlbnN1cyBhcmUKbWFkZSwgSSB3aWxsIG1ha2UgdGhlbSB0byBvbmUgcGF0Y2guCk9yIEkgYW0g ZmluZSBzb21lYm9keSBlbHNlIGNvbWVzIG91dCB3aXRoIGJldHRlciByZXN1bHRzLgoKTXkgYXR0 ZW1wdCB0byBkbyBzaW1pbGFyIHBhdGNoZXMgZGF0ZXMgbG9uZyB0aW1lIGFnbyBpbiAyMDAyLgpo dHRwczovL2xpbnV4LWtlcm5lbC52Z2VyLmtlcm5lbC5uYXJraXZlLmNvbS96VTZPRmxJNi9jZnQt cGF0Y2gtMi01LTQ3LWF0aGxvbi1kcnVvbi1tdWNoLWZhc3Rlci1jb3B5LXVzZXItZnVuY3Rpb24K aHR0cDovL2xrbWwuaXUuZWR1L2h5cGVybWFpbC9saW51eC9rZXJuZWwvMDIxMS4yLzA5MjguaHRt bAoKQWtpcmEKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f CmxpbnV4LXJpc2N2IG1haWxpbmcgbGlzdApsaW51eC1yaXNjdkBsaXN0cy5pbmZyYWRlYWQub3Jn Cmh0dHA6Ly9saXN0cy5pbmZyYWRlYWQub3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtcmlzY3YK