From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1760076AbcCDUse (ORCPT ); Fri, 4 Mar 2016 15:48:34 -0500 Received: from mail-bn1bon0093.outbound.protection.outlook.com ([157.56.111.93]:30208 "EHLO na01-bn1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1758993AbcCDUsc (ORCPT ); Fri, 4 Mar 2016 15:48:32 -0500 Authentication-Results: redhat.com; dkim=none (message not signed) header.d=none;redhat.com; dmarc=none action=none header.from=amd.com; From: Suravee Suthikulpanit To: , , , , , CC: , , , , Suravee Suthikulpanit Subject: [PART1 RFC v2 00/10] KVM: x86: Introduce SVM AVIC support Date: Fri, 4 Mar 2016 14:45:58 -0600 Message-ID: <1457124368-2025-1-git-send-email-Suravee.Suthikulpanit@amd.com> X-Mailer: git-send-email 1.9.1 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-Originating-IP: [124.121.8.20] X-ClientProxiedBy: SINPR01CA0045.apcprd01.prod.exchangelabs.com (10.141.109.45) To SN1PR12MB0445.namprd12.prod.outlook.com (25.162.105.139) X-MS-Office365-Filtering-Correlation-Id: aa499cfa-2419-4e9e-a19d-08d3446e569d X-Microsoft-Exchange-Diagnostics: 1;SN1PR12MB0445;2:ltgO0DgTTE/xFoHcK3ZeNRdGDiOYbAL3XiZst6Vku0YZpOdV8REu3lds2OzUP9PH4LCjw/+FntsmWZ36D0NKixg6weONKHlZuFO9d/b4Qcq7cWkL9lsFOD6SPvpT9KLAajjlJPYchUj6+w7Fcuu6QrXINfC/oSmH/0Rqq/j7QUZSGrvE8vRZzjeSfJ7fwm2C;3:4D36PAIQLCtebPezu5VECVK0RB8wqmb4PWq4rMB2hikiRtEXxN2FCsMLDuSM9nv8T4E/UTwHP3o6FU/c5nhmr+gsywyH/jPBhMX0jAZfDW8TmnarYPow9xqmCs6Q7DDf;25:yHWHCg80sqPFax+rz6ZYkp/ShPj+iAXRtLf6iXvbW5k9oVvHE85x5Sk36ScG8eQMeqbyXeSrjLHqUOgjOZkUMC7B4GAtRs7/icSVDVw9sjc5KLJUCAFssIeiGSBOxgDo2eh/1ynh7kBsdbzQvMVP+r+1NuERb1+PShhOwZpkpXjsDBhk/IybydSJviWqZ7C3ShFGDyMYGox840heYDKuvtprRz3TPxoNU2BRC4kdWk5n9FOeNPRlLXYrFwfTfhl/usB8ylFBqsRlvjiBUBtz3JIjkX5nYx7rZ9PBDSNt4RnwpZTVTkkyW1HrkQMiv8iHgUz/X5iPja5CJyDJ+1mOQA== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:SN1PR12MB0445; X-Microsoft-Exchange-Diagnostics: 1;SN1PR12MB0445;20:mpomiJ40TFs/+ktRzL3VZhueNWepjSjwYPlPLG+kbDx0+Wn8DhhtHKT7IrNZ/LFDFqYqv6XzF1z7+T+URZmB24rOsbx/oEjj01SjyzRCO+L8A7/vZRuxbyxgsVvD5qn88UlotMk5Sf+AdTltU5FihrASUaIy1Q5z2APdOheaU+0lQDrNWkSz6KV3Zv1GPgCwJg3X7BUVhCVkyQyIlJKq3HHO3Kp8WUH2tYSjytdmC5gpi9ly5+9XwOban0GAXQkH9hOjqQqpbJKZgZctILgB0ySo/mQHapwEQtlIy9+xCUmUb54lox1ZzJphg6erxXdA36xLxLzwIY/mUHVWKybzGezeh675ewwTD0wZqyQASq6M8LkIXKx9ZwDfRFGbJDg3goo5M0eSU6E9lNC/COjU0Uwi8bsbMIIrVgoGjxkTe4cM1tcGzCsG0uXAo2OV0Wv+lKfrMliXQZLgjYxjYkK56yTtIRvYI0qbGGKF5LOnj2uGyVUvcP7A2OltFbh+k0Rm;4:Hqwi898e0fyzPtLeDAM2rfmX5iohtO1S5ojoBy7F6cRSS2VtabxIj1jRbikuBfNswxlaBXkrwr94P6c/HKVgaFbinoTadWohqj4Eah4f0llnJ6INvodeKsj2fdmegZVHu08DXhnEQbvqdeY8CWNx1Mm4/Azt58YqiKfL0aQ72mO9kDp1b4b97v43hyZX/0blFgHYTFGJvFVP5rPPkgG3EDakBhdjQR/4WmgIrrHwNqN6qxj0gcfQSTAd/lKUR473YCn6cMDfb+6/sDaevKARlQN4PHi3CCnHlgmHYk1vfEZB3q1JJd6OJJk6ZMPkiNW5v7PtLXY1+jr9Q6Af2SBNci/Qh4GDNmnPF2C8GBodozM2pLuuJa0UEvMh4lhZSYF+ X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(5005006)(8121501046)(3002001)(10201501046);SRVR:SN1PR12MB0445;BCL:0;PCL:0;RULEID:;SRVR:SN1PR12MB0445; X-Forefront-PRVS: 0871917CDA X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4630300001)(6069001)(6009001)(243025005)(15975445007)(5820100001)(77096005)(5001770100001)(87976001)(47776003)(229853001)(36756003)(2201001)(66066001)(81166005)(50466002)(23676002)(2870700001)(5004730100002)(189998001)(40100003)(50226001)(122386002)(5008740100001)(15395725005)(2906002)(19580395003)(86362001)(19273905006)(4326007)(42186005)(50986999)(6116002)(3846002)(1096002)(92566002)(586003)(19300405004)(562404015)(217873001)(563064011);DIR:OUT;SFP:1101;SCL:1;SRVR:SN1PR12MB0445;H:localhost.localdomain;FPR:;SPF:None;MLV:sfv;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtTTjFQUjEyTUIwNDQ1OzIzOkRvd0xxWGpRUE96M01nTGlIWDU0NEVXZXBO?= =?utf-8?B?eml0VGNBK1JOTzJOWHpNL2dMQWkrcm44VE9IcDhtakE4Ym1GczhYYnlSeFFR?= =?utf-8?B?elYzTXg1eEd6M3VrTmRraWZ5N1Q3bTBMWXdJWldwbVJOaWd5LzU3OGFQRnhr?= =?utf-8?B?K2FOY2xQZTI2T3pQZUNjVmkwa2ZaYWJ5Yk1jOGEwMHNEZEhBMUVYSU5rdThv?= =?utf-8?B?V0lVc3kyZkNEdU1zTm9ZK1A3RTBIN1dudHZNVDBQVEF0MHNhRmcrTUhNeXJH?= =?utf-8?B?UTkrVG5GempQem91bU5GRUtWcE10eWgrMnkzdmVlaks2MHdvM2ovVy95dFZz?= =?utf-8?B?bnArdjhuRzBJOFpNR0k4L1NJT3N5YkFmaEZtQVVUREVKdGlZMGRZTGFjQ2ky?= =?utf-8?B?Nm0vTlFscyt6TmRLdnZhNU1LK0ZxblIzUlgyQytrRnlUK1B3djBGenp4RDlF?= =?utf-8?B?VGxGZldTUGdrZFppUTR3T0JpcmlQd09DdUxCZ2hqU3B4ZkFLRGplZmZ3clFP?= =?utf-8?B?Ni95a3BMUFFNUURsREhLNUR3bU5iQ2QrNjNSSy9YZUhmUnN6cEd1K3ZGZmUx?= =?utf-8?B?Z0R4Tkd2YXhudStIQ3ZxL1ZOMUVvTkUwNDhDNmZtRncyeklXMElnMjFoQXll?= =?utf-8?B?K0RpQ2Y5RlJwazJmZmN3cWZCZHNwdTVuenh2NDV0Y1hGTVZULzBUQkIxc3I4?= =?utf-8?B?cnFCeDRZeEJUYmFrV3ZrRFJpaFdXbHQ2OU9oZThCNGZjZXNsSHl5NmVIcitr?= =?utf-8?B?RktqOWtMcGJPSGpZOVd2a2VSYTNTeUZGbWROUDgwS2R6ekhsV1BhRDF6bGN3?= =?utf-8?B?U1Z2cVZ5RjBiY2dkK29YeGhubFdHWHhOZlBtazkwb2ZlR3VGclZoaVoxRzV5?= =?utf-8?B?M242eGpoK3c1TVF6eDlJYTZYS24xQUphdithNXl5WWZHQVJuenJObHBCRVNB?= =?utf-8?B?UXNwNDBiOG5vMVE0RDJzSVY2d3gwOVlvbGZWMXF4b2U0Z09zZHZqUUc2Y3NJ?= =?utf-8?B?cXhwakRLNm05NDhyWmU5K3RldW5OWlZCRHpjc3dJVVdZVnhLd1V2bnQ5N0hs?= =?utf-8?B?RzYveEd1cC9OajIycWI4NFVXM0szY3lseWVFQyt6WENNbnJncHJzVXlQRy9n?= =?utf-8?B?QVFnd1BoRkREZVhpQ2FiVG9ocVpRZk9vN3QwWnAySWlqNTc5WC8vQXlIVkht?= =?utf-8?B?d3ZlcmNPNmd5MjlCT1VDanlzWGNsRjBlek5maW1ZeW1UcWNlZnVPUGxMNXVZ?= =?utf-8?B?Zjk0b2dGRXR6d1F6aWRJOFFYbXhxc0sxSDN1ZTZBL0w5bERsV29ZVDJsREFI?= =?utf-8?B?Y0VUQ2dLZ2V1SUx3YUN5ZTJzTHF0L2cwNk10S1J1cnowTFlEWlVWdElxQnRS?= =?utf-8?B?VTAvbFcrV2ZPY0pYeVlKNXkwQXhWelozei92eFRrTURjKzBvbkRrQTBIS0hD?= =?utf-8?B?ZHFOL3QrRlcvTUlKbG1XRU1sUkl4bzJJODNycVlWNUQ2Mzg2VS92Tk1WNVZ1?= =?utf-8?B?Y0xxdz09?= X-Microsoft-Exchange-Diagnostics: 1;SN1PR12MB0445;5:6IhwI3wZVzTMGUiGjH84Zu8DZu2pWdRKBibLTXEqB11D6GUPv6rW6Y9cCMlY6t+l1gic6CcvH6tO0oIF7nrxe0Mb2xkLLahkjRiImSB5H3VTMZ5ayAwhk9ZJ0j6Ze0/dtSsogDAM5WUel5BWVpjhMA==;24:u+eadpBtazeaO0BI0PXE4KIsmza5/hxiLWMIjqhNFKBIdlc5YXHEADn55XB7BJTNSzaZhOlVvesG6dgLpwfvF07C6TRHRe633r/+2pGCvIs=;20:Al+FfUyibfF8BLqehTBwwG1/xNAJNwUGc2cK+En0Bny2Ha4pYlGwnMEtGv3eykd6Rk59G94lbn9T5NYhXq3tySfkntoKYhQ5uJDcefscS/8Al45kJ7Ydtv7GvgccG2bEYVX2Utp20og6cwAOkrtEpLys7SPYP17jZTjZ+qVedwUaf0DCJFLjUnRVjvLl090MwsX8S3pTIP289kDskM3pTgvQlPTj0I2HDbp02P5XGyLMLbBAsog2EGX7ECteAjaA X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Mar 2016 20:48:24.7601 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN1PR12MB0445 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org CHANGES FROM RFCv1: ================== * Rebased from tip.git * Use the vAPIC backing page as the emulated LAPIC register page. * Clarified with HW engineer that Avic sets the IRR of all the cores targeted whether they are running or not. It sends doorbells to the ones that are running to get them to evaluate their new state. If one or more weren't running it does the VMEXIT to indicate that the host needs to go run the not running guests. It gives it the ID of one of the guests that it observed not running which should be enough of a hint for the hypervisor to track them all down. * Rewrite the logic for handling AVIC_INCOMPLETE_IPI #vmexit for IPI target not running case based on information above. * Rewrite the logic in avic_vcpu_load() and avic_set_running(). * Rewrite the interrupt injection to remove the avic_pending_cnt variable. GITHUB ====== Latest git tree can be found at: http://github.com/ssuthiku/linux.git avic_part1_rfc_v2 OVERVIEW ======== This patch set is the first of the two-part patch series to introduce the new AMD Advance Virtual Interrupt Controller (AVIC) support. Basically, SVM AVIC hardware virtualizes local APIC registers of each vCPU via the virtual APIC (vAPIC) backing page. This allows guest access to certain APIC registers without the need to emulate the hardware behavior in the hypervisor. More information about AVIC can be found in the AMD64 Architecture Programmer’s Manual Volume 2 - System Programming. http://support.amd.com/TechDocs/24593.pdf For SVM AVIC, we extend the existing kvm_amd driver to: * Check CPUID to detect AVIC support in the processor * Program new fields in VMCB to enable AVIC * Introduce new AVIC data structures and add code to manage them * Handle two new AVIC #VMEXITs * Add new interrupt intjection code using vAPIC backing page instead of the existing V_IRQ, V_INTR_PRIO, V_INTR_VECTOR, and V_IGN_TPR fields Currently, this patch series does not enable AVIC by default. Users can enable SVM AVIC by specifying avic=1 during insmod kvm-amd. Later, in part 2, we will introduce the IOMMU AVIC support, which provides speed up for PCI device pass-through use case by allowing the IOMMU hardware to inject interrupt directly into the guest via the vAPIC backing page. PERFORMANCE RESULTS =================== Currently, AVIC is supported in the AMD family 15h models 6Xh (Carrizo) processors. Therefore, it is used to collect the perforamance data shown below. Generaly, SVM AVIC alone (w/o IOMMU AVIC) should provide speedup for IPI interrupt since hypervisor does not require VMEXIT to inject these interrupts. Also, it should speed up the case when hypervisor wants to inject an interrupt into a running guest by setting the corresponded IRR bit in the vAPIC backing page and trigger AVIC_DOORBELL MSR. IPI PERFORMANCE =============== * BENCHMARK 1: HACKBENCH For IPI, I have collected some performance number on 2 and 4 CPU running hackbech with the following detail: hackbench -p -l 100000 Running in process mode with 10 groups using 40 file descriptors each (== 400 tasks) Each sender will pass 100000 messages of 100 bytes | 2 vcpus | 4 vcpus ------------------------------------------------ Vanila | 273.76 | 190.21 AVIC disabled | 260.51 (~5%) | 184.40 (~5%) AVIC | 248.53 (~10%) | 155.01 (~20%) OVERALL PERFORMANCE =================== Enabling AVIC should helps speeding up workloads, which generate large amount of interrupts. However, it requires additional logics to maintain AVIC-specific data structures during vCPU load/unload due to vcpu scheduling. The goal is to minimize the overhead of AVIC in most cases, so that we can achieve equivalent or improvement in overall performance when enabling AVIC. * BENCHMARK 1: TAR DECOMPRESSION This test measures the average running time (of 10 runs) of the following tar decompression command with 1, 2, and 4 vcpus. tar xf linux-4.3.3.tar.xz | 4 vcpus --------------------------------- Vanila | 10.26 AVIC disabled | 10.10 (~1.5%) AVIC | 10.07 (~1.8%) Note: The unit of result below is in seconds (lower is better). * BENCHMARK 2: NETPERF w/ virtual network This test creates a virtual network by setting up bridge and tap device on the host and pass it into the VM as virtio-net-pci device w/ vhost. Then it sets up netserver in the host machine, and run netperf in the VM with following option: netperf -H -l 60 -t TCP_RR -D 2 | 1 vcpu ------------------------------------ Vanila | 21623.887 AVIC disabled | 21538.09 (~-.4%) AVIC | 21712.68 (~0.4%) Note: The unit of result below is trans/sec (higher is better). Preliminary result of both benchmarks show AVIC performance are slightly better than the other two cases. CURRENT UNTESTED USE-CASES =========================== - VM Migration (work in progress) - Nested VM Any feedback and comments are very much appreciated. Thank you, Suravee Suravee Suthikulpanit (10): KVM: x86: Misc LAPIC changes to exposes helper functions KVM: x86: Introducing kvm_x86_ops VCPU blocking/unblocking svm: Introduce new AVIC VMCB registers svm: clean up V_TPR, V_IRQ, V_INTR_PRIO, and V_INTR_MASKING KVM: x86: Detect and Initialize AVIC support svm: Add interrupt injection via AVIC svm: Add VMEXIT handlers for AVIC svm: Do not expose x2APIC when enable AVIC svm: Do not intercept CR8 when enable AVIC svm: Manage vcpu load/unload when enable AVIC arch/x86/include/asm/kvm_host.h | 19 +- arch/x86/include/asm/svm.h | 38 +- arch/x86/include/uapi/asm/svm.h | 9 +- arch/x86/kvm/lapic.c | 51 ++- arch/x86/kvm/lapic.h | 7 + arch/x86/kvm/svm.c | 937 +++++++++++++++++++++++++++++++++++++++- 6 files changed, 1003 insertions(+), 58 deletions(-) -- 1.9.1