From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.2 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC083C73C7C for ; Wed, 10 Jul 2019 09:15:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8DF382054F for ; Wed, 10 Jul 2019 09:15:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727645AbfGJJPK (ORCPT ); Wed, 10 Jul 2019 05:15:10 -0400 Received: from foss.arm.com ([217.140.110.172]:58178 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726580AbfGJJPK (ORCPT ); Wed, 10 Jul 2019 05:15:10 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CA65A344; Wed, 10 Jul 2019 02:15:08 -0700 (PDT) Received: from [10.1.197.61] (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 49BF13F246; Wed, 10 Jul 2019 02:15:07 -0700 (PDT) Subject: Re: [RFC PATCH v2 0/3] Support CPU hotplug for ARM64 To: Maran Wilson , James Morse , Xiongfeng Wang Cc: guohanjun@huawei.com, john.garry@huawei.com, rjw@rjwysocki.net, linux-kernel@vger.kernel.org, linux-acpi@vger.kernel.org, huawei.libin@huawei.com, jonathan.cameron@huawei.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org References: <1561776155-38975-1-git-send-email-wangxiongfeng2@huawei.com> <82879258-46a7-a6e9-ee54-fc3692c1cdc3@arm.com> <51cc9a5c-9968-c4b1-0bc7-870f44a3a761@oracle.com> From: Marc Zyngier Openpgp: preference=signencrypt Autocrypt: addr=marc.zyngier@arm.com; prefer-encrypt=mutual; keydata= mQINBE6Jf0UBEADLCxpix34Ch3kQKA9SNlVQroj9aHAEzzl0+V8jrvT9a9GkK+FjBOIQz4KE g+3p+lqgJH4NfwPm9H5I5e3wa+Scz9wAqWLTT772Rqb6hf6kx0kKd0P2jGv79qXSmwru28vJ t9NNsmIhEYwS5eTfCbsZZDCnR31J6qxozsDHpCGLHlYym/VbC199Uq/pN5gH+5JHZyhyZiNW ozUCjMqC4eNW42nYVKZQfbj/k4W9xFfudFaFEhAf/Vb1r6F05eBP1uopuzNkAN7vqS8XcgQH qXI357YC4ToCbmqLue4HK9+2mtf7MTdHZYGZ939OfTlOGuxFW+bhtPQzsHiW7eNe0ew0+LaL 3wdNzT5abPBscqXWVGsZWCAzBmrZato+Pd2bSCDPLInZV0j+rjt7MWiSxEAEowue3IcZA++7 ifTDIscQdpeKT8hcL+9eHLgoSDH62SlubO/y8bB1hV8JjLW/jQpLnae0oz25h39ij4ijcp8N t5slf5DNRi1NLz5+iaaLg4gaM3ywVK2VEKdBTg+JTg3dfrb3DH7ctTQquyKun9IVY8AsxMc6 lxl4HxrpLX7HgF10685GG5fFla7R1RUnW5svgQhz6YVU33yJjk5lIIrrxKI/wLlhn066mtu1 DoD9TEAjwOmpa6ofV6rHeBPehUwMZEsLqlKfLsl0PpsJwov8TQARAQABtCNNYXJjIFp5bmdp ZXIgPG1hcmMuenluZ2llckBhcm0uY29tPokCTwQTAQIAOQIbAwYLCQgHAwIGFQgCCQoLBBYC AwECHgECF4AWIQSf1RxT4LVjGP2VnD0j0NC60T16QwUCXR3BUgAKCRAj0NC60T16Qyd/D/9s x0puxd3lI+jdLMEY8sTsNxw/+CZfyKaHtysasZlloLK7ftYhRUc63mMW2mrvgB1GEnXYIdj3 g6Qo4csoDuN+9EBmejh7SglM/h0evOtrY2V5QmZA/e/Pqfj0P3N/Eb5BiB3R4ptLtvKCTsqr 3womxCRqQY3IrMn1s2qfpmeNLUIfCUtgh8opzPtFuFJWVBzbzvhPEApZzMe9Vs1O2P8BQaay QXpbzHaKruthoLICRzS/3UCe0N/mBZQRKHrqhPwvjZdO0KMqjSsPqfukOJ8bl5jZxYk+G/3T 66Z4JUpZ7RkcrX7CvBfZqRo19WyWFfjGz79iVMJNIEkJvJBANbTSiWUC6IkP+zT/zWYzZPXx XRlrKWSBBqJrWQKZBwKOLsL62oQG7ARvpCG9rZ6hd5CLQtPI9dasgTwOIA1OW2mWzi20jDjD cGC9ifJiyWL8L/bgwyL3F/G0R1gxAfnRUknyzqfpLy5cSgwKCYrXOrRqgHoB+12HA/XQUG+k vKW8bbdVk5XZPc5ghdFIlza/pb1946SrIg1AsjaEMZqunh0G7oQhOWHKOd6fH0qg8NssMqQl jLfFiOlgEV2mnaz6XXQe/viXPwa4NCmdXqxeBDpJmrNMtbEbq+QUbgcwwle4Xx2/07ICkyZH +7RvbmZ/dM9cpzMAU53sLxSIVQT5lj23WLkCDQROiX9FARAAz/al0tgJaZ/eu0iI/xaPk3DK NIvr9SsKFe2hf3CVjxriHcRfoTfriycglUwtvKvhvB2Y8pQuWfLtP9Hx3H+YI5a78PO2tU1C JdY5Momd3/aJBuUFP5blbx6n+dLDepQhyQrAp2mVC3NIp4T48n4YxL4Og0MORytWNSeygISv Rordw7qDmEsa7wgFsLUIlhKmmV5VVv+wAOdYXdJ9S8n+XgrxSTgHj5f3QqkDtT0yG8NMLLmY kZpOwWoMumeqn/KppPY/uTIwbYTD56q1UirDDB5kDRL626qm63nF00ByyPY+6BXH22XD8smj f2eHw2szECG/lpD4knYjxROIctdC+gLRhz+Nlf8lEHmvjHgiErfgy/lOIf+AV9lvDF3bztjW M5oP2WGeR7VJfkxcXt4JPdyDIH6GBK7jbD7bFiXf6vMiFCrFeFo/bfa39veKUk7TRlnX13go gIZxqR6IvpkG0PxOu2RGJ7Aje/SjytQFa2NwNGCDe1bH89wm9mfDW3BuZF1o2+y+eVqkPZj0 mzfChEsiNIAY6KPDMVdInILYdTUAC5H26jj9CR4itBUcjE/tMll0n2wYRZ14Y/PM+UosfAhf YfN9t2096M9JebksnTbqp20keDMEBvc3KBkboEfoQLU08NDo7ncReitdLW2xICCnlkNIUQGS WlFVPcTQ2sMAEQEAAYkCHwQYAQIACQUCTol/RQIbDAAKCRAj0NC60T16QwsFD/9T4y30O0Wn MwIgcU8T2c2WwKbvmPbaU2LDqZebHdxQDemX65EZCv/NALmKdA22MVSbAaQeqsDD5KYbmCyC czilJ1i+tpZoJY5kJALHWWloI6Uyi2s1zAwlMktAZzgGMnI55Ifn0dAOK0p8oy7/KNGHNPwJ eHKzpHSRgysQ3S1t7VwU4mTFJtXQaBFMMXg8rItP5GdygrFB7yUbG6TnrXhpGkFBrQs9p+SK vCqRS3Gw+dquQ9QR+QGWciEBHwuSad5gu7QC9taN8kJQfup+nJL8VGtAKgGr1AgRx/a/V/QA ikDbt/0oIS/kxlIdcYJ01xuMrDXf1jFhmGZdocUoNJkgLb1iFAl5daV8MQOrqciG+6tnLeZK HY4xCBoigV7E8KwEE5yUfxBS0yRreNb+pjKtX6pSr1Z/dIo+td/sHfEHffaMUIRNvJlBeqaj BX7ZveskVFafmErkH7HC+7ErIaqoM4aOh/Z0qXbMEjFsWA5yVXvCoJWSHFImL9Bo6PbMGpI0 9eBrkNa1fd6RGcktrX6KNfGZ2POECmKGLTyDC8/kb180YpDJERN48S0QBa3Rvt06ozNgFgZF Wvu5Li5PpY/t/M7AAkLiVTtlhZnJWyEJrQi9O2nXTzlG1PeqGH2ahuRxn7txA5j5PHZEZdL1 Z46HaNmN2hZS/oJ69c1DI5Rcww== Organization: ARM Ltd Message-ID: <06ef13e1-fffe-d4a2-721e-f666f331fb3c@arm.com> Date: Wed, 10 Jul 2019 10:15:05 +0100 User-Agent: Mozilla/5.0 (X11; Linux aarch64; rv:60.0) Gecko/20100101 Thunderbird/60.7.2 MIME-Version: 1.0 In-Reply-To: <51cc9a5c-9968-c4b1-0bc7-870f44a3a761@oracle.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: linux-acpi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org On 09/07/2019 20:06, Maran Wilson wrote: > On 7/5/2019 3:12 AM, James Morse wrote: >> Hi guys, >> >> (CC: +kvmarm list) >> >> On 29/06/2019 03:42, Xiongfeng Wang wrote: >>> This patchset mark all the GICC node in MADT as possible CPUs even though it >>> is disabled. But only those enabled GICC node are marked as present CPUs. >>> So that kernel will initialize some CPU related data structure in advance before >>> the CPU is actually hot added into the system. This patchset also implement >>> 'acpi_(un)map_cpu()' and 'arch_(un)register_cpu()' for ARM64. These functions are >>> needed to enable CPU hotplug. >>> >>> To support CPU hotplug, we need to add all the possible GICC node in MADT >>> including those CPUs that are not present but may be hot added later. Those >>> CPUs are marked as disabled in GICC nodes. >> ... what do you need this for? >> >> (The term cpu-hotplug in the arm world almost never means hot-adding a new package/die to >> the platform, we usually mean taking CPUs online/offline for power management. e.g. >> cpuhp_offline_cpu_device()) >> >> It looks like you're adding support for hot-adding a new package/die to the platform ... >> but only for virtualisation. >> >> I don't see why this is needed for virtualisation. The in-kernel irqchip needs to know >> these vcpu exist before you can enter the guest for the first time. You can't create them >> late. At best you're saving the host scheduling a vcpu that is offline. Is this really a >> problem? >> >> If we moved PSCI support to user-space, you could avoid creating host vcpu threads until >> the guest brings the vcpu online, which would solve that problem, and save the host >> resources for the thread too. (and its acpi/dt agnostic) >> >> I don't see the difference here between booting the guest with 'maxcpus=1', and bringing >> the vcpu online later. The only real difference seems to be moving the can-be-online >> policy into the hypervisor/VMM... > > Isn't that an important distinction from a cloud service provider's > perspective? > > As far as I understand it, you also need CPU hotplug capabilities to > support things like Kata runtime under Kubernetes. i.e. when > implementing your containers in the form of light weight VMs for the > additional security ... and the orchestration layer cannot determine > ahead of time how much CPU/memory resources are going to be needed to > run the pod(s). Why would it be any different? You can pre-allocate your vcpus, leave them parked until some external agent decides to signal the container that it it can use another bunch of CPUs. At that point, the container must actively boot these vcpus (they aren't going to come up by magic). Given that you must have sized your virtual platform to deal with the maximum set of resources you anticipate (think of the GIC redistributors, for example), I really wonder what you gain here. > > Thanks, > -Maran > >> >> I think physical package/die hotadd is a much bigger, uglier problem than doing the same >> under virtualisation. Its best to do this on real hardware first so we don't miss >> something. (cpu-topology, numa, memory, errata, timers?) >> I'm worried that doing virtualisation first means the firmware-requirements for physical >> hotadd stuff is "whatever Qemu does". For sure, I want to model the virtualization side after the actual HW, and not the other way around. Live reconfiguration of the interrupt topology (and thus the whole memory map) will certainly be challenging. Thanks, M. -- Jazz is not dead. It just smells funny...