From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8CE14C43387 for ; Thu, 27 Dec 2018 11:59:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 48C3F2148D for ; Thu, 27 Dec 2018 11:59:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730929AbeL0L7U convert rfc822-to-8bit (ORCPT ); Thu, 27 Dec 2018 06:59:20 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:55038 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728213AbeL0L7T (ORCPT ); Thu, 27 Dec 2018 06:59:19 -0500 Received: from pps.filterd (m0098421.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id wBRBtxDr122284 for ; Thu, 27 Dec 2018 06:59:17 -0500 Received: from e06smtp05.uk.ibm.com (e06smtp05.uk.ibm.com [195.75.94.101]) by mx0a-001b2d01.pphosted.com with ESMTP id 2pmv5ucst7-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 27 Dec 2018 06:59:17 -0500 Received: from localhost by e06smtp05.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 27 Dec 2018 11:59:15 -0000 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp05.uk.ibm.com (192.168.101.135) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Thu, 27 Dec 2018 11:59:09 -0000 Received: from d06av21.portsmouth.uk.ibm.com (d06av21.portsmouth.uk.ibm.com [9.149.105.232]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id wBRBx8uD10224100 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Thu, 27 Dec 2018 11:59:08 GMT Received: from d06av21.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id F113952051; Thu, 27 Dec 2018 11:59:07 +0000 (GMT) Received: from oc7455500831.ibm.com (unknown [9.152.224.141]) by d06av21.portsmouth.uk.ibm.com (Postfix) with ESMTP id 858AF5204F; Thu, 27 Dec 2018 11:59:04 +0000 (GMT) Subject: Re: [PATCH v37 0/3] Virtio-balloon: support free page reporting From: Christian Borntraeger To: Wei Wang , virtio-dev@lists.oasis-open.org, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, kvm@vger.kernel.org, linux-mm@kvack.org, mst@redhat.com, mhocko@kernel.org, akpm@linux-foundation.org, dgilbert@redhat.com Cc: torvalds@linux-foundation.org, pbonzini@redhat.com, liliang.opensource@gmail.com, yang.zhang.wz@gmail.com, quan.xu0@gmail.com, nilal@redhat.com, riel@redhat.com, peterx@redhat.com, quintela@redhat.com, Halil Pasic , Cornelia Huck References: <1535333539-32420-1-git-send-email-wei.w.wang@intel.com> <0661b05a-d9d0-d374-44e8-2583463e94c2@de.ibm.com> Openpgp: preference=signencrypt Autocrypt: addr=borntraeger@de.ibm.com; prefer-encrypt=mutual; keydata= xsFNBE6cPPgBEAC2VpALY0UJjGmgAmavkL/iAdqul2/F9ONz42K6NrwmT+SI9CylKHIX+fdf J34pLNJDmDVEdeb+brtpwC9JEZOLVE0nb+SR83CsAINJYKG3V1b3Kfs0hydseYKsBYqJTN2j CmUXDYq9J7uOyQQ7TNVoQejmpp5ifR4EzwIFfmYDekxRVZDJygD0wL/EzUr8Je3/j548NLyL 4Uhv6CIPf3TY3/aLVKXdxz/ntbLgMcfZsDoHgDk3lY3r1iwbWwEM2+eYRdSZaR4VD+JRD7p8 0FBadNwWnBce1fmQp3EklodGi5y7TNZ/CKdJ+jRPAAnw7SINhSd7PhJMruDAJaUlbYaIm23A +82g+IGe4z9tRGQ9TAflezVMhT5J3ccu6cpIjjvwDlbxucSmtVi5VtPAMTLmfjYp7VY2Tgr+ T92v7+V96jAfE3Zy2nq52e8RDdUo/F6faxcumdl+aLhhKLXgrozpoe2nL0Nyc2uqFjkjwXXI OBQiaqGeWtxeKJP+O8MIpjyGuHUGzvjNx5S/592TQO3phpT5IFWfMgbu4OreZ9yekDhf7Cvn /fkYsiLDz9W6Clihd/xlpm79+jlhm4E3xBPiQOPCZowmHjx57mXVAypOP2Eu+i2nyQrkapaY IdisDQfWPdNeHNOiPnPS3+GhVlPcqSJAIWnuO7Ofw1ZVOyg/jwARAQABzTRDaHJpc3RpYW4g Qm9ybnRyYWVnZXIgKElCTSkgPGJvcm50cmFlZ2VyQGRlLmlibS5jb20+wsF4BBMBAgAiBQJO nDz4AhsDBgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAKCRARe7yAtaYcfOYVD/9sqc6ZdYKD bmDIvc2/1LL0g7OgiA8pHJlYN2WHvIhUoZUIqy8Sw2EFny/nlpPVWfG290JizNS2LZ0mCeGZ 80yt0EpQNR8tLVzLSSr0GgoY0lwsKhAnx3p3AOrA8WXsPL6prLAu3yJI5D0ym4MJ6KlYVIjU ppi4NLWz7ncA2nDwiIqk8PBGxsjdc/W767zOOv7117rwhaGHgrJ2tLxoGWj0uoH3ZVhITP1z gqHXYaehPEELDV36WrSKidTarfThCWW0T3y4bH/mjvqi4ji9emp1/pOWs5/fmd4HpKW+44tD Yt4rSJRSa8lsXnZaEPaeY3nkbWPcy3vX6qafIey5d8dc8Uyaan39WslnJFNEx8cCqJrC77kI vcnl65HaW3y48DezrMDH34t3FsNrSVv5fRQ0mbEed8hbn4jguFAjPt4az1xawSp0YvhzwATJ YmZWRMa3LPx/fAxoolq9cNa0UB3D3jmikWktm+Jnp6aPeQ2Db3C0cDyxcOQY/GASYHY3KNra z8iwS7vULyq1lVhOXg1EeSm+lXQ1Ciz3ub3AhzE4c0ASqRrIHloVHBmh4favY4DEFN19Xw1p 76vBu6QjlsJGjvROW3GRKpLGogQTLslbjCdIYyp3AJq2KkoKxqdeQYm0LZXjtAwtRDbDo71C FxS7i/qfvWJv8ie7bE9A6Wsjn87BTQROnDz4ARAAmPI1e8xB0k23TsEg8O1sBCTXkV8HSEq7 JlWz7SWyM8oFkJqYAB7E1GTXV5UZcr9iurCMKGSTrSu3ermLja4+k0w71pLxws859V+3z1jr nhB3dGzVZEUhCr3EuN0t8eHSLSMyrlPL5qJ11JelnuhToT6535cLOzeTlECc51bp5Xf6/XSx SMQaIU1nDM31R13o98oRPQnvSqOeljc25aflKnVkSfqWSrZmb4b0bcWUFFUKVPfQ5Z6JEcJg Hp7qPXHW7+tJTgmI1iM/BIkDwQ8qe3Wz8R6rfupde+T70NiId1M9w5rdo0JJsjKAPePKOSDo RX1kseJsTZH88wyJ30WuqEqH9zBxif0WtPQUTjz/YgFbmZ8OkB1i+lrBCVHPdcmvathknAxS bXL7j37VmYNyVoXez11zPYm+7LA2rvzP9WxR8bPhJvHLhKGk2kZESiNFzP/E4r4Wo24GT4eh YrDo7GBHN82V4O9JxWZtjpxBBl8bH9PvGWBmOXky7/bP6h96jFu9ZYzVgIkBP3UYW+Pb1a+b w4A83/5ImPwtBrN324bNUxPPqUWNW0ftiR5b81ms/rOcDC/k/VoN1B+IHkXrcBf742VOLID4 YP+CB9GXrwuF5KyQ5zEPCAjlOqZoq1fX/xGSsumfM7d6/OR8lvUPmqHfAzW3s9n4lZOW5Jfx bbkAEQEAAcLBXwQYAQIACQUCTpw8+AIbDAAKCRARe7yAtaYcfPzbD/9WNGVf60oXezNzSVCL hfS36l/zy4iy9H9rUZFmmmlBufWOATjiGAXnn0rr/Jh6Zy9NHuvpe3tyNYZLjB9pHT6mRZX7 Z1vDxeLgMjTv983TQ2hUSlhRSc6e6kGDJyG1WnGQaqymUllCmeC/p9q5m3IRxQrd0skfdN1V AMttRwvipmnMduy5SdNayY2YbhWLQ2wS3XHJ39a7D7SQz+gUQfXgE3pf3FlwbwZhRtVR3z5u aKjxqjybS3Ojimx4NkWjidwOaUVZTqEecBV+QCzi2oDr9+XtEs0m5YGI4v+Y/kHocNBP0myd pF3OoXvcWdTb5atk+OKcc8t4TviKy1WCNujC+yBSq3OM8gbmk6NwCwqhHQzXCibMlVF9hq5a FiJb8p4QKSVyLhM8EM3HtiFqFJSV7F+h+2W0kDyzBGyE0D8z3T+L3MOj3JJJkfCwbEbTpk4f n8zMboekuNruDw1OADRMPlhoWb+g6exBWx/YN4AY9LbE2KuaScONqph5/HvJDsUldcRN3a5V RGIN40QWFVlZvkKIEkzlzqpAyGaRLhXJPv/6tpoQaCQQoSAc5Z9kM/wEd9e2zMeojcWjUXgg oWj8A/wY4UXExGBu+UCzzP/6sQRpBiPFgmqPTytrDo/gsUGqjOudLiHQcMU+uunULYQxVghC syiRa+UVlsKmx1hsEg== Date: Thu, 27 Dec 2018 12:59:04 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 In-Reply-To: <0661b05a-d9d0-d374-44e8-2583463e94c2@de.ibm.com> Content-Language: en-US X-TM-AS-GCONF: 00 x-cbid: 18122711-0020-0000-0000-000002FCDF1B X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18122711-0021-0000-0000-0000214D0ECD Message-Id: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8BIT MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2018-12-27_09:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1812270110 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 27.12.2018 12:31, Christian Borntraeger wrote: > This patch triggers random crashes in the guest kernel on s390 early during boot. > No migration and no setting of the balloon is involved. > Adding Conny and Halil, As the QEMU provides no PAGE_HINT feature yet, this quick hack makes the guest boot fine again: diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c index 728ecd1eea305..aa2e1864c5736 100644 --- a/drivers/virtio/virtio_balloon.c +++ b/drivers/virtio/virtio_balloon.c @@ -492,7 +492,7 @@ static int init_vqs(struct virtio_balloon *vb) callbacks[VIRTIO_BALLOON_VQ_FREE_PAGE] = NULL; } - err = vb->vdev->config->find_vqs(vb->vdev, VIRTIO_BALLOON_VQ_MAX, + err = vb->vdev->config->find_vqs(vb->vdev, 3, //VIRTIO_BALLOON_VQ_MAX, vqs, callbacks, names, NULL, NULL); if (err) return err; To me it looks like that virtio_ccw_find_vqs will abort if any of the virtqueues that it is been asked for does not exist (including the earlier ones). Christian > > On 27.08.2018 03:32, Wei Wang wrote: >> The new feature, VIRTIO_BALLOON_F_FREE_PAGE_HINT, implemented by this >> series enables the virtio-balloon driver to report hints of guest free >> pages to host. It can be used to accelerate virtual machine (VM) live >> migration. Here is an introduction of this usage: >> >> Live migration needs to transfer the VM's memory from the source machine >> to the destination round by round. For the 1st round, all the VM's memory >> is transferred. From the 2nd round, only the pieces of memory that were >> written by the guest (after the 1st round) are transferred. One method >> that is popularly used by the hypervisor to track which part of memory is >> written is to have the hypervisor write-protect all the guest memory. >> >> This feature enables the optimization by skipping the transfer of guest >> free pages during VM live migration. It is not concerned that the memory >> pages are used after they are given to the hypervisor as a hint of the >> free pages, because they will be tracked by the hypervisor and transferred >> in the subsequent round if they are used and written. >> >> * Tests >> 1 Test Environment >> Host: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz >> Migration setup: migrate_set_speed 100G, migrate_set_downtime 400ms >> >> 2 Test Results (results are averaged over several repeated runs) >> 2.1 Guest setup: 8G RAM, 4 vCPU >> 2.1.1 Idle guest live migration time >> Optimization v.s. Legacy = 620ms vs 2970ms >> --> ~79% reduction >> 2.1.2 Guest live migration with Linux compilation workload >> (i.e. make bzImage -j4) running >> 1) Live Migration Time: >> Optimization v.s. Legacy = 2273ms v.s. 4502ms >> --> ~50% reduction >> 2) Linux Compilation Time: >> Optimization v.s. Legacy = 8min42s v.s. 8min43s >> --> no obvious difference >> >> 2.2 Guest setup: 128G RAM, 4 vCPU >> 2.2.1 Idle guest live migration time >> Optimization v.s. Legacy = 5294ms vs 41651ms >> --> ~87% reduction >> 2.2.2 Guest live migration with Linux compilation workload >> 1) Live Migration Time: >> Optimization v.s. Legacy = 8816ms v.s. 54201ms >> --> 84% reduction >> 2) Linux Compilation Time: >> Optimization v.s. Legacy = 8min30s v.s. 8min36s >> --> no obvious difference >> >> ChangeLog: >> v36->v37: >> - free the reported pages to mm when receives a DONE cmd from host. >> Please see patch 1's commit log for reasons. Please see patch 1's >> commit for detailed explanations. >> >> For ChangeLogs from v22 to v36, please reference >> https://lkml.org/lkml/2018/7/20/199 >> >> For ChangeLogs before v21, please reference >> https://lwn.net/Articles/743660/ >> >> Wei Wang (3): >> virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_HINT >> mm/page_poison: expose page_poisoning_enabled to kernel modules >> virtio-balloon: VIRTIO_BALLOON_F_PAGE_POISON >> >> drivers/virtio/virtio_balloon.c | 374 ++++++++++++++++++++++++++++++++---- >> include/uapi/linux/virtio_balloon.h | 8 + >> mm/page_poison.c | 6 + >> 3 files changed, 355 insertions(+), 33 deletions(-) >>