From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.7 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A389C433DF for ; Fri, 15 May 2020 02:58:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 81ADA20734 for ; Fri, 15 May 2020 02:58:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728275AbgEOC6Y convert rfc822-to-8bit (ORCPT ); Thu, 14 May 2020 22:58:24 -0400 Received: from out30-133.freemail.mail.aliyun.com ([115.124.30.133]:43156 "EHLO out30-133.freemail.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727029AbgEOC6X (ORCPT ); Thu, 14 May 2020 22:58:23 -0400 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R641e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e07484;MF=teawaterz@linux.alibaba.com;NM=1;PH=DS;RN=41;SR=0;TI=SMTPD_---0TyZxr82_1589511481; Received: from 127.0.0.1(mailfrom:teawaterz@linux.alibaba.com fp:SMTPD_---0TyZxr82_1589511481) by smtp.aliyun-inc.com(127.0.0.1); Fri, 15 May 2020 10:58:15 +0800 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: [virtio-dev] [PATCH v3 00/15] virtio-mem: paravirtualized memory From: teawater In-Reply-To: Date: Fri, 15 May 2020 10:58:00 +0800 Cc: LKML , Linux MM , virtio-dev@lists.oasis-open.org, Linux Virtualization , kvm@vger.kernel.org, Michal Hocko , Andrew Morton , "Michael S . Tsirkin" , Sebastien Boeuf , Samuel Ortiz , Robert Bradford , Luiz Capitulino , Pankaj Gupta , Igor Mammedov , "Dr . David Alan Gilbert" , Alexander Duyck , Alexander Potapenko , Anshuman Khandual , Anthony Yznaga , Dan Williams , Dave Young , Jason Wang , Johannes Weiner , Juergen Gross , Konrad Rzeszutek Wilk , Len Brown , Mel Gorman , Michal Hocko , Mike Rapoport , Oscar Salvador , Oscar Salvador , Pavel Tatashin , Pavel Tatashin , Pingfan Liu , Qian Cai , "Rafael J. Wysocki" , "Rafael J. Wysocki" , Stefan Hajnoczi , Vlastimil Babka , Wei Yang Content-Transfer-Encoding: 8BIT Message-Id: <9708F43A-9BD2-4377-8EE8-7FB1D95C6F69@linux.alibaba.com> References: <20200507103119.11219-1-david@redhat.com> <7848642F-6AA7-4B5E-AE0E-DB0857C94A93@linux.alibaba.com> <31c5d2f9-c104-53e8-d9c8-cb45f7507c85@redhat.com> <389b6bdc-b196-e4b9-b6be-dcac57524fdf@redhat.com> <3c82e149-6c42-690e-9d58-bb8e69870fe0@redhat.com> To: David Hildenbrand X-Mailer: Apple Mail (2.3608.80.23.2.2) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org > 2020年5月14日 20:19，David Hildenbrand 写道： > > On 14.05.20 13:47, David Hildenbrand wrote: >> On 14.05.20 13:10, David Hildenbrand wrote: >>> On 14.05.20 12:12, David Hildenbrand wrote: >>>> On 14.05.20 12:02, teawater wrote: >>>>> >>>>> >>>>>> 2020年5月14日 16:48，David Hildenbrand 写道： >>>>>> >>>>>> On 14.05.20 08:44, teawater wrote: >>>>>>> Hi David, >>>>>>> >>>>>>> I got a kernel warning with v2 and v3. >>>>>> >>>>>> Hi Hui, >>>>>> >>>>>> thanks for playing with the latest versions. Surprisingly, I can >>>>>> reproduce even by hotplugging a DIMM instead as well - that's good, so >>>>>> it's not related to virtio-mem, lol. Seems to be some QEMU setup issue >>>>>> with older machine types. >>>>>> >>>>>> Can you switch to a newer qemu machine version, especially >>>>>> pc-i440fx-5.0? Both, hotplugging DIMMs and virtio-mem works for me with >>>>>> that QEMU machine just fine. >>>>> >>>>> I still could reproduce this issue with pc-i440fx-5.0 or pc. Did I miss anything? >>>>> >>>> >>>> Below I don't even see virtio_mem. I had to repair the image (filesystem >>>> fsck) because it was broken, can you try that as well? >>>> >>>> Also, it would be great if you could test with v4. >>>> >>> >>> Correction, something seems to be broken either in QEMU or the kernel. Once I >>> define a DIMM so it's added and online during boot, I get these issues: >>> >>> (I have virtio-mem v4 installed in the guest) >>> >>> #! /bin/bash >>> sudo x86_64-softmmu/qemu-system-x86_64 \ >>> -machine pc-i440fx-5.0,accel=kvm,usb=off \ >>> -cpu host \ >>> -no-reboot \ >>> -nographic \ >>> -device ide-hd,drive=hd \ >>> -drive if=none,id=hd,file=/home/dhildenb/git/Fedora-Cloud-Base-31-1.9.x86_64.qcow2,format=qcow2 \ >>> -m 1g,slots=10,maxmem=2G \ >>> -smp 1 \ >>> -object memory-backend-ram,id=mem0,size=256m \ >>> -device pc-dimm,id=dimm0,memdev=mem0 \ >>> -s \ >>> -monitor unix:/var/tmp/monitor,server,nowait >>> >>> >>> Without the DIMM it seems to work just fine. >>> >> >> And another correction. >> >> Using QEMU v5.0.0, Linux 5.7-rc5, untouched >> Fedora-Cloud-Base-32-1.6.x86_64.qcow2, I get even without any memory hotplug: >> >> #! /bin/bash >> sudo x86_64-softmmu/qemu-system-x86_64 \ >> -machine pc-i440fx-5.0,accel=kvm,usb=off \ >> -cpu host \ >> -no-reboot \ >> -nographic \ >> -device ide-hd,drive=hd \ >> -drive if=none,id=hd,file=/home/dhildenb/git/Fedora-Cloud-Base-32-1.6.x86_64.qcow2,format=qcow2 \ >> -m 5g,slots=10,maxmem=6G \ >> -smp 1 \ >> -s \ >> -kernel /home/dhildenb/git/linux/arch/x86/boot/bzImage \ >> -append "console=ttyS0 rd.shell nokaslr swiotlb=noforce" \ >> -monitor unix:/var/tmp/monitor,server,nowait >> >> >> Observe how big the initial RAM even is! >> >> >> So this is no DIMM/hotplug/virtio_mem issue. With memory hotplug, it seems to get >> more likely to trigger if "swiotlb=noforce" is not specified. >> >> "swiotlb=noforce" seems to trigger some pre-existing issue here. Without >> "swiotlb=noforce", I was only able to observe this via pc-i440fx-2.1, >> > > (talking to myself :) ) > > I think I finally understood why using "swiotlb=noforce" with hotplugged > memory is wrong - or with memory > 3GB. Via "swiotlb=noforce" you tell > the system to "Never use bounce buffers (for debugging)". This works as > long as all memory is DMA memory (e.g., < 3GB) AFAIK. > > "If specified, trying to map memory that cannot be used with DMA will > fail, and a rate-limited warning will be printed." > > Hotplugged memory (under QEMU) is never added below 4GB, because of the > PCI hole. So both, memory from DIMMs and from virtio-mem will end up at > or above 4GB. To make a device use that memory, you need bounce buffers. > > Hotplugged memory is never DMA memory. > Hi David, It is fixed when I remove "swiotlb=noforce”. Thanks for your help. Best, Hui > -- > Thanks, > > David / dhildenb From mboxrd@z Thu Jan 1 00:00:00 1970 From: teawater Subject: Re: [PATCH v3 00/15] virtio-mem: paravirtualized memory Date: Fri, 15 May 2020 10:58:00 +0800 Message-ID: <9708F43A-9BD2-4377-8EE8-7FB1D95C6F69@linux.alibaba.com> References: <20200507103119.11219-1-david@redhat.com> <7848642F-6AA7-4B5E-AE0E-DB0857C94A93@linux.alibaba.com> <31c5d2f9-c104-53e8-d9c8-cb45f7507c85@redhat.com> <389b6bdc-b196-e4b9-b6be-dcac57524fdf@redhat.com> <3c82e149-6c42-690e-9d58-bb8e69870fe0@redhat.com> Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Sender: List-Post: List-Help: List-Unsubscribe: List-Subscribe: In-Reply-To: To: David Hildenbrand Cc: LKML , Linux MM , virtio-dev@lists.oasis-open.org, Linux Virtualization , kvm@vger.kernel.org, Michal Hocko , Andrew Morton , "Michael S . Tsirkin" , Sebastien Boeuf , Samuel Ortiz , Robert Bradford , Luiz Capitulino , Pankaj Gupta , Igor Mammedov , "Dr . David Alan Gilbert" , Alexander Duyck , Alexander Potapenko , Anshuman Khandual , Anthony Yznaga , Dan Williams , Dave Young , Jason List-Id: virtualization@lists.linuxfoundation.org > 2020=E5=B9=B45=E6=9C=8814=E6=97=A5 20:19=EF=BC=8CDavid Hildenbrand = =E5=86=99=E9=81=93=EF=BC=9A >=20 > On 14.05.20 13:47, David Hildenbrand wrote: >> On 14.05.20 13:10, David Hildenbrand wrote: >>> On 14.05.20 12:12, David Hildenbrand wrote: >>>> On 14.05.20 12:02, teawater wrote: >>>>>=20 >>>>>=20 >>>>>> 2020=E5=B9=B45=E6=9C=8814=E6=97=A5 16:48=EF=BC=8CDavid = Hildenbrand =E5=86=99=E9=81=93=EF=BC=9A >>>>>>=20 >>>>>> On 14.05.20 08:44, teawater wrote: >>>>>>> Hi David, >>>>>>>=20 >>>>>>> I got a kernel warning with v2 and v3. >>>>>>=20 >>>>>> Hi Hui, >>>>>>=20 >>>>>> thanks for playing with the latest versions. Surprisingly, I can >>>>>> reproduce even by hotplugging a DIMM instead as well - that's = good, so >>>>>> it's not related to virtio-mem, lol. Seems to be some QEMU setup = issue >>>>>> with older machine types. >>>>>>=20 >>>>>> Can you switch to a newer qemu machine version, especially >>>>>> pc-i440fx-5.0? Both, hotplugging DIMMs and virtio-mem works for = me with >>>>>> that QEMU machine just fine. >>>>>=20 >>>>> I still could reproduce this issue with pc-i440fx-5.0 or pc. Did = I miss anything? >>>>>=20 >>>>=20 >>>> Below I don't even see virtio_mem. I had to repair the image = (filesystem >>>> fsck) because it was broken, can you try that as well? >>>>=20 >>>> Also, it would be great if you could test with v4. >>>>=20 >>>=20 >>> Correction, something seems to be broken either in QEMU or the = kernel. Once I >>> define a DIMM so it's added and online during boot, I get these = issues: >>>=20 >>> (I have virtio-mem v4 installed in the guest) >>>=20 >>> #! /bin/bash >>> sudo x86_64-softmmu/qemu-system-x86_64 \ >>> -machine pc-i440fx-5.0,accel=3Dkvm,usb=3Doff \ >>> -cpu host \ >>> -no-reboot \ >>> -nographic \ >>> -device ide-hd,drive=3Dhd \ >>> -drive = if=3Dnone,id=3Dhd,file=3D/home/dhildenb/git/Fedora-Cloud-Base-31-1.9.x86_6= 4.qcow2,format=3Dqcow2 \ >>> -m 1g,slots=3D10,maxmem=3D2G \ >>> -smp 1 \ >>> -object memory-backend-ram,id=3Dmem0,size=3D256m \ >>> -device pc-dimm,id=3Ddimm0,memdev=3Dmem0 \ >>> -s \ >>> -monitor unix:/var/tmp/monitor,server,nowait >>>=20 >>>=20 >>> Without the DIMM it seems to work just fine. >>>=20 >>=20 >> And another correction.=20 >>=20 >> Using QEMU v5.0.0, Linux 5.7-rc5, untouched >> Fedora-Cloud-Base-32-1.6.x86_64.qcow2, I get even without any memory = hotplug: >>=20 >> #! /bin/bash >> sudo x86_64-softmmu/qemu-system-x86_64 \ >> -machine pc-i440fx-5.0,accel=3Dkvm,usb=3Doff \ >> -cpu host \ >> -no-reboot \ >> -nographic \ >> -device ide-hd,drive=3Dhd \ >> -drive = if=3Dnone,id=3Dhd,file=3D/home/dhildenb/git/Fedora-Cloud-Base-32-1.6.x86_6= 4.qcow2,format=3Dqcow2 \ >> -m 5g,slots=3D10,maxmem=3D6G \ >> -smp 1 \ >> -s \ >> -kernel /home/dhildenb/git/linux/arch/x86/boot/bzImage \ >> -append "console=3DttyS0 rd.shell nokaslr swiotlb=3Dnoforce" \ >> -monitor unix:/var/tmp/monitor,server,nowait >>=20 >>=20 >> Observe how big the initial RAM even is! >>=20 >>=20 >> So this is no DIMM/hotplug/virtio_mem issue. With memory hotplug, it = seems to get >> more likely to trigger if "swiotlb=3Dnoforce" is not specified. >>=20 >> "swiotlb=3Dnoforce" seems to trigger some pre-existing issue here. = Without >> "swiotlb=3Dnoforce", I was only able to observe this via = pc-i440fx-2.1, >>=20 >=20 > (talking to myself :) ) >=20 > I think I finally understood why using "swiotlb=3Dnoforce" with = hotplugged > memory is wrong - or with memory > 3GB. Via "swiotlb=3Dnoforce" you = tell > the system to "Never use bounce buffers (for debugging)". This works = as > long as all memory is DMA memory (e.g., < 3GB) AFAIK. >=20 > "If specified, trying to map memory that cannot be used with DMA will > fail, and a rate-limited warning will be printed." >=20 > Hotplugged memory (under QEMU) is never added below 4GB, because of = the > PCI hole. So both, memory from DIMMs and from virtio-mem will end up = at > or above 4GB. To make a device use that memory, you need bounce = buffers. >=20 > Hotplugged memory is never DMA memory. >=20 Hi David, It is fixed when I remove "swiotlb=3Dnoforce=E2=80=9D. Thanks for your help. Best, Hui > --=20 > Thanks, >=20 > David / dhildenb From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: virtio-dev-return-7343-cohuck=redhat.com@lists.oasis-open.org Sender: List-Post: List-Help: List-Unsubscribe: List-Subscribe: Received: from lists.oasis-open.org (oasis-open.org [10.110.1.242]) by lists.oasis-open.org (Postfix) with ESMTP id 15520986046 for ; Fri, 15 May 2020 02:58:26 +0000 (UTC) Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) From: teawater In-Reply-To: Date: Fri, 15 May 2020 10:58:00 +0800 Message-Id: <9708F43A-9BD2-4377-8EE8-7FB1D95C6F69@linux.alibaba.com> References: <20200507103119.11219-1-david@redhat.com> <7848642F-6AA7-4B5E-AE0E-DB0857C94A93@linux.alibaba.com> <31c5d2f9-c104-53e8-d9c8-cb45f7507c85@redhat.com> <389b6bdc-b196-e4b9-b6be-dcac57524fdf@redhat.com> <3c82e149-6c42-690e-9d58-bb8e69870fe0@redhat.com> Subject: Re: [virtio-dev] [PATCH v3 00/15] virtio-mem: paravirtualized memory Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable To: David Hildenbrand Cc: LKML , Linux MM , virtio-dev@lists.oasis-open.org, Linux Virtualization , kvm@vger.kernel.org, Michal Hocko , Andrew Morton , "Michael S . Tsirkin" , Sebastien Boeuf , Samuel Ortiz , Robert Bradford , Luiz Capitulino , Pankaj Gupta , Igor Mammedov , "Dr . David Alan Gilbert" , Alexander Duyck , Alexander Potapenko , Anshuman Khandual , Anthony Yznaga , Dan Williams , Dave Young , Jason Wang , Johannes Weiner , Juergen Gross , Konrad Rzeszutek Wilk , Len Brown , Mel Gorman , Michal Hocko , Mike Rapoport , Oscar Salvador , Oscar Salvador , Pavel Tatashin , Pavel Tatashin , Pingfan Liu , Qian Cai , "Rafael J. Wysocki" , "Rafael J. Wysocki" , Stefan Hajnoczi , Vlastimil Babka , Wei Yang List-ID: > 2020=E5=B9=B45=E6=9C=8814=E6=97=A5 20:19=EF=BC=8CDavid Hildenbrand =E5=86=99=E9=81=93=EF=BC=9A >=20 > On 14.05.20 13:47, David Hildenbrand wrote: >> On 14.05.20 13:10, David Hildenbrand wrote: >>> On 14.05.20 12:12, David Hildenbrand wrote: >>>> On 14.05.20 12:02, teawater wrote: >>>>>=20 >>>>>=20 >>>>>> 2020=E5=B9=B45=E6=9C=8814=E6=97=A5 16:48=EF=BC=8CDavid Hildenbrand <= david@redhat.com> =E5=86=99=E9=81=93=EF=BC=9A >>>>>>=20 >>>>>> On 14.05.20 08:44, teawater wrote: >>>>>>> Hi David, >>>>>>>=20 >>>>>>> I got a kernel warning with v2 and v3. >>>>>>=20 >>>>>> Hi Hui, >>>>>>=20 >>>>>> thanks for playing with the latest versions. Surprisingly, I can >>>>>> reproduce even by hotplugging a DIMM instead as well - that's good, = so >>>>>> it's not related to virtio-mem, lol. Seems to be some QEMU setup iss= ue >>>>>> with older machine types. >>>>>>=20 >>>>>> Can you switch to a newer qemu machine version, especially >>>>>> pc-i440fx-5.0? Both, hotplugging DIMMs and virtio-mem works for me w= ith >>>>>> that QEMU machine just fine. >>>>>=20 >>>>> I still could reproduce this issue with pc-i440fx-5.0 or pc. Did I m= iss anything? >>>>>=20 >>>>=20 >>>> Below I don't even see virtio_mem. I had to repair the image (filesyst= em >>>> fsck) because it was broken, can you try that as well? >>>>=20 >>>> Also, it would be great if you could test with v4. >>>>=20 >>>=20 >>> Correction, something seems to be broken either in QEMU or the kernel. = Once I >>> define a DIMM so it's added and online during boot, I get these issues: >>>=20 >>> (I have virtio-mem v4 installed in the guest) >>>=20 >>> #! /bin/bash >>> sudo x86_64-softmmu/qemu-system-x86_64 \ >>> -machine pc-i440fx-5.0,accel=3Dkvm,usb=3Doff \ >>> -cpu host \ >>> -no-reboot \ >>> -nographic \ >>> -device ide-hd,drive=3Dhd \ >>> -drive if=3Dnone,id=3Dhd,file=3D/home/dhildenb/git/Fedora-Cloud-Base= -31-1.9.x86_64.qcow2,format=3Dqcow2 \ >>> -m 1g,slots=3D10,maxmem=3D2G \ >>> -smp 1 \ >>> -object memory-backend-ram,id=3Dmem0,size=3D256m \ >>> -device pc-dimm,id=3Ddimm0,memdev=3Dmem0 \ >>> -s \ >>> -monitor unix:/var/tmp/monitor,server,nowait >>>=20 >>>=20 >>> Without the DIMM it seems to work just fine. >>>=20 >>=20 >> And another correction.=20 >>=20 >> Using QEMU v5.0.0, Linux 5.7-rc5, untouched >> Fedora-Cloud-Base-32-1.6.x86_64.qcow2, I get even without any memory hot= plug: >>=20 >> #! /bin/bash >> sudo x86_64-softmmu/qemu-system-x86_64 \ >> -machine pc-i440fx-5.0,accel=3Dkvm,usb=3Doff \ >> -cpu host \ >> -no-reboot \ >> -nographic \ >> -device ide-hd,drive=3Dhd \ >> -drive if=3Dnone,id=3Dhd,file=3D/home/dhildenb/git/Fedora-Cloud-Base-= 32-1.6.x86_64.qcow2,format=3Dqcow2 \ >> -m 5g,slots=3D10,maxmem=3D6G \ >> -smp 1 \ >> -s \ >> -kernel /home/dhildenb/git/linux/arch/x86/boot/bzImage \ >> -append "console=3DttyS0 rd.shell nokaslr swiotlb=3Dnoforce" \ >> -monitor unix:/var/tmp/monitor,server,nowait >>=20 >>=20 >> Observe how big the initial RAM even is! >>=20 >>=20 >> So this is no DIMM/hotplug/virtio_mem issue. With memory hotplug, it see= ms to get >> more likely to trigger if "swiotlb=3Dnoforce" is not specified. >>=20 >> "swiotlb=3Dnoforce" seems to trigger some pre-existing issue here. Witho= ut >> "swiotlb=3Dnoforce", I was only able to observe this via pc-i440fx-2.1, >>=20 >=20 > (talking to myself :) ) >=20 > I think I finally understood why using "swiotlb=3Dnoforce" with hotplugge= d > memory is wrong - or with memory > 3GB. Via "swiotlb=3Dnoforce" you tell > the system to "Never use bounce buffers (for debugging)". This works as > long as all memory is DMA memory (e.g., < 3GB) AFAIK. >=20 > "If specified, trying to map memory that cannot be used with DMA will > fail, and a rate-limited warning will be printed." >=20 > Hotplugged memory (under QEMU) is never added below 4GB, because of the > PCI hole. So both, memory from DIMMs and from virtio-mem will end up at > or above 4GB. To make a device use that memory, you need bounce buffers. >=20 > Hotplugged memory is never DMA memory. >=20 Hi David, It is fixed when I remove "swiotlb=3Dnoforce=E2=80=9D. Thanks for your help. Best, Hui > --=20 > Thanks, >=20 > David / dhildenb --------------------------------------------------------------------- To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org