Re: 9 TiB vm memory creation

From: Igor Mammedov <imammedo@redhat.com>
To: David Hildenbrand <david@redhat.com>
Cc: Ani Sinha <ani@anisinha.ca>,
	QEMU Developers <qemu-devel@nongnu.org>,
	Paolo Bonzini <pbonzini@redhat.com>,
	kvm@vger.kernel.org
Subject: Re: 9 TiB vm memory creation
Date: Mon, 14 Feb 2022 16:55:15 +0100	[thread overview]
Message-ID: <20220214165515.226a1955@redhat.com> (raw)
In-Reply-To: <b9771171-8d28-b46b-4474-687a8fed0abd@redhat.com>

On Mon, 14 Feb 2022 15:37:53 +0100
David Hildenbrand <david@redhat.com> wrote:

> On 14.02.22 13:36, Igor Mammedov wrote:
> > On Mon, 14 Feb 2022 10:54:22 +0530 (IST)
> > Ani Sinha <ani@anisinha.ca> wrote:
> >   
> >> Hi Igor:
> >>
> >> I failed to spawn a 9 Tib VM. The max I could do was a 2 TiB vm on my
> >> system with the following commandline before either the system
> >> destabilized or the OOM killed killed qemu
> >>
> >> -m 2T,maxmem=9T,slots=1 \
> >> -object memory-backend-file,id=mem0,size=2T,mem-path=/data/temp/memfile,prealloc=off \
> >> -machine memory-backend=mem0 \
> >> -chardev file,path=/tmp/debugcon2.txt,id=debugcon \
> >> -device isa-debugcon,iobase=0x402,chardev=debugcon \
> >>
> >> I have attached the debugcon output from 2 TiB vm.
> >> Is there any other commandline parameters or options I should try?
> >>
> >> thanks
> >> ani  
> > 
> > $ truncate -s 9T 9tb_sparse_disk.img
> > $ qemu-system-x86_64 -m 9T \
> >   -object memory-backend-file,id=mem0,size=9T,mem-path=9tb_sparse_disk.img,prealloc=off,share=on \
> >   -machine memory-backend=mem0
> > 
> > works for me till GRUB menu, with sufficient guest kernel
> > persuasion (i.e. CLI limit ram size to something reasonable) you can boot linux
> > guest on it and inspect SMBIOS tables comfortably.
> > 
> > 
> > With KVM enabled it bails out with:
> >    qemu-system-x86_64: kvm_set_user_memory_region: KVM_SET_USER_MEMORY_REGION failed, slot=1, start=0x100000000, size=0x8ff40000000: Invalid argument
> > 
> > all of that on a host with 32G of RAM/no swap.
> >
> >   
> 
> #define KVM_MEM_MAX_NR_PAGES ((1UL << 31) - 1)
> 
> ~8 TiB (7,999999)

so essentially that's the our max for initial RAM
(ignoring initial RAM slots before 4Gb)

Are you aware of any attempts to make it larger?

But can we use extra pc-dimm devices for additional memory (with 8TiB limit)
as that will use another memslot?

> 
> In QEMU, we have
> 
> static hwaddr kvm_max_slot_size = ~0;
> 
> And only s390x sets
> 
> kvm_set_max_memslot_size(KVM_SLOT_MAX_BYTES);
> 
> with
> 
> #define KVM_SLOT_MAX_BYTES (4UL * TiB)
in QEMU default value is:
  static hwaddr kvm_max_slot_size = ~0
it is kernel side that's failing