* [Qemu-devel] Partial NUMA config
@ 2018-03-13 2:26 Alexey Kardashevskiy
2018-03-19 4:28 ` Alexey Kardashevskiy
0 siblings, 1 reply; 4+ messages in thread
From: Alexey Kardashevskiy @ 2018-03-13 2:26 UTC (permalink / raw)
To: Igor Mammedov; +Cc: qemu-devel, David Gibson, Paolo Bonzini
Hi Igor,
ec78f8114bc4c1 "numa: use possible_cpus for not mapped CPUs check" added a
warning about "All CPU(s) up to maxcpus should be described in NUMA config,
ability to start up with partial NUMA mappings is obsoleted and will be
removed in future" and this is printed when I add a numa node without
attached CPU like this:
-numa node,nodeid=0,cpus=0,mem=4G
-numa node,nodeid=1,mem=131072M
And the reason for this command line is that I am trying to pass some dodgy
host RAM (actually belongs to a GPU but directly accessible via a fast
NVLink, not PCI fabric) which let's say is equally far from all CPUs, at
least in the host's NUMA config this memory is also not bound to any CPU:
[aik@aik ~]$ ssh yc02goos numactl -H
available: 8 nodes (0,8,250-255)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
node 0 size: 261735 MB
node 0 free: 258932 MB
node 8 cpus: 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84
85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106
107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125
126 127
node 8 size: 261739 MB
node 8 free: 261414 MB
node 250 cpus:
node 250 size: 15360 MB
node 250 free: 15359 MB
node 251 cpus:
node 251 size: 0 MB
node 251 free: 0 MB
node 252 cpus:
node 252 size: 15360 MB
node 252 free: 15359 MB
node 253 cpus:
node 253 size: 15360 MB
node 253 free: 15359 MB
node 254 cpus:
node 254 size: 15360 MB
node 254 free: 15359 MB
node 255 cpus:
node 255 size: 15360 MB
node 255 free: 15359 MB
node distances:
node 0 8 250 251 252 253 254 255
0: 10 40 80 80 80 80 80 80
8: 40 10 80 80 80 80 80 80
250: 80 80 10 80 80 80 80 80
251: 80 80 80 10 80 80 80 80
252: 80 80 80 80 10 80 80 80
253: 80 80 80 80 80 10 80 80
254: 80 80 80 80 80 80 10 80
255: 80 80 80 80 80 80 80 10
I am not sure I'll progress far enough to get this working with VFIO but if
I do, I'd like to keep an ability to have such a partial config in the
future. What was the reason for this warning in the first place?
--
Alexey
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Qemu-devel] Partial NUMA config
2018-03-13 2:26 [Qemu-devel] Partial NUMA config Alexey Kardashevskiy
@ 2018-03-19 4:28 ` Alexey Kardashevskiy
2018-03-20 6:46 ` Igor Mammedov
0 siblings, 1 reply; 4+ messages in thread
From: Alexey Kardashevskiy @ 2018-03-19 4:28 UTC (permalink / raw)
To: Igor Mammedov; +Cc: qemu-devel, David Gibson, Paolo Bonzini
On 13/3/18 1:26 pm, Alexey Kardashevskiy wrote:
> Hi Igor,
>
> ec78f8114bc4c1 "numa: use possible_cpus for not mapped CPUs check" added a
> warning about "All CPU(s) up to maxcpus should be described in NUMA config,
> ability to start up with partial NUMA mappings is obsoleted and will be
> removed in future" and this is printed when I add a numa node without
> attached CPU like this:
>
> -numa node,nodeid=0,cpus=0,mem=4G
> -numa node,nodeid=1,mem=131072M
>
> And the reason for this command line is that I am trying to pass some dodgy
> host RAM (actually belongs to a GPU but directly accessible via a fast
> NVLink, not PCI fabric) which let's say is equally far from all CPUs, at
> least in the host's NUMA config this memory is also not bound to any CPU:
>
> [aik@aik ~]$ ssh yc02goos numactl -H
> available: 8 nodes (0,8,250-255)
> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
> 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
> 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
> node 0 size: 261735 MB
> node 0 free: 258932 MB
> node 8 cpus: 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84
> 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106
> 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125
> 126 127
> node 8 size: 261739 MB
> node 8 free: 261414 MB
> node 250 cpus:
> node 250 size: 15360 MB
> node 250 free: 15359 MB
> node 251 cpus:
> node 251 size: 0 MB
> node 251 free: 0 MB
> node 252 cpus:
> node 252 size: 15360 MB
> node 252 free: 15359 MB
> node 253 cpus:
> node 253 size: 15360 MB
> node 253 free: 15359 MB
> node 254 cpus:
> node 254 size: 15360 MB
> node 254 free: 15359 MB
> node 255 cpus:
> node 255 size: 15360 MB
> node 255 free: 15359 MB
> node distances:
> node 0 8 250 251 252 253 254 255
> 0: 10 40 80 80 80 80 80 80
> 8: 40 10 80 80 80 80 80 80
> 250: 80 80 10 80 80 80 80 80
> 251: 80 80 80 10 80 80 80 80
> 252: 80 80 80 80 10 80 80 80
> 253: 80 80 80 80 80 10 80 80
> 254: 80 80 80 80 80 80 10 80
> 255: 80 80 80 80 80 80 80 10
>
>
> I am not sure I'll progress far enough to get this working with VFIO but if
> I do, I'd like to keep an ability to have such a partial config in the
> future. What was the reason for this warning in the first place?
Ping?
--
Alexey
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Qemu-devel] Partial NUMA config
2018-03-19 4:28 ` Alexey Kardashevskiy
@ 2018-03-20 6:46 ` Igor Mammedov
2018-03-21 4:04 ` Alexey Kardashevskiy
0 siblings, 1 reply; 4+ messages in thread
From: Igor Mammedov @ 2018-03-20 6:46 UTC (permalink / raw)
To: Alexey Kardashevskiy; +Cc: qemu-devel, David Gibson, Paolo Bonzini
On Mon, 19 Mar 2018 15:28:50 +1100
Alexey Kardashevskiy <aik@ozlabs.ru> wrote:
> On 13/3/18 1:26 pm, Alexey Kardashevskiy wrote:
> > Hi Igor,
> >
> > ec78f8114bc4c1 "numa: use possible_cpus for not mapped CPUs check" added a
> > warning about "All CPU(s) up to maxcpus should be described in NUMA config,
> > ability to start up with partial NUMA mappings is obsoleted and will be
> > removed in future" and this is printed when I add a numa node without
> > attached CPU like this:
> >
> > -numa node,nodeid=0,cpus=0,mem=4G
> > -numa node,nodeid=1,mem=131072M
Warning you see is about CPU(s) not assigned to any node
and it has noting to do with cpu-less node.
What's the full CLI you are using?
> > And the reason for this command line is that I am trying to pass some dodgy
> > host RAM (actually belongs to a GPU but directly accessible via a fast
> > NVLink, not PCI fabric) which let's say is equally far from all CPUs, at
> > least in the host's NUMA config this memory is also not bound to any CPU:
-numa node,nodeid=1,mem=131072M - will assign some chunk of initial RAM to
cpu-less node. It might serve purpose of cpu-less node simulation and nothing else.
You won't be able to pass it GPU memory this way though.
It might be possible using -numa node,nodeid=1,memdev=foo
but knowing nothing about NVLink, I'm not sure how it's shared with
GPU (PCI) and per node memory controllers.
> > [aik@aik ~]$ ssh yc02goos numactl -H
> > available: 8 nodes (0,8,250-255)
> > node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
> > 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
> > 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
> > node 0 size: 261735 MB
> > node 0 free: 258932 MB
> > node 8 cpus: 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84
> > 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106
> > 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125
> > 126 127
> > node 8 size: 261739 MB
> > node 8 free: 261414 MB
> > node 250 cpus:
> > node 250 size: 15360 MB
> > node 250 free: 15359 MB
> > node 251 cpus:
> > node 251 size: 0 MB
> > node 251 free: 0 MB
> > node 252 cpus:
> > node 252 size: 15360 MB
> > node 252 free: 15359 MB
> > node 253 cpus:
> > node 253 size: 15360 MB
> > node 253 free: 15359 MB
> > node 254 cpus:
> > node 254 size: 15360 MB
> > node 254 free: 15359 MB
> > node 255 cpus:
> > node 255 size: 15360 MB
> > node 255 free: 15359 MB
> > node distances:
> > node 0 8 250 251 252 253 254 255
> > 0: 10 40 80 80 80 80 80 80
> > 8: 40 10 80 80 80 80 80 80
> > 250: 80 80 10 80 80 80 80 80
> > 251: 80 80 80 10 80 80 80 80
> > 252: 80 80 80 80 10 80 80 80
> > 253: 80 80 80 80 80 10 80 80
> > 254: 80 80 80 80 80 80 10 80
> > 255: 80 80 80 80 80 80 80 10
> >
> >
> > I am not sure I'll progress far enough to get this working with VFIO but if
> > I do, I'd like to keep an ability to have such a partial config in the
> > future. What was the reason for this warning in the first place?
a CPU(including possible ones) shouldn't hung in nowhere,
it should be assigned to some node to describe its relation to memory on nodes.
>
> Ping?
>
>
>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Qemu-devel] Partial NUMA config
2018-03-20 6:46 ` Igor Mammedov
@ 2018-03-21 4:04 ` Alexey Kardashevskiy
0 siblings, 0 replies; 4+ messages in thread
From: Alexey Kardashevskiy @ 2018-03-21 4:04 UTC (permalink / raw)
To: Igor Mammedov; +Cc: qemu-devel, David Gibson, Paolo Bonzini
On 20/3/18 5:46 pm, Igor Mammedov wrote:
> On Mon, 19 Mar 2018 15:28:50 +1100
> Alexey Kardashevskiy <aik@ozlabs.ru> wrote:
>
>> On 13/3/18 1:26 pm, Alexey Kardashevskiy wrote:
>>> Hi Igor,
>>>
>>> ec78f8114bc4c1 "numa: use possible_cpus for not mapped CPUs check" added a
>>> warning about "All CPU(s) up to maxcpus should be described in NUMA config,
>>> ability to start up with partial NUMA mappings is obsoleted and will be
>>> removed in future" and this is printed when I add a numa node without
>>> attached CPU like this:
>>>
>>> -numa node,nodeid=0,cpus=0,mem=4G
>>> -numa node,nodeid=1,mem=131072M
> Warning you see is about CPU(s) not assigned to any node
> and it has noting to do with cpu-less node.
>
> What's the full CLI you are using?
I do not have it handy but it included:
-smp 2 -numa node,nodeid=0,cpus=0,mem=4G -numa node,nodeid=1,mem=131072M
And your explanation about nonassigned CPUs makes sense, thanks!
--
Alexey
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2018-03-21 4:04 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-03-13 2:26 [Qemu-devel] Partial NUMA config Alexey Kardashevskiy
2018-03-19 4:28 ` Alexey Kardashevskiy
2018-03-20 6:46 ` Igor Mammedov
2018-03-21 4:04 ` Alexey Kardashevskiy
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.