* [Qemu-devel] How do we do user input bitmap properties? @ 2019-04-18 9:28 ` Andrew Jones 0 siblings, 0 replies; 45+ messages in thread From: Andrew Jones @ 2019-04-18 9:28 UTC (permalink / raw) To: qemu-devel; +Cc: armbru, dgilbert, peter.maydell, Dave.Martin, imammedo Hi all, First some background: For the userspace side of AArch64 guest SVE support we need to expose KVM's allowed vector lengths bitmap to the user and allow the user to choose a subset of that bitmap. Since bitmaps are a bit awkward to work with then we'll likely want to expose it as an array of vector lengths instead. Also, assuming we want to expose the lengths as number-of-quadwords (quadword == 128 bits for AArch64 and vector lengths must be multiples of quadwords) rather than number-of-bits, then an example array (which will always be a sequence) might be [ 8, 16, 32 ] The user may choose a subsequence, but only through truncation, i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. Furthermore, different hosts may support different sequences which have the same maximum. For example, if the above sequence is for Host_A, then Host_B could be [ 8, 16, 24, 32 ] The host must support all lengths in the sequence, which means that while Host_A supports 32, since it doesn't support 24 and we can only truncate sequences, we must use either [ 8 ] or [ 8, 16 ] for a compatible sequence if we intend to migrate between the hosts. Now to the $SUBJECT question: My feeling is that we should require the sequence to be provided on the command line as a cpu property. Something like -cpu host,sve-vl-list=8:16 (I chose ':' for the delimiter because ',' can't work, but if there's a better choice, then that's fine by me.) Afaict a property list like this will require a new parser, which feels a bit funny since it seems we should already have support for this type of thing somewhere in QEMU. So, the question is: do we? I see we have array properties, but I don't believe that works with the command line. Should we only use QMP for this? We already want some QMP in order to query the supported vector lengths. Maybe we should use QMP to set the selection too? But then what about command line support for developers? And if the property is on the command line then we don't have to add it to the migration stream. Thanks, drew ^ permalink raw reply [flat|nested] 45+ messages in thread
* [Qemu-devel] How do we do user input bitmap properties? @ 2019-04-18 9:28 ` Andrew Jones 0 siblings, 0 replies; 45+ messages in thread From: Andrew Jones @ 2019-04-18 9:28 UTC (permalink / raw) To: qemu-devel; +Cc: peter.maydell, imammedo, Dave.Martin, armbru, dgilbert Hi all, First some background: For the userspace side of AArch64 guest SVE support we need to expose KVM's allowed vector lengths bitmap to the user and allow the user to choose a subset of that bitmap. Since bitmaps are a bit awkward to work with then we'll likely want to expose it as an array of vector lengths instead. Also, assuming we want to expose the lengths as number-of-quadwords (quadword == 128 bits for AArch64 and vector lengths must be multiples of quadwords) rather than number-of-bits, then an example array (which will always be a sequence) might be [ 8, 16, 32 ] The user may choose a subsequence, but only through truncation, i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. Furthermore, different hosts may support different sequences which have the same maximum. For example, if the above sequence is for Host_A, then Host_B could be [ 8, 16, 24, 32 ] The host must support all lengths in the sequence, which means that while Host_A supports 32, since it doesn't support 24 and we can only truncate sequences, we must use either [ 8 ] or [ 8, 16 ] for a compatible sequence if we intend to migrate between the hosts. Now to the $SUBJECT question: My feeling is that we should require the sequence to be provided on the command line as a cpu property. Something like -cpu host,sve-vl-list=8:16 (I chose ':' for the delimiter because ',' can't work, but if there's a better choice, then that's fine by me.) Afaict a property list like this will require a new parser, which feels a bit funny since it seems we should already have support for this type of thing somewhere in QEMU. So, the question is: do we? I see we have array properties, but I don't believe that works with the command line. Should we only use QMP for this? We already want some QMP in order to query the supported vector lengths. Maybe we should use QMP to set the selection too? But then what about command line support for developers? And if the property is on the command line then we don't have to add it to the migration stream. Thanks, drew ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? @ 2019-04-18 9:46 ` Andrew Jones 0 siblings, 0 replies; 45+ messages in thread From: Andrew Jones @ 2019-04-18 9:46 UTC (permalink / raw) To: qemu-devel; +Cc: armbru, dgilbert, peter.maydell, Dave.Martin, imammedo On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote: > Hi all, > > First some background: > > For the userspace side of AArch64 guest SVE support we need to > expose KVM's allowed vector lengths bitmap to the user and allow > the user to choose a subset of that bitmap. Since bitmaps are a > bit awkward to work with then we'll likely want to expose it as > an array of vector lengths instead. Also, assuming we want to > expose the lengths as number-of-quadwords (quadword == 128 bits > for AArch64 and vector lengths must be multiples of quadwords) > rather than number-of-bits, then an example array (which will > always be a sequence) might be > > [ 8, 16, 32 ] > > The user may choose a subsequence, but only through truncation, > i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. > > Furthermore, different hosts may support different sequences > which have the same maximum. For example, if the above sequence > is for Host_A, then Host_B could be > > [ 8, 16, 24, 32 ] It doesn't really matter for this discussion, but I just realized that I picked bogus numbers for these examples. 32 would be too big. The largest supported is 16. I probably should have just used the simple [ 1, 2, 4 ] and [ 1, 2, 3, 4 ] arrays, corresponding to vector lengths (in bits) 128, 256, 512 and 128, 256, 384, 512. > > The host must support all lengths in the sequence, which means > that while Host_A supports 32, since it doesn't support 24 and > we can only truncate sequences, we must use either [ 8 ] or > [ 8, 16 ] for a compatible sequence if we intend to migrate > between the hosts. > > Now to the $SUBJECT question: > > My feeling is that we should require the sequence to be > provided on the command line as a cpu property. Something > like > > -cpu host,sve-vl-list=8:16 > > (I chose ':' for the delimiter because ',' can't work, but > if there's a better choice, then that's fine by me.) > > Afaict a property list like this will require a new parser, > which feels a bit funny since it seems we should already > have support for this type of thing somewhere in QEMU. So, > the question is: do we? I see we have array properties, but > I don't believe that works with the command line. Should we > only use QMP for this? We already want some QMP in order to > query the supported vector lengths. Maybe we should use QMP > to set the selection too? But then what about command line > support for developers? And if the property is on the command > line then we don't have to add it to the migration stream. > > Thanks, > drew ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? @ 2019-04-18 9:46 ` Andrew Jones 0 siblings, 0 replies; 45+ messages in thread From: Andrew Jones @ 2019-04-18 9:46 UTC (permalink / raw) To: qemu-devel; +Cc: peter.maydell, imammedo, Dave.Martin, armbru, dgilbert On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote: > Hi all, > > First some background: > > For the userspace side of AArch64 guest SVE support we need to > expose KVM's allowed vector lengths bitmap to the user and allow > the user to choose a subset of that bitmap. Since bitmaps are a > bit awkward to work with then we'll likely want to expose it as > an array of vector lengths instead. Also, assuming we want to > expose the lengths as number-of-quadwords (quadword == 128 bits > for AArch64 and vector lengths must be multiples of quadwords) > rather than number-of-bits, then an example array (which will > always be a sequence) might be > > [ 8, 16, 32 ] > > The user may choose a subsequence, but only through truncation, > i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. > > Furthermore, different hosts may support different sequences > which have the same maximum. For example, if the above sequence > is for Host_A, then Host_B could be > > [ 8, 16, 24, 32 ] It doesn't really matter for this discussion, but I just realized that I picked bogus numbers for these examples. 32 would be too big. The largest supported is 16. I probably should have just used the simple [ 1, 2, 4 ] and [ 1, 2, 3, 4 ] arrays, corresponding to vector lengths (in bits) 128, 256, 512 and 128, 256, 384, 512. > > The host must support all lengths in the sequence, which means > that while Host_A supports 32, since it doesn't support 24 and > we can only truncate sequences, we must use either [ 8 ] or > [ 8, 16 ] for a compatible sequence if we intend to migrate > between the hosts. > > Now to the $SUBJECT question: > > My feeling is that we should require the sequence to be > provided on the command line as a cpu property. Something > like > > -cpu host,sve-vl-list=8:16 > > (I chose ':' for the delimiter because ',' can't work, but > if there's a better choice, then that's fine by me.) > > Afaict a property list like this will require a new parser, > which feels a bit funny since it seems we should already > have support for this type of thing somewhere in QEMU. So, > the question is: do we? I see we have array properties, but > I don't believe that works with the command line. Should we > only use QMP for this? We already want some QMP in order to > query the supported vector lengths. Maybe we should use QMP > to set the selection too? But then what about command line > support for developers? And if the property is on the command > line then we don't have to add it to the migration stream. > > Thanks, > drew ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? @ 2019-04-18 10:52 ` Dave Martin 0 siblings, 0 replies; 45+ messages in thread From: Dave Martin @ 2019-04-18 10:52 UTC (permalink / raw) To: Andrew Jones; +Cc: qemu-devel, armbru, dgilbert, peter.maydell, imammedo On Thu, Apr 18, 2019 at 10:46:34AM +0100, Andrew Jones wrote: > On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote: > > Hi all, > > > > First some background: > > > > For the userspace side of AArch64 guest SVE support we need to > > expose KVM's allowed vector lengths bitmap to the user and allow > > the user to choose a subset of that bitmap. Since bitmaps are a > > bit awkward to work with then we'll likely want to expose it as > > an array of vector lengths instead. Also, assuming we want to > > expose the lengths as number-of-quadwords (quadword == 128 bits > > for AArch64 and vector lengths must be multiples of quadwords) > > rather than number-of-bits, then an example array (which will > > always be a sequence) might be > > > > [ 8, 16, 32 ] > > > > The user may choose a subsequence, but only through truncation, > > i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. > > > > Furthermore, different hosts may support different sequences > > which have the same maximum. For example, if the above sequence > > is for Host_A, then Host_B could be > > > > [ 8, 16, 24, 32 ] > > It doesn't really matter for this discussion, but I just realized > that I picked bogus numbers for these examples. 32 would be too > big. The largest supported is 16. I probably should have just used > the simple [ 1, 2, 4 ] and [ 1, 2, 3, 4 ] arrays, corresponding to > vector lengths (in bits) 128, 256, 512 and 128, 256, 384, 512. I'd argue differently from the ABI perspective. If the host supports vq = [ 1,2,3,4 ], then it is entirely valid to ask for [ 1,2,4 ]. KVM will fail the KVM_REG_ARM64_SVE_VLS write with EINVAL, but I don't distinguish between a set that is not satisfiable on this hardware and a set that is not valid at all. The kernel basically doesn't check for the latter: an architecturally invalid set will always not be satisfiable on the hardware. Ideally, userspace should not preempt this decision, in case the architecture becomes more flexible someday in what it can virtualise. > > The host must support all lengths in the sequence, which means > > that while Host_A supports 32, since it doesn't support 24 and > > we can only truncate sequences, we must use either [ 8 ] or > > [ 8, 16 ] for a compatible sequence if we intend to migrate > > between the hosts. > > > > Now to the $SUBJECT question: > > > > My feeling is that we should require the sequence to be > > provided on the command line as a cpu property. Something > > like > > > > -cpu host,sve-vl-list=8:16 I can't decide whether it's more or less user-friendly to denote VL in terms of bytes in this kind of context. Usually when I say "VL" in the kernel ABI I try to mean this. (KVM_REG_ARM64_SVE_VLS is a special case: here the lengths are fancily encoded rather than being passed as integers, so the precise unit used is a more abstract concept here ... or anyway, that's my excuse. _SVE_VQS felt odd here.) For kvmtool I went with quadwords on the command line, but only as a quick hack to simplify the parser slightly. OTOH, specifying 1,2,4 on the command line is clearly less annoying and error-prone than 16,32,64. Naming the option with "vq" instead of "vl" is another option, though "vq" is Linux terminology not endorsed by the architecture. For kvmtool I've considered a range / filtering syntax, but I didn't implement it yet. [...] Cheers ---Dave ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? @ 2019-04-18 10:52 ` Dave Martin 0 siblings, 0 replies; 45+ messages in thread From: Dave Martin @ 2019-04-18 10:52 UTC (permalink / raw) To: Andrew Jones; +Cc: peter.maydell, imammedo, qemu-devel, dgilbert, armbru On Thu, Apr 18, 2019 at 10:46:34AM +0100, Andrew Jones wrote: > On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote: > > Hi all, > > > > First some background: > > > > For the userspace side of AArch64 guest SVE support we need to > > expose KVM's allowed vector lengths bitmap to the user and allow > > the user to choose a subset of that bitmap. Since bitmaps are a > > bit awkward to work with then we'll likely want to expose it as > > an array of vector lengths instead. Also, assuming we want to > > expose the lengths as number-of-quadwords (quadword == 128 bits > > for AArch64 and vector lengths must be multiples of quadwords) > > rather than number-of-bits, then an example array (which will > > always be a sequence) might be > > > > [ 8, 16, 32 ] > > > > The user may choose a subsequence, but only through truncation, > > i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. > > > > Furthermore, different hosts may support different sequences > > which have the same maximum. For example, if the above sequence > > is for Host_A, then Host_B could be > > > > [ 8, 16, 24, 32 ] > > It doesn't really matter for this discussion, but I just realized > that I picked bogus numbers for these examples. 32 would be too > big. The largest supported is 16. I probably should have just used > the simple [ 1, 2, 4 ] and [ 1, 2, 3, 4 ] arrays, corresponding to > vector lengths (in bits) 128, 256, 512 and 128, 256, 384, 512. I'd argue differently from the ABI perspective. If the host supports vq = [ 1,2,3,4 ], then it is entirely valid to ask for [ 1,2,4 ]. KVM will fail the KVM_REG_ARM64_SVE_VLS write with EINVAL, but I don't distinguish between a set that is not satisfiable on this hardware and a set that is not valid at all. The kernel basically doesn't check for the latter: an architecturally invalid set will always not be satisfiable on the hardware. Ideally, userspace should not preempt this decision, in case the architecture becomes more flexible someday in what it can virtualise. > > The host must support all lengths in the sequence, which means > > that while Host_A supports 32, since it doesn't support 24 and > > we can only truncate sequences, we must use either [ 8 ] or > > [ 8, 16 ] for a compatible sequence if we intend to migrate > > between the hosts. > > > > Now to the $SUBJECT question: > > > > My feeling is that we should require the sequence to be > > provided on the command line as a cpu property. Something > > like > > > > -cpu host,sve-vl-list=8:16 I can't decide whether it's more or less user-friendly to denote VL in terms of bytes in this kind of context. Usually when I say "VL" in the kernel ABI I try to mean this. (KVM_REG_ARM64_SVE_VLS is a special case: here the lengths are fancily encoded rather than being passed as integers, so the precise unit used is a more abstract concept here ... or anyway, that's my excuse. _SVE_VQS felt odd here.) For kvmtool I went with quadwords on the command line, but only as a quick hack to simplify the parser slightly. OTOH, specifying 1,2,4 on the command line is clearly less annoying and error-prone than 16,32,64. Naming the option with "vq" instead of "vl" is another option, though "vq" is Linux terminology not endorsed by the architecture. For kvmtool I've considered a range / filtering syntax, but I didn't implement it yet. [...] Cheers ---Dave ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? @ 2019-04-18 11:28 ` Andrew Jones 0 siblings, 0 replies; 45+ messages in thread From: Andrew Jones @ 2019-04-18 11:28 UTC (permalink / raw) To: Dave Martin; +Cc: peter.maydell, imammedo, qemu-devel, dgilbert, armbru On Thu, Apr 18, 2019 at 11:52:04AM +0100, Dave Martin wrote: > On Thu, Apr 18, 2019 at 10:46:34AM +0100, Andrew Jones wrote: > > On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote: > > > Hi all, > > > > > > First some background: > > > > > > For the userspace side of AArch64 guest SVE support we need to > > > expose KVM's allowed vector lengths bitmap to the user and allow > > > the user to choose a subset of that bitmap. Since bitmaps are a > > > bit awkward to work with then we'll likely want to expose it as > > > an array of vector lengths instead. Also, assuming we want to > > > expose the lengths as number-of-quadwords (quadword == 128 bits > > > for AArch64 and vector lengths must be multiples of quadwords) > > > rather than number-of-bits, then an example array (which will > > > always be a sequence) might be > > > > > > [ 8, 16, 32 ] > > > > > > The user may choose a subsequence, but only through truncation, > > > i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. > > > > > > Furthermore, different hosts may support different sequences > > > which have the same maximum. For example, if the above sequence > > > is for Host_A, then Host_B could be > > > > > > [ 8, 16, 24, 32 ] > > > > It doesn't really matter for this discussion, but I just realized > > that I picked bogus numbers for these examples. 32 would be too > > big. The largest supported is 16. I probably should have just used > > the simple [ 1, 2, 4 ] and [ 1, 2, 3, 4 ] arrays, corresponding to > > vector lengths (in bits) 128, 256, 512 and 128, 256, 384, 512. > > I'd argue differently from the ABI perspective. > > If the host supports vq = [ 1,2,3,4 ], then it is entirely valid to ask > for [ 1,2,4 ]. KVM will fail the KVM_REG_ARM64_SVE_VLS write with > EINVAL, but I don't distinguish between a set that is not satisfiable on > this hardware and a set that is not valid at all. The kernel basically > doesn't check for the latter: an architecturally invalid set will always > not be satisfiable on the hardware. I agree that 1,2,4 should be a valid set, but how do we avoid the user attempting to select it on a platform that supports 1,2,3,4 now? Until the hardware and KVM will allow it, then I don't think we should try to "support" it in any way. Maybe we should name the property such that it's clear we need to use truncation when constructing a subset now. This would allow us to add another, more generally named, property later when we can select nearly arbitrary subsets. > > Ideally, userspace should not preempt this decision, in case the > architecture becomes more flexible someday in what it can virtualise. > > > > The host must support all lengths in the sequence, which means > > > that while Host_A supports 32, since it doesn't support 24 and > > > we can only truncate sequences, we must use either [ 8 ] or > > > [ 8, 16 ] for a compatible sequence if we intend to migrate > > > between the hosts. > > > > > > Now to the $SUBJECT question: > > > > > > My feeling is that we should require the sequence to be > > > provided on the command line as a cpu property. Something > > > like > > > > > > -cpu host,sve-vl-list=8:16 > > I can't decide whether it's more or less user-friendly to denote VL in > terms of bytes in this kind of context. I'm not sure either. Switching to my second take at quadwords this should have been '1:2'. In bytes [ 1, 2 ] would be '16:32'. > > Usually when I say "VL" in the kernel ABI I try to mean this. > (KVM_REG_ARM64_SVE_VLS is a special case: here the lengths are fancily > encoded rather than being passed as integers, so the precise unit used > is a more abstract concept here ... or anyway, that's my excuse. > _SVE_VQS felt odd here.) My thinking is that using the number of quadwords keeps the numbers smaller. But, if people are mostly going to want to think about vector lengths in terms of bits, like the specification does, then maybe we should just let the numbers be bigger: [ 128, 256, 512 ] > > For kvmtool I went with quadwords on the command line, but only as a > quick hack to simplify the parser slightly. > > OTOH, specifying 1,2,4 on the command line is clearly less annoying and > error-prone than 16,32,64. > > Naming the option with "vq" instead of "vl" is another option, though > "vq" is Linux terminology not endorsed by the architecture. I think VL is OK as long we document the unit. > > For kvmtool I've considered a range / filtering syntax, but I didn't > implement it yet. I think we want each VL to be explicitly listed on the command line for the compatibility issue described above. On one host the range [1,4] can be different than the range [1,4] on anther host, depending on whether '3' is in there. And syntax like 1-4:-3 would be overly complicated, IMO. So I'm inclined to just require all of them to be listed, but of course order wouldn't matter. Thanks, drew ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? @ 2019-04-18 11:28 ` Andrew Jones 0 siblings, 0 replies; 45+ messages in thread From: Andrew Jones @ 2019-04-18 11:28 UTC (permalink / raw) To: Dave Martin; +Cc: peter.maydell, armbru, qemu-devel, dgilbert, imammedo On Thu, Apr 18, 2019 at 11:52:04AM +0100, Dave Martin wrote: > On Thu, Apr 18, 2019 at 10:46:34AM +0100, Andrew Jones wrote: > > On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote: > > > Hi all, > > > > > > First some background: > > > > > > For the userspace side of AArch64 guest SVE support we need to > > > expose KVM's allowed vector lengths bitmap to the user and allow > > > the user to choose a subset of that bitmap. Since bitmaps are a > > > bit awkward to work with then we'll likely want to expose it as > > > an array of vector lengths instead. Also, assuming we want to > > > expose the lengths as number-of-quadwords (quadword == 128 bits > > > for AArch64 and vector lengths must be multiples of quadwords) > > > rather than number-of-bits, then an example array (which will > > > always be a sequence) might be > > > > > > [ 8, 16, 32 ] > > > > > > The user may choose a subsequence, but only through truncation, > > > i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. > > > > > > Furthermore, different hosts may support different sequences > > > which have the same maximum. For example, if the above sequence > > > is for Host_A, then Host_B could be > > > > > > [ 8, 16, 24, 32 ] > > > > It doesn't really matter for this discussion, but I just realized > > that I picked bogus numbers for these examples. 32 would be too > > big. The largest supported is 16. I probably should have just used > > the simple [ 1, 2, 4 ] and [ 1, 2, 3, 4 ] arrays, corresponding to > > vector lengths (in bits) 128, 256, 512 and 128, 256, 384, 512. > > I'd argue differently from the ABI perspective. > > If the host supports vq = [ 1,2,3,4 ], then it is entirely valid to ask > for [ 1,2,4 ]. KVM will fail the KVM_REG_ARM64_SVE_VLS write with > EINVAL, but I don't distinguish between a set that is not satisfiable on > this hardware and a set that is not valid at all. The kernel basically > doesn't check for the latter: an architecturally invalid set will always > not be satisfiable on the hardware. I agree that 1,2,4 should be a valid set, but how do we avoid the user attempting to select it on a platform that supports 1,2,3,4 now? Until the hardware and KVM will allow it, then I don't think we should try to "support" it in any way. Maybe we should name the property such that it's clear we need to use truncation when constructing a subset now. This would allow us to add another, more generally named, property later when we can select nearly arbitrary subsets. > > Ideally, userspace should not preempt this decision, in case the > architecture becomes more flexible someday in what it can virtualise. > > > > The host must support all lengths in the sequence, which means > > > that while Host_A supports 32, since it doesn't support 24 and > > > we can only truncate sequences, we must use either [ 8 ] or > > > [ 8, 16 ] for a compatible sequence if we intend to migrate > > > between the hosts. > > > > > > Now to the $SUBJECT question: > > > > > > My feeling is that we should require the sequence to be > > > provided on the command line as a cpu property. Something > > > like > > > > > > -cpu host,sve-vl-list=8:16 > > I can't decide whether it's more or less user-friendly to denote VL in > terms of bytes in this kind of context. I'm not sure either. Switching to my second take at quadwords this should have been '1:2'. In bytes [ 1, 2 ] would be '16:32'. > > Usually when I say "VL" in the kernel ABI I try to mean this. > (KVM_REG_ARM64_SVE_VLS is a special case: here the lengths are fancily > encoded rather than being passed as integers, so the precise unit used > is a more abstract concept here ... or anyway, that's my excuse. > _SVE_VQS felt odd here.) My thinking is that using the number of quadwords keeps the numbers smaller. But, if people are mostly going to want to think about vector lengths in terms of bits, like the specification does, then maybe we should just let the numbers be bigger: [ 128, 256, 512 ] > > For kvmtool I went with quadwords on the command line, but only as a > quick hack to simplify the parser slightly. > > OTOH, specifying 1,2,4 on the command line is clearly less annoying and > error-prone than 16,32,64. > > Naming the option with "vq" instead of "vl" is another option, though > "vq" is Linux terminology not endorsed by the architecture. I think VL is OK as long we document the unit. > > For kvmtool I've considered a range / filtering syntax, but I didn't > implement it yet. I think we want each VL to be explicitly listed on the command line for the compatibility issue described above. On one host the range [1,4] can be different than the range [1,4] on anther host, depending on whether '3' is in there. And syntax like 1-4:-3 would be overly complicated, IMO. So I'm inclined to just require all of them to be listed, but of course order wouldn't matter. Thanks, drew ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? @ 2019-04-18 14:03 ` Dave Martin 0 siblings, 0 replies; 45+ messages in thread From: Dave Martin @ 2019-04-18 14:03 UTC (permalink / raw) To: Andrew Jones; +Cc: peter.maydell, imammedo, qemu-devel, dgilbert, armbru On Thu, Apr 18, 2019 at 12:28:47PM +0100, Andrew Jones wrote: > On Thu, Apr 18, 2019 at 11:52:04AM +0100, Dave Martin wrote: > > On Thu, Apr 18, 2019 at 10:46:34AM +0100, Andrew Jones wrote: > > > On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote: > > > > Hi all, > > > > > > > > First some background: > > > > > > > > For the userspace side of AArch64 guest SVE support we need to > > > > expose KVM's allowed vector lengths bitmap to the user and allow > > > > the user to choose a subset of that bitmap. Since bitmaps are a > > > > bit awkward to work with then we'll likely want to expose it as > > > > an array of vector lengths instead. Also, assuming we want to > > > > expose the lengths as number-of-quadwords (quadword == 128 bits > > > > for AArch64 and vector lengths must be multiples of quadwords) > > > > rather than number-of-bits, then an example array (which will > > > > always be a sequence) might be > > > > > > > > [ 8, 16, 32 ] > > > > > > > > The user may choose a subsequence, but only through truncation, > > > > i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. > > > > > > > > Furthermore, different hosts may support different sequences > > > > which have the same maximum. For example, if the above sequence > > > > is for Host_A, then Host_B could be > > > > > > > > [ 8, 16, 24, 32 ] > > > > > > It doesn't really matter for this discussion, but I just realized > > > that I picked bogus numbers for these examples. 32 would be too > > > big. The largest supported is 16. I probably should have just used > > > the simple [ 1, 2, 4 ] and [ 1, 2, 3, 4 ] arrays, corresponding to > > > vector lengths (in bits) 128, 256, 512 and 128, 256, 384, 512. > > > > I'd argue differently from the ABI perspective. > > > > If the host supports vq = [ 1,2,3,4 ], then it is entirely valid to ask > > for [ 1,2,4 ]. KVM will fail the KVM_REG_ARM64_SVE_VLS write with > > EINVAL, but I don't distinguish between a set that is not satisfiable on > > this hardware and a set that is not valid at all. The kernel basically > > doesn't check for the latter: an architecturally invalid set will always > > not be satisfiable on the hardware. > > I agree that 1,2,4 should be a valid set, but how do we avoid the user > attempting to select it on a platform that supports 1,2,3,4 now? Until > the hardware and KVM will allow it, then I don't think we should try > to "support" it in any way. Maybe we should name the property such that > it's clear we need to use truncation when constructing a subset now. > This would allow us to add another, more generally named, property later > when we can select nearly arbitrary subsets. > > > > > Ideally, userspace should not preempt this decision, in case the > > architecture becomes more flexible someday in what it can virtualise. > > > > > > The host must support all lengths in the sequence, which means > > > > that while Host_A supports 32, since it doesn't support 24 and > > > > we can only truncate sequences, we must use either [ 8 ] or > > > > [ 8, 16 ] for a compatible sequence if we intend to migrate > > > > between the hosts. > > > > > > > > Now to the $SUBJECT question: > > > > > > > > My feeling is that we should require the sequence to be > > > > provided on the command line as a cpu property. Something > > > > like > > > > > > > > -cpu host,sve-vl-list=8:16 > > > > I can't decide whether it's more or less user-friendly to denote VL in > > terms of bytes in this kind of context. > > I'm not sure either. Switching to my second take at quadwords this should > have been '1:2'. In bytes [ 1, 2 ] would be '16:32'. > > > > > Usually when I say "VL" in the kernel ABI I try to mean this. > > (KVM_REG_ARM64_SVE_VLS is a special case: here the lengths are fancily > > encoded rather than being passed as integers, so the precise unit used > > is a more abstract concept here ... or anyway, that's my excuse. > > _SVE_VQS felt odd here.) > > My thinking is that using the number of quadwords keeps the numbers > smaller. But, if people are mostly going to want to think about vector > lengths in terms of bits, like the specification does, then maybe we > should just let the numbers be bigger: [ 128, 256, 512 ] > > > > > For kvmtool I went with quadwords on the command line, but only as a > > quick hack to simplify the parser slightly. > > > > OTOH, specifying 1,2,4 on the command line is clearly less annoying and > > error-prone than 16,32,64. > > > > Naming the option with "vq" instead of "vl" is another option, though > > "vq" is Linux terminology not endorsed by the architecture. > > I think VL is OK as long we document the unit. As you wish (and that's basically where I ended up for now with kvmtool). > > > > For kvmtool I've considered a range / filtering syntax, but I didn't > > implement it yet. > > I think we want each VL to be explicitly listed on the command line > for the compatibility issue described above. On one host the range > [1,4] can be different than the range [1,4] on anther host, depending > on whether '3' is in there. And syntax like 1-4:-3 would be overly > complicated, IMO. So I'm inclined to just require all of them to be > listed, but of course order wouldn't matter. Probably reasonable. For now (and quite likely forever) the length of the list is at most 16 elements. To avoid overengineering I stuck with a flat comma-separated list for now. It's worth nothing that there are two problems to be solved here: one is to specify an exact set unambiguously, which is important for migration scenarios. The other is to be able to clamp the vector length for user convenience, but without particularly considering migration. There's no direct way for the user to know the set of vector lengths supported by KVM today, so cooking up the right magic to go on the command line is non-trivial: you have to create a dummy vcpu and read KVM_REG_ARM64_SVE_VLS to find the host's supported set. Or you just have to know. A separate max-vl option (or whatever) might be a useful alternative. Cheers ---Dave ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? @ 2019-04-18 14:03 ` Dave Martin 0 siblings, 0 replies; 45+ messages in thread From: Dave Martin @ 2019-04-18 14:03 UTC (permalink / raw) To: Andrew Jones; +Cc: peter.maydell, armbru, qemu-devel, dgilbert, imammedo On Thu, Apr 18, 2019 at 12:28:47PM +0100, Andrew Jones wrote: > On Thu, Apr 18, 2019 at 11:52:04AM +0100, Dave Martin wrote: > > On Thu, Apr 18, 2019 at 10:46:34AM +0100, Andrew Jones wrote: > > > On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote: > > > > Hi all, > > > > > > > > First some background: > > > > > > > > For the userspace side of AArch64 guest SVE support we need to > > > > expose KVM's allowed vector lengths bitmap to the user and allow > > > > the user to choose a subset of that bitmap. Since bitmaps are a > > > > bit awkward to work with then we'll likely want to expose it as > > > > an array of vector lengths instead. Also, assuming we want to > > > > expose the lengths as number-of-quadwords (quadword == 128 bits > > > > for AArch64 and vector lengths must be multiples of quadwords) > > > > rather than number-of-bits, then an example array (which will > > > > always be a sequence) might be > > > > > > > > [ 8, 16, 32 ] > > > > > > > > The user may choose a subsequence, but only through truncation, > > > > i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. > > > > > > > > Furthermore, different hosts may support different sequences > > > > which have the same maximum. For example, if the above sequence > > > > is for Host_A, then Host_B could be > > > > > > > > [ 8, 16, 24, 32 ] > > > > > > It doesn't really matter for this discussion, but I just realized > > > that I picked bogus numbers for these examples. 32 would be too > > > big. The largest supported is 16. I probably should have just used > > > the simple [ 1, 2, 4 ] and [ 1, 2, 3, 4 ] arrays, corresponding to > > > vector lengths (in bits) 128, 256, 512 and 128, 256, 384, 512. > > > > I'd argue differently from the ABI perspective. > > > > If the host supports vq = [ 1,2,3,4 ], then it is entirely valid to ask > > for [ 1,2,4 ]. KVM will fail the KVM_REG_ARM64_SVE_VLS write with > > EINVAL, but I don't distinguish between a set that is not satisfiable on > > this hardware and a set that is not valid at all. The kernel basically > > doesn't check for the latter: an architecturally invalid set will always > > not be satisfiable on the hardware. > > I agree that 1,2,4 should be a valid set, but how do we avoid the user > attempting to select it on a platform that supports 1,2,3,4 now? Until > the hardware and KVM will allow it, then I don't think we should try > to "support" it in any way. Maybe we should name the property such that > it's clear we need to use truncation when constructing a subset now. > This would allow us to add another, more generally named, property later > when we can select nearly arbitrary subsets. > > > > > Ideally, userspace should not preempt this decision, in case the > > architecture becomes more flexible someday in what it can virtualise. > > > > > > The host must support all lengths in the sequence, which means > > > > that while Host_A supports 32, since it doesn't support 24 and > > > > we can only truncate sequences, we must use either [ 8 ] or > > > > [ 8, 16 ] for a compatible sequence if we intend to migrate > > > > between the hosts. > > > > > > > > Now to the $SUBJECT question: > > > > > > > > My feeling is that we should require the sequence to be > > > > provided on the command line as a cpu property. Something > > > > like > > > > > > > > -cpu host,sve-vl-list=8:16 > > > > I can't decide whether it's more or less user-friendly to denote VL in > > terms of bytes in this kind of context. > > I'm not sure either. Switching to my second take at quadwords this should > have been '1:2'. In bytes [ 1, 2 ] would be '16:32'. > > > > > Usually when I say "VL" in the kernel ABI I try to mean this. > > (KVM_REG_ARM64_SVE_VLS is a special case: here the lengths are fancily > > encoded rather than being passed as integers, so the precise unit used > > is a more abstract concept here ... or anyway, that's my excuse. > > _SVE_VQS felt odd here.) > > My thinking is that using the number of quadwords keeps the numbers > smaller. But, if people are mostly going to want to think about vector > lengths in terms of bits, like the specification does, then maybe we > should just let the numbers be bigger: [ 128, 256, 512 ] > > > > > For kvmtool I went with quadwords on the command line, but only as a > > quick hack to simplify the parser slightly. > > > > OTOH, specifying 1,2,4 on the command line is clearly less annoying and > > error-prone than 16,32,64. > > > > Naming the option with "vq" instead of "vl" is another option, though > > "vq" is Linux terminology not endorsed by the architecture. > > I think VL is OK as long we document the unit. As you wish (and that's basically where I ended up for now with kvmtool). > > > > For kvmtool I've considered a range / filtering syntax, but I didn't > > implement it yet. > > I think we want each VL to be explicitly listed on the command line > for the compatibility issue described above. On one host the range > [1,4] can be different than the range [1,4] on anther host, depending > on whether '3' is in there. And syntax like 1-4:-3 would be overly > complicated, IMO. So I'm inclined to just require all of them to be > listed, but of course order wouldn't matter. Probably reasonable. For now (and quite likely forever) the length of the list is at most 16 elements. To avoid overengineering I stuck with a flat comma-separated list for now. It's worth nothing that there are two problems to be solved here: one is to specify an exact set unambiguously, which is important for migration scenarios. The other is to be able to clamp the vector length for user convenience, but without particularly considering migration. There's no direct way for the user to know the set of vector lengths supported by KVM today, so cooking up the right magic to go on the command line is non-trivial: you have to create a dummy vcpu and read KVM_REG_ARM64_SVE_VLS to find the host's supported set. Or you just have to know. A separate max-vl option (or whatever) might be a useful alternative. Cheers ---Dave ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? @ 2019-04-18 14:43 ` Andrew Jones 0 siblings, 0 replies; 45+ messages in thread From: Andrew Jones @ 2019-04-18 14:43 UTC (permalink / raw) To: Dave Martin; +Cc: peter.maydell, armbru, qemu-devel, dgilbert, imammedo On Thu, Apr 18, 2019 at 03:03:02PM +0100, Dave Martin wrote: > On Thu, Apr 18, 2019 at 12:28:47PM +0100, Andrew Jones wrote: > > On Thu, Apr 18, 2019 at 11:52:04AM +0100, Dave Martin wrote: > > > On Thu, Apr 18, 2019 at 10:46:34AM +0100, Andrew Jones wrote: > > > > On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote: > > > > > Hi all, > > > > > > > > > > First some background: > > > > > > > > > > For the userspace side of AArch64 guest SVE support we need to > > > > > expose KVM's allowed vector lengths bitmap to the user and allow > > > > > the user to choose a subset of that bitmap. Since bitmaps are a > > > > > bit awkward to work with then we'll likely want to expose it as > > > > > an array of vector lengths instead. Also, assuming we want to > > > > > expose the lengths as number-of-quadwords (quadword == 128 bits > > > > > for AArch64 and vector lengths must be multiples of quadwords) > > > > > rather than number-of-bits, then an example array (which will > > > > > always be a sequence) might be > > > > > > > > > > [ 8, 16, 32 ] > > > > > > > > > > The user may choose a subsequence, but only through truncation, > > > > > i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. > > > > > > > > > > Furthermore, different hosts may support different sequences > > > > > which have the same maximum. For example, if the above sequence > > > > > is for Host_A, then Host_B could be > > > > > > > > > > [ 8, 16, 24, 32 ] > > > > > > > > It doesn't really matter for this discussion, but I just realized > > > > that I picked bogus numbers for these examples. 32 would be too > > > > big. The largest supported is 16. I probably should have just used > > > > the simple [ 1, 2, 4 ] and [ 1, 2, 3, 4 ] arrays, corresponding to > > > > vector lengths (in bits) 128, 256, 512 and 128, 256, 384, 512. > > > > > > I'd argue differently from the ABI perspective. > > > > > > If the host supports vq = [ 1,2,3,4 ], then it is entirely valid to ask > > > for [ 1,2,4 ]. KVM will fail the KVM_REG_ARM64_SVE_VLS write with > > > EINVAL, but I don't distinguish between a set that is not satisfiable on > > > this hardware and a set that is not valid at all. The kernel basically > > > doesn't check for the latter: an architecturally invalid set will always > > > not be satisfiable on the hardware. > > > > I agree that 1,2,4 should be a valid set, but how do we avoid the user > > attempting to select it on a platform that supports 1,2,3,4 now? Until > > the hardware and KVM will allow it, then I don't think we should try > > to "support" it in any way. Maybe we should name the property such that > > it's clear we need to use truncation when constructing a subset now. > > This would allow us to add another, more generally named, property later > > when we can select nearly arbitrary subsets. > > > > > > > > Ideally, userspace should not preempt this decision, in case the > > > architecture becomes more flexible someday in what it can virtualise. > > > > > > > > The host must support all lengths in the sequence, which means > > > > > that while Host_A supports 32, since it doesn't support 24 and > > > > > we can only truncate sequences, we must use either [ 8 ] or > > > > > [ 8, 16 ] for a compatible sequence if we intend to migrate > > > > > between the hosts. > > > > > > > > > > Now to the $SUBJECT question: > > > > > > > > > > My feeling is that we should require the sequence to be > > > > > provided on the command line as a cpu property. Something > > > > > like > > > > > > > > > > -cpu host,sve-vl-list=8:16 > > > > > > I can't decide whether it's more or less user-friendly to denote VL in > > > terms of bytes in this kind of context. > > > > I'm not sure either. Switching to my second take at quadwords this should > > have been '1:2'. In bytes [ 1, 2 ] would be '16:32'. > > > > > > > > Usually when I say "VL" in the kernel ABI I try to mean this. > > > (KVM_REG_ARM64_SVE_VLS is a special case: here the lengths are fancily > > > encoded rather than being passed as integers, so the precise unit used > > > is a more abstract concept here ... or anyway, that's my excuse. > > > _SVE_VQS felt odd here.) > > > > My thinking is that using the number of quadwords keeps the numbers > > smaller. But, if people are mostly going to want to think about vector > > lengths in terms of bits, like the specification does, then maybe we > > should just let the numbers be bigger: [ 128, 256, 512 ] > > > > > > > > For kvmtool I went with quadwords on the command line, but only as a > > > quick hack to simplify the parser slightly. > > > > > > OTOH, specifying 1,2,4 on the command line is clearly less annoying and > > > error-prone than 16,32,64. > > > > > > Naming the option with "vq" instead of "vl" is another option, though > > > "vq" is Linux terminology not endorsed by the architecture. > > > > I think VL is OK as long we document the unit. > > As you wish (and that's basically where I ended up for now with kvmtool). > > > > > > > For kvmtool I've considered a range / filtering syntax, but I didn't > > > implement it yet. > > > > I think we want each VL to be explicitly listed on the command line > > for the compatibility issue described above. On one host the range > > [1,4] can be different than the range [1,4] on anther host, depending > > on whether '3' is in there. And syntax like 1-4:-3 would be overly > > complicated, IMO. So I'm inclined to just require all of them to be > > listed, but of course order wouldn't matter. > > Probably reasonable. For now (and quite likely forever) the length of > the list is at most 16 elements. To avoid overengineering I stuck with > a flat comma-separated list for now. > > > It's worth nothing that there are two problems to be solved here: one is > to specify an exact set unambiguously, which is important for migration > scenarios. > > The other is to be able to clamp the vector length for user convenience, > but without particularly considering migration. There's no direct way > for the user to know the set of vector lengths supported by KVM today, > so cooking up the right magic to go on the command line is non-trivial: > you have to create a dummy vcpu and read KVM_REG_ARM64_SVE_VLS to find > the host's supported set. Or you just have to know. Right, we'll have to query first to know what's available. libvirt will do that and tools built on libvirt should help provide the user with sane defaults or a relatively painless selection process. The QEMU command line is primarily for developers, so they'll likely just know what they want. > > A separate max-vl option (or whatever) might be a useful alternative. Yeah, we could offer this to be nicer to QEMU command line users that don't intend to migrate, or don't care if a migration can fail. This would be analogous to '-cpu max' which enables all available cpu features. Bad for migration, but a nice shorthand if you want your guest to have everything. Thanks, drew ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? @ 2019-04-18 14:43 ` Andrew Jones 0 siblings, 0 replies; 45+ messages in thread From: Andrew Jones @ 2019-04-18 14:43 UTC (permalink / raw) To: Dave Martin; +Cc: peter.maydell, imammedo, armbru, dgilbert, qemu-devel On Thu, Apr 18, 2019 at 03:03:02PM +0100, Dave Martin wrote: > On Thu, Apr 18, 2019 at 12:28:47PM +0100, Andrew Jones wrote: > > On Thu, Apr 18, 2019 at 11:52:04AM +0100, Dave Martin wrote: > > > On Thu, Apr 18, 2019 at 10:46:34AM +0100, Andrew Jones wrote: > > > > On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote: > > > > > Hi all, > > > > > > > > > > First some background: > > > > > > > > > > For the userspace side of AArch64 guest SVE support we need to > > > > > expose KVM's allowed vector lengths bitmap to the user and allow > > > > > the user to choose a subset of that bitmap. Since bitmaps are a > > > > > bit awkward to work with then we'll likely want to expose it as > > > > > an array of vector lengths instead. Also, assuming we want to > > > > > expose the lengths as number-of-quadwords (quadword == 128 bits > > > > > for AArch64 and vector lengths must be multiples of quadwords) > > > > > rather than number-of-bits, then an example array (which will > > > > > always be a sequence) might be > > > > > > > > > > [ 8, 16, 32 ] > > > > > > > > > > The user may choose a subsequence, but only through truncation, > > > > > i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. > > > > > > > > > > Furthermore, different hosts may support different sequences > > > > > which have the same maximum. For example, if the above sequence > > > > > is for Host_A, then Host_B could be > > > > > > > > > > [ 8, 16, 24, 32 ] > > > > > > > > It doesn't really matter for this discussion, but I just realized > > > > that I picked bogus numbers for these examples. 32 would be too > > > > big. The largest supported is 16. I probably should have just used > > > > the simple [ 1, 2, 4 ] and [ 1, 2, 3, 4 ] arrays, corresponding to > > > > vector lengths (in bits) 128, 256, 512 and 128, 256, 384, 512. > > > > > > I'd argue differently from the ABI perspective. > > > > > > If the host supports vq = [ 1,2,3,4 ], then it is entirely valid to ask > > > for [ 1,2,4 ]. KVM will fail the KVM_REG_ARM64_SVE_VLS write with > > > EINVAL, but I don't distinguish between a set that is not satisfiable on > > > this hardware and a set that is not valid at all. The kernel basically > > > doesn't check for the latter: an architecturally invalid set will always > > > not be satisfiable on the hardware. > > > > I agree that 1,2,4 should be a valid set, but how do we avoid the user > > attempting to select it on a platform that supports 1,2,3,4 now? Until > > the hardware and KVM will allow it, then I don't think we should try > > to "support" it in any way. Maybe we should name the property such that > > it's clear we need to use truncation when constructing a subset now. > > This would allow us to add another, more generally named, property later > > when we can select nearly arbitrary subsets. > > > > > > > > Ideally, userspace should not preempt this decision, in case the > > > architecture becomes more flexible someday in what it can virtualise. > > > > > > > > The host must support all lengths in the sequence, which means > > > > > that while Host_A supports 32, since it doesn't support 24 and > > > > > we can only truncate sequences, we must use either [ 8 ] or > > > > > [ 8, 16 ] for a compatible sequence if we intend to migrate > > > > > between the hosts. > > > > > > > > > > Now to the $SUBJECT question: > > > > > > > > > > My feeling is that we should require the sequence to be > > > > > provided on the command line as a cpu property. Something > > > > > like > > > > > > > > > > -cpu host,sve-vl-list=8:16 > > > > > > I can't decide whether it's more or less user-friendly to denote VL in > > > terms of bytes in this kind of context. > > > > I'm not sure either. Switching to my second take at quadwords this should > > have been '1:2'. In bytes [ 1, 2 ] would be '16:32'. > > > > > > > > Usually when I say "VL" in the kernel ABI I try to mean this. > > > (KVM_REG_ARM64_SVE_VLS is a special case: here the lengths are fancily > > > encoded rather than being passed as integers, so the precise unit used > > > is a more abstract concept here ... or anyway, that's my excuse. > > > _SVE_VQS felt odd here.) > > > > My thinking is that using the number of quadwords keeps the numbers > > smaller. But, if people are mostly going to want to think about vector > > lengths in terms of bits, like the specification does, then maybe we > > should just let the numbers be bigger: [ 128, 256, 512 ] > > > > > > > > For kvmtool I went with quadwords on the command line, but only as a > > > quick hack to simplify the parser slightly. > > > > > > OTOH, specifying 1,2,4 on the command line is clearly less annoying and > > > error-prone than 16,32,64. > > > > > > Naming the option with "vq" instead of "vl" is another option, though > > > "vq" is Linux terminology not endorsed by the architecture. > > > > I think VL is OK as long we document the unit. > > As you wish (and that's basically where I ended up for now with kvmtool). > > > > > > > For kvmtool I've considered a range / filtering syntax, but I didn't > > > implement it yet. > > > > I think we want each VL to be explicitly listed on the command line > > for the compatibility issue described above. On one host the range > > [1,4] can be different than the range [1,4] on anther host, depending > > on whether '3' is in there. And syntax like 1-4:-3 would be overly > > complicated, IMO. So I'm inclined to just require all of them to be > > listed, but of course order wouldn't matter. > > Probably reasonable. For now (and quite likely forever) the length of > the list is at most 16 elements. To avoid overengineering I stuck with > a flat comma-separated list for now. > > > It's worth nothing that there are two problems to be solved here: one is > to specify an exact set unambiguously, which is important for migration > scenarios. > > The other is to be able to clamp the vector length for user convenience, > but without particularly considering migration. There's no direct way > for the user to know the set of vector lengths supported by KVM today, > so cooking up the right magic to go on the command line is non-trivial: > you have to create a dummy vcpu and read KVM_REG_ARM64_SVE_VLS to find > the host's supported set. Or you just have to know. Right, we'll have to query first to know what's available. libvirt will do that and tools built on libvirt should help provide the user with sane defaults or a relatively painless selection process. The QEMU command line is primarily for developers, so they'll likely just know what they want. > > A separate max-vl option (or whatever) might be a useful alternative. Yeah, we could offer this to be nicer to QEMU command line users that don't intend to migrate, or don't care if a migration can fail. This would be analogous to '-cpu max' which enables all available cpu features. Bad for migration, but a nice shorthand if you want your guest to have everything. Thanks, drew ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? @ 2019-04-18 14:46 ` Dave Martin 0 siblings, 0 replies; 45+ messages in thread From: Dave Martin @ 2019-04-18 14:46 UTC (permalink / raw) To: Andrew Jones; +Cc: peter.maydell, armbru, qemu-devel, dgilbert, imammedo On Thu, Apr 18, 2019 at 03:43:06PM +0100, Andrew Jones wrote: > On Thu, Apr 18, 2019 at 03:03:02PM +0100, Dave Martin wrote: [...] > > It's worth nothing that there are two problems to be solved here: one is > > to specify an exact set unambiguously, which is important for migration > > scenarios. > > > > The other is to be able to clamp the vector length for user convenience, > > but without particularly considering migration. There's no direct way > > for the user to know the set of vector lengths supported by KVM today, > > so cooking up the right magic to go on the command line is non-trivial: > > you have to create a dummy vcpu and read KVM_REG_ARM64_SVE_VLS to find > > the host's supported set. Or you just have to know. > > Right, we'll have to query first to know what's available. libvirt will > do that and tools built on libvirt should help provide the user with > sane defaults or a relatively painless selection process. The QEMU command > line is primarily for developers, so they'll likely just know what they > want. > > > > > A separate max-vl option (or whatever) might be a useful alternative. > > Yeah, we could offer this to be nicer to QEMU command line users that > don't intend to migrate, or don't care if a migration can fail. This > would be analogous to '-cpu max' which enables all available cpu > features. Bad for migration, but a nice shorthand if you want your > guest to have everything. Sounds reasonable. Cheers ---Dave ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? @ 2019-04-18 14:46 ` Dave Martin 0 siblings, 0 replies; 45+ messages in thread From: Dave Martin @ 2019-04-18 14:46 UTC (permalink / raw) To: Andrew Jones; +Cc: peter.maydell, imammedo, armbru, dgilbert, qemu-devel On Thu, Apr 18, 2019 at 03:43:06PM +0100, Andrew Jones wrote: > On Thu, Apr 18, 2019 at 03:03:02PM +0100, Dave Martin wrote: [...] > > It's worth nothing that there are two problems to be solved here: one is > > to specify an exact set unambiguously, which is important for migration > > scenarios. > > > > The other is to be able to clamp the vector length for user convenience, > > but without particularly considering migration. There's no direct way > > for the user to know the set of vector lengths supported by KVM today, > > so cooking up the right magic to go on the command line is non-trivial: > > you have to create a dummy vcpu and read KVM_REG_ARM64_SVE_VLS to find > > the host's supported set. Or you just have to know. > > Right, we'll have to query first to know what's available. libvirt will > do that and tools built on libvirt should help provide the user with > sane defaults or a relatively painless selection process. The QEMU command > line is primarily for developers, so they'll likely just know what they > want. > > > > > A separate max-vl option (or whatever) might be a useful alternative. > > Yeah, we could offer this to be nicer to QEMU command line users that > don't intend to migrate, or don't care if a migration can fail. This > would be analogous to '-cpu max' which enables all available cpu > features. Bad for migration, but a nice shorthand if you want your > guest to have everything. Sounds reasonable. Cheers ---Dave ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? @ 2019-04-18 14:57 ` Dr. David Alan Gilbert 0 siblings, 0 replies; 45+ messages in thread From: Dr. David Alan Gilbert @ 2019-04-18 14:57 UTC (permalink / raw) To: Andrew Jones; +Cc: Dave Martin, peter.maydell, armbru, qemu-devel, imammedo * Andrew Jones (drjones@redhat.com) wrote: > On Thu, Apr 18, 2019 at 03:03:02PM +0100, Dave Martin wrote: > > On Thu, Apr 18, 2019 at 12:28:47PM +0100, Andrew Jones wrote: > > > On Thu, Apr 18, 2019 at 11:52:04AM +0100, Dave Martin wrote: > > > > On Thu, Apr 18, 2019 at 10:46:34AM +0100, Andrew Jones wrote: > > > > > On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote: > > > > > > Hi all, > > > > > > > > > > > > First some background: > > > > > > > > > > > > For the userspace side of AArch64 guest SVE support we need to > > > > > > expose KVM's allowed vector lengths bitmap to the user and allow > > > > > > the user to choose a subset of that bitmap. Since bitmaps are a > > > > > > bit awkward to work with then we'll likely want to expose it as > > > > > > an array of vector lengths instead. Also, assuming we want to > > > > > > expose the lengths as number-of-quadwords (quadword == 128 bits > > > > > > for AArch64 and vector lengths must be multiples of quadwords) > > > > > > rather than number-of-bits, then an example array (which will > > > > > > always be a sequence) might be > > > > > > > > > > > > [ 8, 16, 32 ] > > > > > > > > > > > > The user may choose a subsequence, but only through truncation, > > > > > > i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. > > > > > > > > > > > > Furthermore, different hosts may support different sequences > > > > > > which have the same maximum. For example, if the above sequence > > > > > > is for Host_A, then Host_B could be > > > > > > > > > > > > [ 8, 16, 24, 32 ] > > > > > > > > > > It doesn't really matter for this discussion, but I just realized > > > > > that I picked bogus numbers for these examples. 32 would be too > > > > > big. The largest supported is 16. I probably should have just used > > > > > the simple [ 1, 2, 4 ] and [ 1, 2, 3, 4 ] arrays, corresponding to > > > > > vector lengths (in bits) 128, 256, 512 and 128, 256, 384, 512. > > > > > > > > I'd argue differently from the ABI perspective. > > > > > > > > If the host supports vq = [ 1,2,3,4 ], then it is entirely valid to ask > > > > for [ 1,2,4 ]. KVM will fail the KVM_REG_ARM64_SVE_VLS write with > > > > EINVAL, but I don't distinguish between a set that is not satisfiable on > > > > this hardware and a set that is not valid at all. The kernel basically > > > > doesn't check for the latter: an architecturally invalid set will always > > > > not be satisfiable on the hardware. > > > > > > I agree that 1,2,4 should be a valid set, but how do we avoid the user > > > attempting to select it on a platform that supports 1,2,3,4 now? Until > > > the hardware and KVM will allow it, then I don't think we should try > > > to "support" it in any way. Maybe we should name the property such that > > > it's clear we need to use truncation when constructing a subset now. > > > This would allow us to add another, more generally named, property later > > > when we can select nearly arbitrary subsets. > > > > > > > > > > > Ideally, userspace should not preempt this decision, in case the > > > > architecture becomes more flexible someday in what it can virtualise. > > > > > > > > > > The host must support all lengths in the sequence, which means > > > > > > that while Host_A supports 32, since it doesn't support 24 and > > > > > > we can only truncate sequences, we must use either [ 8 ] or > > > > > > [ 8, 16 ] for a compatible sequence if we intend to migrate > > > > > > between the hosts. > > > > > > > > > > > > Now to the $SUBJECT question: > > > > > > > > > > > > My feeling is that we should require the sequence to be > > > > > > provided on the command line as a cpu property. Something > > > > > > like > > > > > > > > > > > > -cpu host,sve-vl-list=8:16 > > > > > > > > I can't decide whether it's more or less user-friendly to denote VL in > > > > terms of bytes in this kind of context. > > > > > > I'm not sure either. Switching to my second take at quadwords this should > > > have been '1:2'. In bytes [ 1, 2 ] would be '16:32'. > > > > > > > > > > > Usually when I say "VL" in the kernel ABI I try to mean this. > > > > (KVM_REG_ARM64_SVE_VLS is a special case: here the lengths are fancily > > > > encoded rather than being passed as integers, so the precise unit used > > > > is a more abstract concept here ... or anyway, that's my excuse. > > > > _SVE_VQS felt odd here.) > > > > > > My thinking is that using the number of quadwords keeps the numbers > > > smaller. But, if people are mostly going to want to think about vector > > > lengths in terms of bits, like the specification does, then maybe we > > > should just let the numbers be bigger: [ 128, 256, 512 ] > > > > > > > > > > > For kvmtool I went with quadwords on the command line, but only as a > > > > quick hack to simplify the parser slightly. > > > > > > > > OTOH, specifying 1,2,4 on the command line is clearly less annoying and > > > > error-prone than 16,32,64. > > > > > > > > Naming the option with "vq" instead of "vl" is another option, though > > > > "vq" is Linux terminology not endorsed by the architecture. > > > > > > I think VL is OK as long we document the unit. > > > > As you wish (and that's basically where I ended up for now with kvmtool). > > > > > > > > > > For kvmtool I've considered a range / filtering syntax, but I didn't > > > > implement it yet. > > > > > > I think we want each VL to be explicitly listed on the command line > > > for the compatibility issue described above. On one host the range > > > [1,4] can be different than the range [1,4] on anther host, depending > > > on whether '3' is in there. And syntax like 1-4:-3 would be overly > > > complicated, IMO. So I'm inclined to just require all of them to be > > > listed, but of course order wouldn't matter. > > > > Probably reasonable. For now (and quite likely forever) the length of > > the list is at most 16 elements. To avoid overengineering I stuck with > > a flat comma-separated list for now. > > > > > > It's worth nothing that there are two problems to be solved here: one is > > to specify an exact set unambiguously, which is important for migration > > scenarios. > > > > The other is to be able to clamp the vector length for user convenience, > > but without particularly considering migration. There's no direct way > > for the user to know the set of vector lengths supported by KVM today, > > so cooking up the right magic to go on the command line is non-trivial: > > you have to create a dummy vcpu and read KVM_REG_ARM64_SVE_VLS to find > > the host's supported set. Or you just have to know. > > Right, we'll have to query first to know what's available. libvirt will > do that and tools built on libvirt should help provide the user with > sane defaults or a relatively painless selection process. The QEMU command > line is primarily for developers, so they'll likely just know what they > want. > > > > > A separate max-vl option (or whatever) might be a useful alternative. > > Yeah, we could offer this to be nicer to QEMU command line users that > don't intend to migrate, or don't care if a migration can fail. This > would be analogous to '-cpu max' which enables all available cpu > features. Bad for migration, but a nice shorthand if you want your > guest to have everything. One that printed out what was available on that host might be nice (and/or via a qmp query- command) for when you're trying to figure out what's going. Dave > Thanks, > drew -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? @ 2019-04-18 14:57 ` Dr. David Alan Gilbert 0 siblings, 0 replies; 45+ messages in thread From: Dr. David Alan Gilbert @ 2019-04-18 14:57 UTC (permalink / raw) To: Andrew Jones; +Cc: qemu-devel, peter.maydell, imammedo, Dave Martin, armbru * Andrew Jones (drjones@redhat.com) wrote: > On Thu, Apr 18, 2019 at 03:03:02PM +0100, Dave Martin wrote: > > On Thu, Apr 18, 2019 at 12:28:47PM +0100, Andrew Jones wrote: > > > On Thu, Apr 18, 2019 at 11:52:04AM +0100, Dave Martin wrote: > > > > On Thu, Apr 18, 2019 at 10:46:34AM +0100, Andrew Jones wrote: > > > > > On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote: > > > > > > Hi all, > > > > > > > > > > > > First some background: > > > > > > > > > > > > For the userspace side of AArch64 guest SVE support we need to > > > > > > expose KVM's allowed vector lengths bitmap to the user and allow > > > > > > the user to choose a subset of that bitmap. Since bitmaps are a > > > > > > bit awkward to work with then we'll likely want to expose it as > > > > > > an array of vector lengths instead. Also, assuming we want to > > > > > > expose the lengths as number-of-quadwords (quadword == 128 bits > > > > > > for AArch64 and vector lengths must be multiples of quadwords) > > > > > > rather than number-of-bits, then an example array (which will > > > > > > always be a sequence) might be > > > > > > > > > > > > [ 8, 16, 32 ] > > > > > > > > > > > > The user may choose a subsequence, but only through truncation, > > > > > > i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. > > > > > > > > > > > > Furthermore, different hosts may support different sequences > > > > > > which have the same maximum. For example, if the above sequence > > > > > > is for Host_A, then Host_B could be > > > > > > > > > > > > [ 8, 16, 24, 32 ] > > > > > > > > > > It doesn't really matter for this discussion, but I just realized > > > > > that I picked bogus numbers for these examples. 32 would be too > > > > > big. The largest supported is 16. I probably should have just used > > > > > the simple [ 1, 2, 4 ] and [ 1, 2, 3, 4 ] arrays, corresponding to > > > > > vector lengths (in bits) 128, 256, 512 and 128, 256, 384, 512. > > > > > > > > I'd argue differently from the ABI perspective. > > > > > > > > If the host supports vq = [ 1,2,3,4 ], then it is entirely valid to ask > > > > for [ 1,2,4 ]. KVM will fail the KVM_REG_ARM64_SVE_VLS write with > > > > EINVAL, but I don't distinguish between a set that is not satisfiable on > > > > this hardware and a set that is not valid at all. The kernel basically > > > > doesn't check for the latter: an architecturally invalid set will always > > > > not be satisfiable on the hardware. > > > > > > I agree that 1,2,4 should be a valid set, but how do we avoid the user > > > attempting to select it on a platform that supports 1,2,3,4 now? Until > > > the hardware and KVM will allow it, then I don't think we should try > > > to "support" it in any way. Maybe we should name the property such that > > > it's clear we need to use truncation when constructing a subset now. > > > This would allow us to add another, more generally named, property later > > > when we can select nearly arbitrary subsets. > > > > > > > > > > > Ideally, userspace should not preempt this decision, in case the > > > > architecture becomes more flexible someday in what it can virtualise. > > > > > > > > > > The host must support all lengths in the sequence, which means > > > > > > that while Host_A supports 32, since it doesn't support 24 and > > > > > > we can only truncate sequences, we must use either [ 8 ] or > > > > > > [ 8, 16 ] for a compatible sequence if we intend to migrate > > > > > > between the hosts. > > > > > > > > > > > > Now to the $SUBJECT question: > > > > > > > > > > > > My feeling is that we should require the sequence to be > > > > > > provided on the command line as a cpu property. Something > > > > > > like > > > > > > > > > > > > -cpu host,sve-vl-list=8:16 > > > > > > > > I can't decide whether it's more or less user-friendly to denote VL in > > > > terms of bytes in this kind of context. > > > > > > I'm not sure either. Switching to my second take at quadwords this should > > > have been '1:2'. In bytes [ 1, 2 ] would be '16:32'. > > > > > > > > > > > Usually when I say "VL" in the kernel ABI I try to mean this. > > > > (KVM_REG_ARM64_SVE_VLS is a special case: here the lengths are fancily > > > > encoded rather than being passed as integers, so the precise unit used > > > > is a more abstract concept here ... or anyway, that's my excuse. > > > > _SVE_VQS felt odd here.) > > > > > > My thinking is that using the number of quadwords keeps the numbers > > > smaller. But, if people are mostly going to want to think about vector > > > lengths in terms of bits, like the specification does, then maybe we > > > should just let the numbers be bigger: [ 128, 256, 512 ] > > > > > > > > > > > For kvmtool I went with quadwords on the command line, but only as a > > > > quick hack to simplify the parser slightly. > > > > > > > > OTOH, specifying 1,2,4 on the command line is clearly less annoying and > > > > error-prone than 16,32,64. > > > > > > > > Naming the option with "vq" instead of "vl" is another option, though > > > > "vq" is Linux terminology not endorsed by the architecture. > > > > > > I think VL is OK as long we document the unit. > > > > As you wish (and that's basically where I ended up for now with kvmtool). > > > > > > > > > > For kvmtool I've considered a range / filtering syntax, but I didn't > > > > implement it yet. > > > > > > I think we want each VL to be explicitly listed on the command line > > > for the compatibility issue described above. On one host the range > > > [1,4] can be different than the range [1,4] on anther host, depending > > > on whether '3' is in there. And syntax like 1-4:-3 would be overly > > > complicated, IMO. So I'm inclined to just require all of them to be > > > listed, but of course order wouldn't matter. > > > > Probably reasonable. For now (and quite likely forever) the length of > > the list is at most 16 elements. To avoid overengineering I stuck with > > a flat comma-separated list for now. > > > > > > It's worth nothing that there are two problems to be solved here: one is > > to specify an exact set unambiguously, which is important for migration > > scenarios. > > > > The other is to be able to clamp the vector length for user convenience, > > but without particularly considering migration. There's no direct way > > for the user to know the set of vector lengths supported by KVM today, > > so cooking up the right magic to go on the command line is non-trivial: > > you have to create a dummy vcpu and read KVM_REG_ARM64_SVE_VLS to find > > the host's supported set. Or you just have to know. > > Right, we'll have to query first to know what's available. libvirt will > do that and tools built on libvirt should help provide the user with > sane defaults or a relatively painless selection process. The QEMU command > line is primarily for developers, so they'll likely just know what they > want. > > > > > A separate max-vl option (or whatever) might be a useful alternative. > > Yeah, we could offer this to be nicer to QEMU command line users that > don't intend to migrate, or don't care if a migration can fail. This > would be analogous to '-cpu max' which enables all available cpu > features. Bad for migration, but a nice shorthand if you want your > guest to have everything. One that printed out what was available on that host might be nice (and/or via a qmp query- command) for when you're trying to figure out what's going. Dave > Thanks, > drew -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? @ 2019-04-18 11:26 ` Daniel P. Berrangé 0 siblings, 0 replies; 45+ messages in thread From: Daniel P. Berrangé @ 2019-04-18 11:26 UTC (permalink / raw) To: Andrew Jones Cc: qemu-devel, peter.maydell, imammedo, Dave.Martin, armbru, dgilbert On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote: > Hi all, > > First some background: > > For the userspace side of AArch64 guest SVE support we need to > expose KVM's allowed vector lengths bitmap to the user and allow > the user to choose a subset of that bitmap. Since bitmaps are a > bit awkward to work with then we'll likely want to expose it as > an array of vector lengths instead. Also, assuming we want to > expose the lengths as number-of-quadwords (quadword == 128 bits > for AArch64 and vector lengths must be multiples of quadwords) > rather than number-of-bits, then an example array (which will > always be a sequence) might be > > [ 8, 16, 32 ] > > The user may choose a subsequence, but only through truncation, > i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. > > Furthermore, different hosts may support different sequences > which have the same maximum. For example, if the above sequence > is for Host_A, then Host_B could be > > [ 8, 16, 24, 32 ] > > The host must support all lengths in the sequence, which means > that while Host_A supports 32, since it doesn't support 24 and > we can only truncate sequences, we must use either [ 8 ] or > [ 8, 16 ] for a compatible sequence if we intend to migrate > between the hosts. > > Now to the $SUBJECT question: > > My feeling is that we should require the sequence to be > provided on the command line as a cpu property. Something > like > > -cpu host,sve-vl-list=8:16 > > (I chose ':' for the delimiter because ',' can't work, but > if there's a better choice, then that's fine by me.) > > Afaict a property list like this will require a new parser, > which feels a bit funny since it seems we should already > have support for this type of thing somewhere in QEMU. So, > the question is: do we? I see we have array properties, but > I don't believe that works with the command line. Should we > only use QMP for this? We already want some QMP in order to > query the supported vector lengths. Maybe we should use QMP > to set the selection too? But then what about command line > support for developers? And if the property is on the command > line then we don't have to add it to the migration stream. You should be able to use arrays from the CLI with QemuOpts by repeating the same option name many times, though I can't say it is a very nice approach if you have many values to list as it gets very repetative. That's the curse of not having a good CLI syntax for non-scalar data in QemuOpts & why Markus believes we should switch to JSON for the CLI too -cpu host,sve-vl=8,sve-vl=16 Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? @ 2019-04-18 11:26 ` Daniel P. Berrangé 0 siblings, 0 replies; 45+ messages in thread From: Daniel P. Berrangé @ 2019-04-18 11:26 UTC (permalink / raw) To: Andrew Jones Cc: peter.maydell, armbru, dgilbert, qemu-devel, imammedo, Dave.Martin On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote: > Hi all, > > First some background: > > For the userspace side of AArch64 guest SVE support we need to > expose KVM's allowed vector lengths bitmap to the user and allow > the user to choose a subset of that bitmap. Since bitmaps are a > bit awkward to work with then we'll likely want to expose it as > an array of vector lengths instead. Also, assuming we want to > expose the lengths as number-of-quadwords (quadword == 128 bits > for AArch64 and vector lengths must be multiples of quadwords) > rather than number-of-bits, then an example array (which will > always be a sequence) might be > > [ 8, 16, 32 ] > > The user may choose a subsequence, but only through truncation, > i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. > > Furthermore, different hosts may support different sequences > which have the same maximum. For example, if the above sequence > is for Host_A, then Host_B could be > > [ 8, 16, 24, 32 ] > > The host must support all lengths in the sequence, which means > that while Host_A supports 32, since it doesn't support 24 and > we can only truncate sequences, we must use either [ 8 ] or > [ 8, 16 ] for a compatible sequence if we intend to migrate > between the hosts. > > Now to the $SUBJECT question: > > My feeling is that we should require the sequence to be > provided on the command line as a cpu property. Something > like > > -cpu host,sve-vl-list=8:16 > > (I chose ':' for the delimiter because ',' can't work, but > if there's a better choice, then that's fine by me.) > > Afaict a property list like this will require a new parser, > which feels a bit funny since it seems we should already > have support for this type of thing somewhere in QEMU. So, > the question is: do we? I see we have array properties, but > I don't believe that works with the command line. Should we > only use QMP for this? We already want some QMP in order to > query the supported vector lengths. Maybe we should use QMP > to set the selection too? But then what about command line > support for developers? And if the property is on the command > line then we don't have to add it to the migration stream. You should be able to use arrays from the CLI with QemuOpts by repeating the same option name many times, though I can't say it is a very nice approach if you have many values to list as it gets very repetative. That's the curse of not having a good CLI syntax for non-scalar data in QemuOpts & why Markus believes we should switch to JSON for the CLI too -cpu host,sve-vl=8,sve-vl=16 Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? @ 2019-04-18 15:09 ` Andrew Jones 0 siblings, 0 replies; 45+ messages in thread From: Andrew Jones @ 2019-04-18 15:09 UTC (permalink / raw) To: Daniel P. Berrangé Cc: qemu-devel, peter.maydell, imammedo, Dave.Martin, armbru, dgilbert On Thu, Apr 18, 2019 at 12:26:10PM +0100, Daniel P. Berrangé wrote: > On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote: > > Hi all, > > > > First some background: > > > > For the userspace side of AArch64 guest SVE support we need to > > expose KVM's allowed vector lengths bitmap to the user and allow > > the user to choose a subset of that bitmap. Since bitmaps are a > > bit awkward to work with then we'll likely want to expose it as > > an array of vector lengths instead. Also, assuming we want to > > expose the lengths as number-of-quadwords (quadword == 128 bits > > for AArch64 and vector lengths must be multiples of quadwords) > > rather than number-of-bits, then an example array (which will > > always be a sequence) might be > > > > [ 8, 16, 32 ] > > > > The user may choose a subsequence, but only through truncation, > > i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. > > > > Furthermore, different hosts may support different sequences > > which have the same maximum. For example, if the above sequence > > is for Host_A, then Host_B could be > > > > [ 8, 16, 24, 32 ] > > > > The host must support all lengths in the sequence, which means > > that while Host_A supports 32, since it doesn't support 24 and > > we can only truncate sequences, we must use either [ 8 ] or > > [ 8, 16 ] for a compatible sequence if we intend to migrate > > between the hosts. > > > > Now to the $SUBJECT question: > > > > My feeling is that we should require the sequence to be > > provided on the command line as a cpu property. Something > > like > > > > -cpu host,sve-vl-list=8:16 > > > > (I chose ':' for the delimiter because ',' can't work, but > > if there's a better choice, then that's fine by me.) > > > > Afaict a property list like this will require a new parser, > > which feels a bit funny since it seems we should already > > have support for this type of thing somewhere in QEMU. So, > > the question is: do we? I see we have array properties, but > > I don't believe that works with the command line. Should we > > only use QMP for this? We already want some QMP in order to > > query the supported vector lengths. Maybe we should use QMP > > to set the selection too? But then what about command line > > support for developers? And if the property is on the command > > line then we don't have to add it to the migration stream. > > You should be able to use arrays from the CLI with QemuOpts by repeating > the same option name many times, though I can't say it is a very > nice approach if you have many values to list as it gets very repetative. > That's the curse of not having a good CLI syntax for non-scalar data in > QemuOpts & why Markus believes we should switch to JSON for the CLI too > > -cpu host,sve-vl=8,sve-vl=16 This might be OK if it integrates nicely with QMP, etc, as the need for a more verbose command line may be better than adding another parser, etc. that Markus will have to rework someday :) Can you point me to a property that needs/uses this already so I can look at its code? Thanks, drew ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? @ 2019-04-18 15:09 ` Andrew Jones 0 siblings, 0 replies; 45+ messages in thread From: Andrew Jones @ 2019-04-18 15:09 UTC (permalink / raw) To: Daniel P. Berrangé Cc: peter.maydell, armbru, dgilbert, qemu-devel, imammedo, Dave.Martin [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain; charset="UTF-8", Size: 3150 bytes --] On Thu, Apr 18, 2019 at 12:26:10PM +0100, Daniel P. Berrangé wrote: > On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote: > > Hi all, > > > > First some background: > > > > For the userspace side of AArch64 guest SVE support we need to > > expose KVM's allowed vector lengths bitmap to the user and allow > > the user to choose a subset of that bitmap. Since bitmaps are a > > bit awkward to work with then we'll likely want to expose it as > > an array of vector lengths instead. Also, assuming we want to > > expose the lengths as number-of-quadwords (quadword == 128 bits > > for AArch64 and vector lengths must be multiples of quadwords) > > rather than number-of-bits, then an example array (which will > > always be a sequence) might be > > > > [ 8, 16, 32 ] > > > > The user may choose a subsequence, but only through truncation, > > i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. > > > > Furthermore, different hosts may support different sequences > > which have the same maximum. For example, if the above sequence > > is for Host_A, then Host_B could be > > > > [ 8, 16, 24, 32 ] > > > > The host must support all lengths in the sequence, which means > > that while Host_A supports 32, since it doesn't support 24 and > > we can only truncate sequences, we must use either [ 8 ] or > > [ 8, 16 ] for a compatible sequence if we intend to migrate > > between the hosts. > > > > Now to the $SUBJECT question: > > > > My feeling is that we should require the sequence to be > > provided on the command line as a cpu property. Something > > like > > > > -cpu host,sve-vl-list=8:16 > > > > (I chose ':' for the delimiter because ',' can't work, but > > if there's a better choice, then that's fine by me.) > > > > Afaict a property list like this will require a new parser, > > which feels a bit funny since it seems we should already > > have support for this type of thing somewhere in QEMU. So, > > the question is: do we? I see we have array properties, but > > I don't believe that works with the command line. Should we > > only use QMP for this? We already want some QMP in order to > > query the supported vector lengths. Maybe we should use QMP > > to set the selection too? But then what about command line > > support for developers? And if the property is on the command > > line then we don't have to add it to the migration stream. > > You should be able to use arrays from the CLI with QemuOpts by repeating > the same option name many times, though I can't say it is a very > nice approach if you have many values to list as it gets very repetative. > That's the curse of not having a good CLI syntax for non-scalar data in > QemuOpts & why Markus believes we should switch to JSON for the CLI too > > -cpu host,sve-vl=8,sve-vl=16 This might be OK if it integrates nicely with QMP, etc, as the need for a more verbose command line may be better than adding another parser, etc. that Markus will have to rework someday :) Can you point me to a property that needs/uses this already so I can look at its code? Thanks, drew ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? @ 2019-04-18 17:48 ` Markus Armbruster 0 siblings, 0 replies; 45+ messages in thread From: Markus Armbruster @ 2019-04-18 17:48 UTC (permalink / raw) To: Daniel P. Berrangé Cc: Andrew Jones, peter.maydell, dgilbert, qemu-devel, imammedo, Dave.Martin Daniel P. Berrangé <berrange@redhat.com> writes: > On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote: >> Hi all, >> >> First some background: >> >> For the userspace side of AArch64 guest SVE support we need to >> expose KVM's allowed vector lengths bitmap to the user and allow >> the user to choose a subset of that bitmap. Since bitmaps are a >> bit awkward to work with then we'll likely want to expose it as >> an array of vector lengths instead. Also, assuming we want to >> expose the lengths as number-of-quadwords (quadword == 128 bits >> for AArch64 and vector lengths must be multiples of quadwords) >> rather than number-of-bits, then an example array (which will >> always be a sequence) might be >> >> [ 8, 16, 32 ] >> >> The user may choose a subsequence, but only through truncation, >> i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. >> >> Furthermore, different hosts may support different sequences >> which have the same maximum. For example, if the above sequence >> is for Host_A, then Host_B could be >> >> [ 8, 16, 24, 32 ] >> >> The host must support all lengths in the sequence, which means >> that while Host_A supports 32, since it doesn't support 24 and >> we can only truncate sequences, we must use either [ 8 ] or >> [ 8, 16 ] for a compatible sequence if we intend to migrate >> between the hosts. >> >> Now to the $SUBJECT question: >> >> My feeling is that we should require the sequence to be >> provided on the command line as a cpu property. Something >> like >> >> -cpu host,sve-vl-list=8:16 >> >> (I chose ':' for the delimiter because ',' can't work, but >> if there's a better choice, then that's fine by me.) >> >> Afaict a property list like this will require a new parser, We had 20+ of those when I last counted. Among the more annoying reasons CLI QAPIfication is hard[1]. >> which feels a bit funny since it seems we should already >> have support for this type of thing somewhere in QEMU. So, >> the question is: do we? I see we have array properties, but >> I don't believe that works with the command line. Should we >> only use QMP for this? We already want some QMP in order to >> query the supported vector lengths. Maybe we should use QMP >> to set the selection too? But then what about command line >> support for developers? And if the property is on the command >> line then we don't have to add it to the migration stream. > > You should be able to use arrays from the CLI with QemuOpts by repeating > the same option name many times, though I can't say it is a very > nice approach if you have many values to list as it gets very repetative. Yes, this is one of the ways the current CLI does lists. It's also one of the more annoying reasons CLI QAPIfication is hard[2]. QemuOpts let the last param=value win the stupidest way that could possibly work (I respect that): add to the front of the list, search it front to back. Then somebody discovered that if you search the list manually, you can see them all, and abuse that to get a list-valued param. I'm sure that felt clever at the time. Another way to do lists the funky list feature of string input and opts visitor. Yet another annoying reason CLI QAPIfication is hard[3]. We use the opts visitor's list feature for -numa node,cpus=... Hmm, looks like we even combine it with the "multiple param=value build up a list" technique: -smp node,cpus=0-1,cpus=4-5 denotes [0,1,4,5]. > That's the curse of not having a good CLI syntax for non-scalar data in > QemuOpts & why Markus believes we should switch to JSON for the CLI too > > -cpu host,sve-vl=8,sve-vl=16 We actually have CLI syntax for non-scalar data: dotted keys. Dotted keys are syntactic sugar for JSON. It looks friendlier than JSON for simple cases, then gets uglier as things get more complex, and then it falls apart: it can't quite express all of JSON. Example: sve-vl.0=8,sve-vl.1=16 gets desugared into {"sve": [8, 16]} if the QAPI schema has 'sve': ['int']. The comment at the beginning of util/keyval.c explains it in more detail. It powers -blockdev and -display. Both options accept either JSON or dotted keys. If the option argument starts with '{', it's JSON. Management applications should stick to JSON. [1] Towards a more expressive and introspectable QEMU command line https://www.linux-kvm.org/images/f/f2/Armbru-qapi-cmdline_1.pdf Slide 34 "Backward compatibility" item 1 [2] ibid, item 4 [3] ibid, item 3 ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? @ 2019-04-18 17:48 ` Markus Armbruster 0 siblings, 0 replies; 45+ messages in thread From: Markus Armbruster @ 2019-04-18 17:48 UTC (permalink / raw) To: Daniel P. Berrangé Cc: peter.maydell, Andrew Jones, dgilbert, qemu-devel, imammedo, Dave.Martin Daniel P. Berrangé <berrange@redhat.com> writes: > On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote: >> Hi all, >> >> First some background: >> >> For the userspace side of AArch64 guest SVE support we need to >> expose KVM's allowed vector lengths bitmap to the user and allow >> the user to choose a subset of that bitmap. Since bitmaps are a >> bit awkward to work with then we'll likely want to expose it as >> an array of vector lengths instead. Also, assuming we want to >> expose the lengths as number-of-quadwords (quadword == 128 bits >> for AArch64 and vector lengths must be multiples of quadwords) >> rather than number-of-bits, then an example array (which will >> always be a sequence) might be >> >> [ 8, 16, 32 ] >> >> The user may choose a subsequence, but only through truncation, >> i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. >> >> Furthermore, different hosts may support different sequences >> which have the same maximum. For example, if the above sequence >> is for Host_A, then Host_B could be >> >> [ 8, 16, 24, 32 ] >> >> The host must support all lengths in the sequence, which means >> that while Host_A supports 32, since it doesn't support 24 and >> we can only truncate sequences, we must use either [ 8 ] or >> [ 8, 16 ] for a compatible sequence if we intend to migrate >> between the hosts. >> >> Now to the $SUBJECT question: >> >> My feeling is that we should require the sequence to be >> provided on the command line as a cpu property. Something >> like >> >> -cpu host,sve-vl-list=8:16 >> >> (I chose ':' for the delimiter because ',' can't work, but >> if there's a better choice, then that's fine by me.) >> >> Afaict a property list like this will require a new parser, We had 20+ of those when I last counted. Among the more annoying reasons CLI QAPIfication is hard[1]. >> which feels a bit funny since it seems we should already >> have support for this type of thing somewhere in QEMU. So, >> the question is: do we? I see we have array properties, but >> I don't believe that works with the command line. Should we >> only use QMP for this? We already want some QMP in order to >> query the supported vector lengths. Maybe we should use QMP >> to set the selection too? But then what about command line >> support for developers? And if the property is on the command >> line then we don't have to add it to the migration stream. > > You should be able to use arrays from the CLI with QemuOpts by repeating > the same option name many times, though I can't say it is a very > nice approach if you have many values to list as it gets very repetative. Yes, this is one of the ways the current CLI does lists. It's also one of the more annoying reasons CLI QAPIfication is hard[2]. QemuOpts let the last param=value win the stupidest way that could possibly work (I respect that): add to the front of the list, search it front to back. Then somebody discovered that if you search the list manually, you can see them all, and abuse that to get a list-valued param. I'm sure that felt clever at the time. Another way to do lists the funky list feature of string input and opts visitor. Yet another annoying reason CLI QAPIfication is hard[3]. We use the opts visitor's list feature for -numa node,cpus=... Hmm, looks like we even combine it with the "multiple param=value build up a list" technique: -smp node,cpus=0-1,cpus=4-5 denotes [0,1,4,5]. > That's the curse of not having a good CLI syntax for non-scalar data in > QemuOpts & why Markus believes we should switch to JSON for the CLI too > > -cpu host,sve-vl=8,sve-vl=16 We actually have CLI syntax for non-scalar data: dotted keys. Dotted keys are syntactic sugar for JSON. It looks friendlier than JSON for simple cases, then gets uglier as things get more complex, and then it falls apart: it can't quite express all of JSON. Example: sve-vl.0=8,sve-vl.1=16 gets desugared into {"sve": [8, 16]} if the QAPI schema has 'sve': ['int']. The comment at the beginning of util/keyval.c explains it in more detail. It powers -blockdev and -display. Both options accept either JSON or dotted keys. If the option argument starts with '{', it's JSON. Management applications should stick to JSON. [1] Towards a more expressive and introspectable QEMU command line https://www.linux-kvm.org/images/f/f2/Armbru-qapi-cmdline_1.pdf Slide 34 "Backward compatibility" item 1 [2] ibid, item 4 [3] ibid, item 3 ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? 2019-04-18 17:48 ` Markus Armbruster (?) @ 2019-05-13 18:42 ` Andrew Jones 2019-05-14 4:54 ` Markus Armbruster -1 siblings, 1 reply; 45+ messages in thread From: Andrew Jones @ 2019-05-13 18:42 UTC (permalink / raw) To: Markus Armbruster Cc: peter.maydell, Daniel P. Berrangé, dgilbert, qemu-devel, imammedo, Dave.Martin On Thu, Apr 18, 2019 at 07:48:09PM +0200, Markus Armbruster wrote: > Daniel P. Berrangé <berrange@redhat.com> writes: > > > On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote: > >> Hi all, > >> > >> First some background: > >> > >> For the userspace side of AArch64 guest SVE support we need to > >> expose KVM's allowed vector lengths bitmap to the user and allow > >> the user to choose a subset of that bitmap. Since bitmaps are a > >> bit awkward to work with then we'll likely want to expose it as > >> an array of vector lengths instead. Also, assuming we want to > >> expose the lengths as number-of-quadwords (quadword == 128 bits > >> for AArch64 and vector lengths must be multiples of quadwords) > >> rather than number-of-bits, then an example array (which will > >> always be a sequence) might be > >> > >> [ 8, 16, 32 ] > >> > >> The user may choose a subsequence, but only through truncation, > >> i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. > >> > >> Furthermore, different hosts may support different sequences > >> which have the same maximum. For example, if the above sequence > >> is for Host_A, then Host_B could be > >> > >> [ 8, 16, 24, 32 ] > >> > >> The host must support all lengths in the sequence, which means > >> that while Host_A supports 32, since it doesn't support 24 and > >> we can only truncate sequences, we must use either [ 8 ] or > >> [ 8, 16 ] for a compatible sequence if we intend to migrate > >> between the hosts. > >> > >> Now to the $SUBJECT question: > >> > >> My feeling is that we should require the sequence to be > >> provided on the command line as a cpu property. Something > >> like > >> > >> -cpu host,sve-vl-list=8:16 > >> > >> (I chose ':' for the delimiter because ',' can't work, but > >> if there's a better choice, then that's fine by me.) > >> > >> Afaict a property list like this will require a new parser, > > We had 20+ of those when I last counted. Among the more annoying > reasons CLI QAPIfication is hard[1]. > > >> which feels a bit funny since it seems we should already > >> have support for this type of thing somewhere in QEMU. So, > >> the question is: do we? I see we have array properties, but > >> I don't believe that works with the command line. Should we > >> only use QMP for this? We already want some QMP in order to > >> query the supported vector lengths. Maybe we should use QMP > >> to set the selection too? But then what about command line > >> support for developers? And if the property is on the command > >> line then we don't have to add it to the migration stream. > > > > You should be able to use arrays from the CLI with QemuOpts by repeating > > the same option name many times, though I can't say it is a very > > nice approach if you have many values to list as it gets very repetative. > > Yes, this is one of the ways the current CLI does lists. It's also one > of the more annoying reasons CLI QAPIfication is hard[2]. > > QemuOpts let the last param=value win the stupidest way that could > possibly work (I respect that): add to the front of the list, search it > front to back. > > Then somebody discovered that if you search the list manually, you can > see them all, and abuse that to get a list-valued param. I'm sure that > felt clever at the time. > > Another way to do lists the funky list feature of string input and opts > visitor. Yet another annoying reason CLI QAPIfication is hard[3]. > > We use the opts visitor's list feature for -numa node,cpus=... Hmm, > looks like we even combine it with the "multiple param=value build up a > list" technique: -smp node,cpus=0-1,cpus=4-5 denotes [0,1,4,5]. > > > That's the curse of not having a good CLI syntax for non-scalar data in > > QemuOpts & why Markus believes we should switch to JSON for the CLI too > > > > -cpu host,sve-vl=8,sve-vl=16 > > We actually have CLI syntax for non-scalar data: dotted keys. Dotted > keys are syntactic sugar for JSON. It looks friendlier than JSON for > simple cases, then gets uglier as things get more complex, and then it > falls apart: it can't quite express all of JSON. > > Example: sve-vl.0=8,sve-vl.1=16 > gets desugared into {"sve": [8, 16]} > if the QAPI schema has 'sve': ['int']. > > The comment at the beginning of util/keyval.c explains it in more > detail. > > It powers -blockdev and -display. Both options accept either JSON or > dotted keys. If the option argument starts with '{', it's JSON. > Management applications should stick to JSON. > > > [1] Towards a more expressive and introspectable QEMU command line > https://www.linux-kvm.org/images/f/f2/Armbru-qapi-cmdline_1.pdf > Slide 34 "Backward compatibility" item 1 > > [2] ibid, item 4 > > [3] ibid, item 3 > Sorry I forgot to follow up to this earlier. I looked at the examples provided and saw they were all for independent command line options, rather than command line options like '-cpu' that then accepts additional properties. I couldn't see how I could use ',' to separate array members when using properties or to use an array property input on the command line. In the end I opted to use a single uint64_t for a bitmap, as 64 is big enough for now, and even though passing some hex number on the command line isn't user friendly at all, it didn't seem like a long list of a repeated parameter was that user friendly either. Of course I'm still open to suggestions to try to find the best balance between user friendliness, current QEMU command line parsing support, and just getting a bitmap into cpu state one way or another. Thanks, drew ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? 2019-05-13 18:42 ` Andrew Jones @ 2019-05-14 4:54 ` Markus Armbruster 2019-05-14 9:02 ` Andrew Jones 2019-05-14 15:28 ` Dave Martin 0 siblings, 2 replies; 45+ messages in thread From: Markus Armbruster @ 2019-05-14 4:54 UTC (permalink / raw) To: Andrew Jones Cc: peter.maydell, Daniel P. Berrangé, dgilbert, qemu-devel, imammedo, Dave.Martin Andrew Jones <drjones@redhat.com> writes: > On Thu, Apr 18, 2019 at 07:48:09PM +0200, Markus Armbruster wrote: >> Daniel P. Berrangé <berrange@redhat.com> writes: >> >> > On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote: >> >> Hi all, >> >> >> >> First some background: >> >> >> >> For the userspace side of AArch64 guest SVE support we need to >> >> expose KVM's allowed vector lengths bitmap to the user and allow >> >> the user to choose a subset of that bitmap. Since bitmaps are a >> >> bit awkward to work with then we'll likely want to expose it as >> >> an array of vector lengths instead. Also, assuming we want to >> >> expose the lengths as number-of-quadwords (quadword == 128 bits >> >> for AArch64 and vector lengths must be multiples of quadwords) >> >> rather than number-of-bits, then an example array (which will >> >> always be a sequence) might be >> >> >> >> [ 8, 16, 32 ] >> >> >> >> The user may choose a subsequence, but only through truncation, >> >> i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. >> >> >> >> Furthermore, different hosts may support different sequences >> >> which have the same maximum. For example, if the above sequence >> >> is for Host_A, then Host_B could be >> >> >> >> [ 8, 16, 24, 32 ] >> >> >> >> The host must support all lengths in the sequence, which means >> >> that while Host_A supports 32, since it doesn't support 24 and >> >> we can only truncate sequences, we must use either [ 8 ] or >> >> [ 8, 16 ] for a compatible sequence if we intend to migrate >> >> between the hosts. >> >> >> >> Now to the $SUBJECT question: >> >> >> >> My feeling is that we should require the sequence to be >> >> provided on the command line as a cpu property. Something >> >> like >> >> >> >> -cpu host,sve-vl-list=8:16 >> >> >> >> (I chose ':' for the delimiter because ',' can't work, but >> >> if there's a better choice, then that's fine by me.) >> >> >> >> Afaict a property list like this will require a new parser, >> >> We had 20+ of those when I last counted. Among the more annoying >> reasons CLI QAPIfication is hard[1]. >> >> >> which feels a bit funny since it seems we should already >> >> have support for this type of thing somewhere in QEMU. So, >> >> the question is: do we? I see we have array properties, but >> >> I don't believe that works with the command line. Should we >> >> only use QMP for this? We already want some QMP in order to >> >> query the supported vector lengths. Maybe we should use QMP >> >> to set the selection too? But then what about command line >> >> support for developers? And if the property is on the command >> >> line then we don't have to add it to the migration stream. >> > >> > You should be able to use arrays from the CLI with QemuOpts by repeating >> > the same option name many times, though I can't say it is a very >> > nice approach if you have many values to list as it gets very repetative. >> >> Yes, this is one of the ways the current CLI does lists. It's also one >> of the more annoying reasons CLI QAPIfication is hard[2]. >> >> QemuOpts let the last param=value win the stupidest way that could >> possibly work (I respect that): add to the front of the list, search it >> front to back. >> >> Then somebody discovered that if you search the list manually, you can >> see them all, and abuse that to get a list-valued param. I'm sure that >> felt clever at the time. >> >> Another way to do lists the funky list feature of string input and opts >> visitor. Yet another annoying reason CLI QAPIfication is hard[3]. >> >> We use the opts visitor's list feature for -numa node,cpus=... Hmm, >> looks like we even combine it with the "multiple param=value build up a >> list" technique: -smp node,cpus=0-1,cpus=4-5 denotes [0,1,4,5]. >> >> > That's the curse of not having a good CLI syntax for non-scalar data in >> > QemuOpts & why Markus believes we should switch to JSON for the CLI too >> > >> > -cpu host,sve-vl=8,sve-vl=16 >> >> We actually have CLI syntax for non-scalar data: dotted keys. Dotted >> keys are syntactic sugar for JSON. It looks friendlier than JSON for >> simple cases, then gets uglier as things get more complex, and then it >> falls apart: it can't quite express all of JSON. >> >> Example: sve-vl.0=8,sve-vl.1=16 >> gets desugared into {"sve": [8, 16]} >> if the QAPI schema has 'sve': ['int']. >> >> The comment at the beginning of util/keyval.c explains it in more >> detail. >> >> It powers -blockdev and -display. Both options accept either JSON or >> dotted keys. If the option argument starts with '{', it's JSON. >> Management applications should stick to JSON. >> >> >> [1] Towards a more expressive and introspectable QEMU command line >> https://www.linux-kvm.org/images/f/f2/Armbru-qapi-cmdline_1.pdf >> Slide 34 "Backward compatibility" item 1 >> >> [2] ibid, item 4 >> >> [3] ibid, item 3 >> > > Sorry I forgot to follow up to this earlier. I looked at the examples > provided and saw they were all for independent command line options, > rather than command line options like '-cpu' that then accepts additional > properties. I couldn't see how I could use ',' to separate array members > when using properties or to use an array property input on the command > line. The argument of -cpu is parsed ad hoc. Unlike QemuOpts and dotted keys, parse_cpu_option() doesn't seem to support escaping ','. Not that escaping would be a user-friendly solution. > In the end I opted to use a single uint64_t for a bitmap, as 64 is > big enough for now, Do you think it'll remain big enough? > and even though passing some hex number on the command > line isn't user friendly at all, it didn't seem like a long list of a > repeated parameter was that user friendly either. Of course I'm still open > to suggestions to try to find the best balance between user friendliness, > current QEMU command line parsing support, and just getting a bitmap into > cpu state one way or another. I'd ask for consistency with existing practice no matter how flawed if we had such consistency. If I understand your "[PATCH 00/13] target/arm/kvm: enable SVE in guests" correctly, the bitmap form of [1, 2, 4] is -cpu max,sve-vls-map=11 Observe bit#0 means 1; better document that clearly. If we used dotted keys to produce an intList, we'd do -cpu max,sve-vls-map.0=1,sve-vls-map.1=2,sve-vls-map.2=4 If the option argument is QAPIfied, we additionally get -cpu '{"type": "max", "sve-vls-map": [1, 2, 4]}' for free. If we did it like -numa (please don't), we'd get something like -cpu max,sve-vls-map=1-2,sve-vls-map=4 None of the above is exactly a pinnacle of user-friendliness. JSON is at least ugly in a regular way. Your bitmap encoded in a number is at least concise. If a numerically encoded bitmap is the least bad option here, I wonder why it's not the least bad option for -numa... Perhaps because there 64 isn't big enough. I'm afraid the numerically encoded bitmap will make its way into the QAPI schema sooner or later. This will create an unfortunate inconsistency with the [int] encoding already there. Who's going to use sve-vls-map? Humans, or pretty much only machines? ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? 2019-05-14 4:54 ` Markus Armbruster @ 2019-05-14 9:02 ` Andrew Jones 2019-05-14 13:32 ` Markus Armbruster 2019-05-14 14:48 ` Igor Mammedov 2019-05-14 15:28 ` Dave Martin 1 sibling, 2 replies; 45+ messages in thread From: Andrew Jones @ 2019-05-14 9:02 UTC (permalink / raw) To: Markus Armbruster Cc: peter.maydell, Daniel P. Berrangé, dgilbert, qemu-devel, imammedo, Dave.Martin On Tue, May 14, 2019 at 06:54:03AM +0200, Markus Armbruster wrote: > Andrew Jones <drjones@redhat.com> writes: > > > On Thu, Apr 18, 2019 at 07:48:09PM +0200, Markus Armbruster wrote: > >> Daniel P. Berrangé <berrange@redhat.com> writes: > >> > >> > On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote: > >> >> Hi all, > >> >> > >> >> First some background: > >> >> > >> >> For the userspace side of AArch64 guest SVE support we need to > >> >> expose KVM's allowed vector lengths bitmap to the user and allow > >> >> the user to choose a subset of that bitmap. Since bitmaps are a > >> >> bit awkward to work with then we'll likely want to expose it as > >> >> an array of vector lengths instead. Also, assuming we want to > >> >> expose the lengths as number-of-quadwords (quadword == 128 bits > >> >> for AArch64 and vector lengths must be multiples of quadwords) > >> >> rather than number-of-bits, then an example array (which will > >> >> always be a sequence) might be > >> >> > >> >> [ 8, 16, 32 ] > >> >> > >> >> The user may choose a subsequence, but only through truncation, > >> >> i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. > >> >> > >> >> Furthermore, different hosts may support different sequences > >> >> which have the same maximum. For example, if the above sequence > >> >> is for Host_A, then Host_B could be > >> >> > >> >> [ 8, 16, 24, 32 ] > >> >> > >> >> The host must support all lengths in the sequence, which means > >> >> that while Host_A supports 32, since it doesn't support 24 and > >> >> we can only truncate sequences, we must use either [ 8 ] or > >> >> [ 8, 16 ] for a compatible sequence if we intend to migrate > >> >> between the hosts. > >> >> > >> >> Now to the $SUBJECT question: > >> >> > >> >> My feeling is that we should require the sequence to be > >> >> provided on the command line as a cpu property. Something > >> >> like > >> >> > >> >> -cpu host,sve-vl-list=8:16 > >> >> > >> >> (I chose ':' for the delimiter because ',' can't work, but > >> >> if there's a better choice, then that's fine by me.) > >> >> > >> >> Afaict a property list like this will require a new parser, > >> > >> We had 20+ of those when I last counted. Among the more annoying > >> reasons CLI QAPIfication is hard[1]. > >> > >> >> which feels a bit funny since it seems we should already > >> >> have support for this type of thing somewhere in QEMU. So, > >> >> the question is: do we? I see we have array properties, but > >> >> I don't believe that works with the command line. Should we > >> >> only use QMP for this? We already want some QMP in order to > >> >> query the supported vector lengths. Maybe we should use QMP > >> >> to set the selection too? But then what about command line > >> >> support for developers? And if the property is on the command > >> >> line then we don't have to add it to the migration stream. > >> > > >> > You should be able to use arrays from the CLI with QemuOpts by repeating > >> > the same option name many times, though I can't say it is a very > >> > nice approach if you have many values to list as it gets very repetative. > >> > >> Yes, this is one of the ways the current CLI does lists. It's also one > >> of the more annoying reasons CLI QAPIfication is hard[2]. > >> > >> QemuOpts let the last param=value win the stupidest way that could > >> possibly work (I respect that): add to the front of the list, search it > >> front to back. > >> > >> Then somebody discovered that if you search the list manually, you can > >> see them all, and abuse that to get a list-valued param. I'm sure that > >> felt clever at the time. > >> > >> Another way to do lists the funky list feature of string input and opts > >> visitor. Yet another annoying reason CLI QAPIfication is hard[3]. > >> > >> We use the opts visitor's list feature for -numa node,cpus=... Hmm, > >> looks like we even combine it with the "multiple param=value build up a > >> list" technique: -smp node,cpus=0-1,cpus=4-5 denotes [0,1,4,5]. > >> > >> > That's the curse of not having a good CLI syntax for non-scalar data in > >> > QemuOpts & why Markus believes we should switch to JSON for the CLI too > >> > > >> > -cpu host,sve-vl=8,sve-vl=16 > >> > >> We actually have CLI syntax for non-scalar data: dotted keys. Dotted > >> keys are syntactic sugar for JSON. It looks friendlier than JSON for > >> simple cases, then gets uglier as things get more complex, and then it > >> falls apart: it can't quite express all of JSON. > >> > >> Example: sve-vl.0=8,sve-vl.1=16 > >> gets desugared into {"sve": [8, 16]} > >> if the QAPI schema has 'sve': ['int']. > >> > >> The comment at the beginning of util/keyval.c explains it in more > >> detail. > >> > >> It powers -blockdev and -display. Both options accept either JSON or > >> dotted keys. If the option argument starts with '{', it's JSON. > >> Management applications should stick to JSON. > >> > >> > >> [1] Towards a more expressive and introspectable QEMU command line > >> https://www.linux-kvm.org/images/f/f2/Armbru-qapi-cmdline_1.pdf > >> Slide 34 "Backward compatibility" item 1 > >> > >> [2] ibid, item 4 > >> > >> [3] ibid, item 3 > >> > > > > Sorry I forgot to follow up to this earlier. I looked at the examples > > provided and saw they were all for independent command line options, > > rather than command line options like '-cpu' that then accepts additional > > properties. I couldn't see how I could use ',' to separate array members > > when using properties or to use an array property input on the command > > line. > > The argument of -cpu is parsed ad hoc. Unlike QemuOpts and dotted keys, > parse_cpu_option() doesn't seem to support escaping ','. Not that > escaping would be a user-friendly solution. > > > In the end I opted to use a single uint64_t for a bitmap, as 64 is > > big enough for now, > > Do you think it'll remain big enough? Probably not forever, and TBH I can't even give an estimate for how long. Based on the current state, I "feel" like it'll be quite some time though. I think we can extend this map by adding more ad hoc parsing to -cpu later. If we added dotted key support then each array member could be another bitmap word, for example. > > > and even though passing some hex number on the command > > line isn't user friendly at all, it didn't seem like a long list of a > > repeated parameter was that user friendly either. Of course I'm still open > > to suggestions to try to find the best balance between user friendliness, > > current QEMU command line parsing support, and just getting a bitmap into > > cpu state one way or another. > > I'd ask for consistency with existing practice no matter how flawed if > we had such consistency. > > If I understand your "[PATCH 00/13] target/arm/kvm: enable SVE in > guests" correctly, the bitmap form of [1, 2, 4] is > > -cpu max,sve-vls-map=11 > > Observe bit#0 means 1; better document that clearly. > > If we used dotted keys to produce an intList, we'd do > > -cpu max,sve-vls-map.0=1,sve-vls-map.1=2,sve-vls-map.2=4 > > If the option argument is QAPIfied, we additionally get > > -cpu '{"type": "max", "sve-vls-map": [1, 2, 4]}' > > for free. > > If we did it like -numa (please don't), we'd get something like > > -cpu max,sve-vls-map=1-2,sve-vls-map=4 > > None of the above is exactly a pinnacle of user-friendliness. JSON is > at least ugly in a regular way. Your bitmap encoded in a number is at > least concise. > > If a numerically encoded bitmap is the least bad option here, I wonder > why it's not the least bad option for -numa... Perhaps because there 64 > isn't big enough. > > I'm afraid the numerically encoded bitmap will make its way into the > QAPI schema sooner or later. This will create an unfortunate > inconsistency with the [int] encoding already there. > > Who's going to use sve-vls-map? Humans, or pretty much only machines? My thought is primarily machines. If a human wants to use the command line and SVE, then I'm assuming they'll be happy with sve-max-vq or figuring out a map they like once and then sticking to it. Thanks, drew ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? 2019-05-14 9:02 ` Andrew Jones @ 2019-05-14 13:32 ` Markus Armbruster 2019-05-15 8:15 ` Andrew Jones 2019-05-14 14:48 ` Igor Mammedov 1 sibling, 1 reply; 45+ messages in thread From: Markus Armbruster @ 2019-05-14 13:32 UTC (permalink / raw) To: Andrew Jones Cc: peter.maydell, Daniel P. Berrangé, dgilbert, qemu-devel, imammedo, Dave.Martin Andrew Jones <drjones@redhat.com> writes: > On Tue, May 14, 2019 at 06:54:03AM +0200, Markus Armbruster wrote: >> Andrew Jones <drjones@redhat.com> writes: >> >> > On Thu, Apr 18, 2019 at 07:48:09PM +0200, Markus Armbruster wrote: >> >> Daniel P. Berrangé <berrange@redhat.com> writes: >> >> >> >> > On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote: >> >> >> Hi all, >> >> >> >> >> >> First some background: >> >> >> >> >> >> For the userspace side of AArch64 guest SVE support we need to >> >> >> expose KVM's allowed vector lengths bitmap to the user and allow >> >> >> the user to choose a subset of that bitmap. Since bitmaps are a >> >> >> bit awkward to work with then we'll likely want to expose it as >> >> >> an array of vector lengths instead. Also, assuming we want to >> >> >> expose the lengths as number-of-quadwords (quadword == 128 bits >> >> >> for AArch64 and vector lengths must be multiples of quadwords) >> >> >> rather than number-of-bits, then an example array (which will >> >> >> always be a sequence) might be >> >> >> >> >> >> [ 8, 16, 32 ] >> >> >> >> >> >> The user may choose a subsequence, but only through truncation, >> >> >> i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. >> >> >> >> >> >> Furthermore, different hosts may support different sequences >> >> >> which have the same maximum. For example, if the above sequence >> >> >> is for Host_A, then Host_B could be >> >> >> >> >> >> [ 8, 16, 24, 32 ] >> >> >> >> >> >> The host must support all lengths in the sequence, which means >> >> >> that while Host_A supports 32, since it doesn't support 24 and >> >> >> we can only truncate sequences, we must use either [ 8 ] or >> >> >> [ 8, 16 ] for a compatible sequence if we intend to migrate >> >> >> between the hosts. >> >> >> >> >> >> Now to the $SUBJECT question: >> >> >> >> >> >> My feeling is that we should require the sequence to be >> >> >> provided on the command line as a cpu property. Something >> >> >> like >> >> >> >> >> >> -cpu host,sve-vl-list=8:16 >> >> >> >> >> >> (I chose ':' for the delimiter because ',' can't work, but >> >> >> if there's a better choice, then that's fine by me.) >> >> >> >> >> >> Afaict a property list like this will require a new parser, >> >> >> >> We had 20+ of those when I last counted. Among the more annoying >> >> reasons CLI QAPIfication is hard[1]. >> >> >> >> >> which feels a bit funny since it seems we should already >> >> >> have support for this type of thing somewhere in QEMU. So, >> >> >> the question is: do we? I see we have array properties, but >> >> >> I don't believe that works with the command line. Should we >> >> >> only use QMP for this? We already want some QMP in order to >> >> >> query the supported vector lengths. Maybe we should use QMP >> >> >> to set the selection too? But then what about command line >> >> >> support for developers? And if the property is on the command >> >> >> line then we don't have to add it to the migration stream. >> >> > >> >> > You should be able to use arrays from the CLI with QemuOpts by repeating >> >> > the same option name many times, though I can't say it is a very >> >> > nice approach if you have many values to list as it gets very repetative. >> >> >> >> Yes, this is one of the ways the current CLI does lists. It's also one >> >> of the more annoying reasons CLI QAPIfication is hard[2]. >> >> >> >> QemuOpts let the last param=value win the stupidest way that could >> >> possibly work (I respect that): add to the front of the list, search it >> >> front to back. >> >> >> >> Then somebody discovered that if you search the list manually, you can >> >> see them all, and abuse that to get a list-valued param. I'm sure that >> >> felt clever at the time. >> >> >> >> Another way to do lists the funky list feature of string input and opts >> >> visitor. Yet another annoying reason CLI QAPIfication is hard[3]. >> >> >> >> We use the opts visitor's list feature for -numa node,cpus=... Hmm, >> >> looks like we even combine it with the "multiple param=value build up a >> >> list" technique: -smp node,cpus=0-1,cpus=4-5 denotes [0,1,4,5]. >> >> >> >> > That's the curse of not having a good CLI syntax for non-scalar data in >> >> > QemuOpts & why Markus believes we should switch to JSON for the CLI too >> >> > >> >> > -cpu host,sve-vl=8,sve-vl=16 >> >> >> >> We actually have CLI syntax for non-scalar data: dotted keys. Dotted >> >> keys are syntactic sugar for JSON. It looks friendlier than JSON for >> >> simple cases, then gets uglier as things get more complex, and then it >> >> falls apart: it can't quite express all of JSON. >> >> >> >> Example: sve-vl.0=8,sve-vl.1=16 >> >> gets desugared into {"sve": [8, 16]} >> >> if the QAPI schema has 'sve': ['int']. >> >> >> >> The comment at the beginning of util/keyval.c explains it in more >> >> detail. >> >> >> >> It powers -blockdev and -display. Both options accept either JSON or >> >> dotted keys. If the option argument starts with '{', it's JSON. >> >> Management applications should stick to JSON. >> >> >> >> >> >> [1] Towards a more expressive and introspectable QEMU command line >> >> https://www.linux-kvm.org/images/f/f2/Armbru-qapi-cmdline_1.pdf >> >> Slide 34 "Backward compatibility" item 1 >> >> >> >> [2] ibid, item 4 >> >> >> >> [3] ibid, item 3 >> >> >> > >> > Sorry I forgot to follow up to this earlier. I looked at the examples >> > provided and saw they were all for independent command line options, >> > rather than command line options like '-cpu' that then accepts additional >> > properties. I couldn't see how I could use ',' to separate array members >> > when using properties or to use an array property input on the command >> > line. >> >> The argument of -cpu is parsed ad hoc. Unlike QemuOpts and dotted keys, >> parse_cpu_option() doesn't seem to support escaping ','. Not that >> escaping would be a user-friendly solution. >> >> > In the end I opted to use a single uint64_t for a bitmap, as 64 is >> > big enough for now, >> >> Do you think it'll remain big enough? > > Probably not forever, and TBH I can't even give an estimate for how long. > Based on the current state, I "feel" like it'll be quite some time though. > I think we can extend this map by adding more ad hoc parsing to -cpu > later. If we added dotted key support then each array member could be > another bitmap word, for example. Syntax that can support such growth would be nice. To grow a single unsigned number, we can make it wider (but we don't have infrastructure for numbers wider than 64 bits), or we can add more numbers (but under what name?). Dotted keys syntax could grow more easily, but it's rather awkward. Looking more closely at your "[PATCH 11/13] target/arm/cpu64: max cpu: Introduce sve-vls-map"... your syntax reflects your data structure: property "sve-vls-map" is of type uint64_t, and interpreted as bit set. This data type would have to grow, too. We could make widen the integer property (but we don't have infrastructure for integer properties wider than 64 bits), or we can turn it into an array of integers (compatibility?), or we can add more properties to hold the additional integers (yet another silly way to represent a list/array of integers). I'm not asking you to complicate things just to future-proof this. Just pause and think whether you can pick a data type that's similarly convenient now, and easier to grow. Then pick an external syntax for this data type. You may have to pick a reasonable compromise between ease of implementation and ease of use. >> > and even though passing some hex number on the command >> > line isn't user friendly at all, it didn't seem like a long list of a >> > repeated parameter was that user friendly either. Of course I'm still open >> > to suggestions to try to find the best balance between user friendliness, >> > current QEMU command line parsing support, and just getting a bitmap into >> > cpu state one way or another. >> >> I'd ask for consistency with existing practice no matter how flawed if >> we had such consistency. >> >> If I understand your "[PATCH 00/13] target/arm/kvm: enable SVE in >> guests" correctly, the bitmap form of [1, 2, 4] is >> >> -cpu max,sve-vls-map=11 >> >> Observe bit#0 means 1; better document that clearly. >> >> If we used dotted keys to produce an intList, we'd do >> >> -cpu max,sve-vls-map.0=1,sve-vls-map.1=2,sve-vls-map.2=4 >> >> If the option argument is QAPIfied, we additionally get >> >> -cpu '{"type": "max", "sve-vls-map": [1, 2, 4]}' >> >> for free. >> >> If we did it like -numa (please don't), we'd get something like >> >> -cpu max,sve-vls-map=1-2,sve-vls-map=4 >> >> None of the above is exactly a pinnacle of user-friendliness. JSON is >> at least ugly in a regular way. Your bitmap encoded in a number is at >> least concise. >> >> If a numerically encoded bitmap is the least bad option here, I wonder >> why it's not the least bad option for -numa... Perhaps because there 64 >> isn't big enough. >> >> I'm afraid the numerically encoded bitmap will make its way into the >> QAPI schema sooner or later. This will create an unfortunate >> inconsistency with the [int] encoding already there. >> >> Who's going to use sve-vls-map? Humans, or pretty much only machines? > > My thought is primarily machines. If a human wants to use the command > line and SVE, then I'm assuming they'll be happy with sve-max-vq or > figuring out a map they like once and then sticking to it. Primarily machines means we can accept more verbosity. If I understand the cover letter of your "[PATCH 00/13] target/arm/kvm: enable SVE in guests" correctly, then sve-max-vq and sve-vls-map are alternative interfaces for the same thing. The latter is more general, but awkward on the command line and verbose everywhere. The former isn't usable with -cpu host. Correct? If there wasn't "not usable with -cpu host", I'd ask whether we really need the generality. ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? 2019-05-14 13:32 ` Markus Armbruster @ 2019-05-15 8:15 ` Andrew Jones 2019-05-15 10:53 ` Dave Martin 0 siblings, 1 reply; 45+ messages in thread From: Andrew Jones @ 2019-05-15 8:15 UTC (permalink / raw) To: Markus Armbruster Cc: peter.maydell, Daniel P. Berrangé, dgilbert, qemu-devel, imammedo, Dave.Martin On Tue, May 14, 2019 at 03:32:13PM +0200, Markus Armbruster wrote: > Syntax that can support such growth would be nice. > > To grow a single unsigned number, we can make it wider (but we don't > have infrastructure for numbers wider than 64 bits), or we can add more > numbers (but under what name?). > > Dotted keys syntax could grow more easily, but it's rather awkward. > > Looking more closely at your "[PATCH 11/13] target/arm/cpu64: max cpu: > Introduce sve-vls-map"... your syntax reflects your data structure: > property "sve-vls-map" is of type uint64_t, and interpreted as bit set. > This data type would have to grow, too. > > We could make widen the integer property (but we don't have > infrastructure for integer properties wider than 64 bits), or we can > turn it into an array of integers (compatibility?), or we can add more > properties to hold the additional integers (yet another silly way to > represent a list/array of integers). > > I'm not asking you to complicate things just to future-proof this. Just > pause and think whether you can pick a data type that's similarly > convenient now, and easier to grow. > > Then pick an external syntax for this data type. You may have to pick a > reasonable compromise between ease of implementation and ease of use. Widening the integer property sounds good to me. I just hadn't thought of it (implementation tunnel vision affecting my user interface design). Andrea also mentioned that as a possibility in a reply to the series. I think we can leave the property as a uint64_t right now and then, when/if it needs to expand past 64 bits we can change the property to a string and start parsing arbitrarily large integers from it. The internal state, 'uint64_t sve_vls_map' can easily be changed to a 'uint64_t sve_vls_map[]' at that point too. How's that sound? > Primarily machines means we can accept more verbosity. > > If I understand the cover letter of your "[PATCH 00/13] target/arm/kvm: > enable SVE in guests" correctly, then sve-max-vq and sve-vls-map are > alternative interfaces for the same thing. The latter is more general, > but awkward on the command line and verbose everywhere. The former > isn't usable with -cpu host. Correct? > > If there wasn't "not usable with -cpu host", I'd ask whether we really > need the generality. > The 'not usable with -cpu host' was a choice, not a limitation to deal with. The problem with sve-max-vq is that it doesn't provide enough information by itself. Multiple vector sets with the same max-vq can be valid. For migration of KVM guests the exact set needs to be explicitly requested to ensure the host supports it. For TCG it may be nice to model sets besides the everything up to max-vq set that sve-max-vq would choose. sve-max-vq could be used with -cpu host the same way I've implemented it for -cpu max, which is to take the fully supported vector set from the KVM query and use that one, but I didn't want to encourage its use with AArch64's primary KVM cpu type (even if it's -cpu host and thus not a great choice for migration anyway). Thanks, drew ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? 2019-05-15 8:15 ` Andrew Jones @ 2019-05-15 10:53 ` Dave Martin 2019-05-15 10:59 ` Dr. David Alan Gilbert 0 siblings, 1 reply; 45+ messages in thread From: Dave Martin @ 2019-05-15 10:53 UTC (permalink / raw) To: Andrew Jones Cc: peter.maydell, Daniel P. Berrangé, qemu-devel, Markus Armbruster, imammedo, dgilbert On Wed, May 15, 2019 at 09:15:20AM +0100, Andrew Jones wrote: > On Tue, May 14, 2019 at 03:32:13PM +0200, Markus Armbruster wrote: > > Syntax that can support such growth would be nice. > > > > To grow a single unsigned number, we can make it wider (but we don't > > have infrastructure for numbers wider than 64 bits), or we can add more > > numbers (but under what name?). > > > > Dotted keys syntax could grow more easily, but it's rather awkward. > > > > Looking more closely at your "[PATCH 11/13] target/arm/cpu64: max cpu: > > Introduce sve-vls-map"... your syntax reflects your data structure: > > property "sve-vls-map" is of type uint64_t, and interpreted as bit set. > > This data type would have to grow, too. > > > > We could make widen the integer property (but we don't have > > infrastructure for integer properties wider than 64 bits), or we can > > turn it into an array of integers (compatibility?), or we can add more > > properties to hold the additional integers (yet another silly way to > > represent a list/array of integers). > > > > I'm not asking you to complicate things just to future-proof this. Just > > pause and think whether you can pick a data type that's similarly > > convenient now, and easier to grow. > > > > Then pick an external syntax for this data type. You may have to pick a > > reasonable compromise between ease of implementation and ease of use. > > Widening the integer property sounds good to me. I just hadn't thought of > it (implementation tunnel vision affecting my user interface design). > Andrea also mentioned that as a possibility in a reply to the series. I > think we can leave the property as a uint64_t right now and then, when/if > it needs to expand past 64 bits we can change the property to a string > and start parsing arbitrarily large integers from it. The internal state, > 'uint64_t sve_vls_map' can easily be changed to a 'uint64_t sve_vls_map[]' > at that point too. How's that sound? Having an arbitrary-width integer should work. It will suck a bit for the common case of sparse vector length support 0x8000000000000000800000008000808b (= [ 1, 2, 4, 8, 16, 32, 64, 128 ] quadwords) Since lengths above 16 quadwords remain theoretical for now though it's probably OK as a compromise, though. The most human-compatible approach would be some kind of list comprehension syntax, but it's hard to justify that adding a whole new syntax is justified at this point. [...] Cheers ---Dave ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? 2019-05-15 10:53 ` Dave Martin @ 2019-05-15 10:59 ` Dr. David Alan Gilbert 0 siblings, 0 replies; 45+ messages in thread From: Dr. David Alan Gilbert @ 2019-05-15 10:59 UTC (permalink / raw) To: Dave Martin Cc: peter.maydell, Andrew Jones, Daniel P. Berrangé, Markus Armbruster, qemu-devel, imammedo * Dave Martin (Dave.Martin@arm.com) wrote: > On Wed, May 15, 2019 at 09:15:20AM +0100, Andrew Jones wrote: > > On Tue, May 14, 2019 at 03:32:13PM +0200, Markus Armbruster wrote: > > > Syntax that can support such growth would be nice. > > > > > > To grow a single unsigned number, we can make it wider (but we don't > > > have infrastructure for numbers wider than 64 bits), or we can add more > > > numbers (but under what name?). > > > > > > Dotted keys syntax could grow more easily, but it's rather awkward. > > > > > > Looking more closely at your "[PATCH 11/13] target/arm/cpu64: max cpu: > > > Introduce sve-vls-map"... your syntax reflects your data structure: > > > property "sve-vls-map" is of type uint64_t, and interpreted as bit set. > > > This data type would have to grow, too. > > > > > > We could make widen the integer property (but we don't have > > > infrastructure for integer properties wider than 64 bits), or we can > > > turn it into an array of integers (compatibility?), or we can add more > > > properties to hold the additional integers (yet another silly way to > > > represent a list/array of integers). > > > > > > I'm not asking you to complicate things just to future-proof this. Just > > > pause and think whether you can pick a data type that's similarly > > > convenient now, and easier to grow. > > > > > > Then pick an external syntax for this data type. You may have to pick a > > > reasonable compromise between ease of implementation and ease of use. > > > > Widening the integer property sounds good to me. I just hadn't thought of > > it (implementation tunnel vision affecting my user interface design). > > Andrea also mentioned that as a possibility in a reply to the series. I > > think we can leave the property as a uint64_t right now and then, when/if > > it needs to expand past 64 bits we can change the property to a string > > and start parsing arbitrarily large integers from it. The internal state, > > 'uint64_t sve_vls_map' can easily be changed to a 'uint64_t sve_vls_map[]' > > at that point too. How's that sound? > > Having an arbitrary-width integer should work. > > It will suck a bit for the common case of sparse vector length support > > 0x8000000000000000800000008000808b > > (= [ 1, 2, 4, 8, 16, 32, 64, 128 ] quadwords) You could allow _'s as a readability feature just to allow you to see boundaries. 0x8000_0000_0000_0000_8000_0000_8000_808b (Some might suggest IPv6 address syntax but I could never read that). Dave -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? 2019-05-14 9:02 ` Andrew Jones 2019-05-14 13:32 ` Markus Armbruster @ 2019-05-14 14:48 ` Igor Mammedov 2019-05-15 8:18 ` Andrew Jones 1 sibling, 1 reply; 45+ messages in thread From: Igor Mammedov @ 2019-05-14 14:48 UTC (permalink / raw) To: Andrew Jones Cc: peter.maydell, Daniel P. Berrangé, qemu-devel, Markus Armbruster, Dave.Martin, dgilbert On Tue, 14 May 2019 11:02:25 +0200 Andrew Jones <drjones@redhat.com> wrote: > On Tue, May 14, 2019 at 06:54:03AM +0200, Markus Armbruster wrote: > > Andrew Jones <drjones@redhat.com> writes: > > > > > On Thu, Apr 18, 2019 at 07:48:09PM +0200, Markus Armbruster wrote: > > >> Daniel P. Berrangé <berrange@redhat.com> writes: > > >> > > >> > On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote: > > >> >> Hi all, > > >> >> > > >> >> First some background: > > >> >> > > >> >> For the userspace side of AArch64 guest SVE support we need to > > >> >> expose KVM's allowed vector lengths bitmap to the user and allow > > >> >> the user to choose a subset of that bitmap. Since bitmaps are a > > >> >> bit awkward to work with then we'll likely want to expose it as > > >> >> an array of vector lengths instead. Also, assuming we want to > > >> >> expose the lengths as number-of-quadwords (quadword == 128 bits > > >> >> for AArch64 and vector lengths must be multiples of quadwords) > > >> >> rather than number-of-bits, then an example array (which will > > >> >> always be a sequence) might be > > >> >> > > >> >> [ 8, 16, 32 ] > > >> >> > > >> >> The user may choose a subsequence, but only through truncation, > > >> >> i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. > > >> >> > > >> >> Furthermore, different hosts may support different sequences > > >> >> which have the same maximum. For example, if the above sequence > > >> >> is for Host_A, then Host_B could be > > >> >> > > >> >> [ 8, 16, 24, 32 ] > > >> >> > > >> >> The host must support all lengths in the sequence, which means > > >> >> that while Host_A supports 32, since it doesn't support 24 and > > >> >> we can only truncate sequences, we must use either [ 8 ] or > > >> >> [ 8, 16 ] for a compatible sequence if we intend to migrate > > >> >> between the hosts. > > >> >> > > >> >> Now to the $SUBJECT question: > > >> >> > > >> >> My feeling is that we should require the sequence to be > > >> >> provided on the command line as a cpu property. Something > > >> >> like > > >> >> > > >> >> -cpu host,sve-vl-list=8:16 > > >> >> > > >> >> (I chose ':' for the delimiter because ',' can't work, but > > >> >> if there's a better choice, then that's fine by me.) > > >> >> > > >> >> Afaict a property list like this will require a new parser, > > >> > > >> We had 20+ of those when I last counted. Among the more annoying > > >> reasons CLI QAPIfication is hard[1]. > > >> > > >> >> which feels a bit funny since it seems we should already > > >> >> have support for this type of thing somewhere in QEMU. So, > > >> >> the question is: do we? I see we have array properties, but > > >> >> I don't believe that works with the command line. Should we > > >> >> only use QMP for this? We already want some QMP in order to > > >> >> query the supported vector lengths. Maybe we should use QMP > > >> >> to set the selection too? But then what about command line > > >> >> support for developers? And if the property is on the command > > >> >> line then we don't have to add it to the migration stream. > > >> > > > >> > You should be able to use arrays from the CLI with QemuOpts by repeating > > >> > the same option name many times, though I can't say it is a very > > >> > nice approach if you have many values to list as it gets very repetative. > > >> > > >> Yes, this is one of the ways the current CLI does lists. It's also one > > >> of the more annoying reasons CLI QAPIfication is hard[2]. > > >> > > >> QemuOpts let the last param=value win the stupidest way that could > > >> possibly work (I respect that): add to the front of the list, search it > > >> front to back. > > >> > > >> Then somebody discovered that if you search the list manually, you can > > >> see them all, and abuse that to get a list-valued param. I'm sure that > > >> felt clever at the time. > > >> > > >> Another way to do lists the funky list feature of string input and opts > > >> visitor. Yet another annoying reason CLI QAPIfication is hard[3]. > > >> > > >> We use the opts visitor's list feature for -numa node,cpus=... Hmm, > > >> looks like we even combine it with the "multiple param=value build up a > > >> list" technique: -smp node,cpus=0-1,cpus=4-5 denotes [0,1,4,5]. > > >> > > >> > That's the curse of not having a good CLI syntax for non-scalar data in > > >> > QemuOpts & why Markus believes we should switch to JSON for the CLI too > > >> > > > >> > -cpu host,sve-vl=8,sve-vl=16 > > >> > > >> We actually have CLI syntax for non-scalar data: dotted keys. Dotted > > >> keys are syntactic sugar for JSON. It looks friendlier than JSON for > > >> simple cases, then gets uglier as things get more complex, and then it > > >> falls apart: it can't quite express all of JSON. > > >> > > >> Example: sve-vl.0=8,sve-vl.1=16 > > >> gets desugared into {"sve": [8, 16]} > > >> if the QAPI schema has 'sve': ['int']. > > >> > > >> The comment at the beginning of util/keyval.c explains it in more > > >> detail. > > >> > > >> It powers -blockdev and -display. Both options accept either JSON or > > >> dotted keys. If the option argument starts with '{', it's JSON. > > >> Management applications should stick to JSON. > > >> > > >> > > >> [1] Towards a more expressive and introspectable QEMU command line > > >> https://www.linux-kvm.org/images/f/f2/Armbru-qapi-cmdline_1.pdf > > >> Slide 34 "Backward compatibility" item 1 > > >> > > >> [2] ibid, item 4 > > >> > > >> [3] ibid, item 3 > > >> > > > > > > Sorry I forgot to follow up to this earlier. I looked at the examples > > > provided and saw they were all for independent command line options, > > > rather than command line options like '-cpu' that then accepts additional > > > properties. I couldn't see how I could use ',' to separate array members > > > when using properties or to use an array property input on the command > > > line. > > > > The argument of -cpu is parsed ad hoc. Unlike QemuOpts and dotted keys, > > parse_cpu_option() doesn't seem to support escaping ','. Not that > > escaping would be a user-friendly solution. > > > > > In the end I opted to use a single uint64_t for a bitmap, as 64 is > > > big enough for now, > > > > Do you think it'll remain big enough? > > Probably not forever, and TBH I can't even give an estimate for how long. > Based on the current state, I "feel" like it'll be quite some time though. > I think we can extend this map by adding more ad hoc parsing to -cpu > later. If we added dotted key support then each array member could be > another bitmap word, for example. > > > > > > and even though passing some hex number on the command > > > line isn't user friendly at all, it didn't seem like a long list of a > > > repeated parameter was that user friendly either. Of course I'm still open > > > to suggestions to try to find the best balance between user friendliness, > > > current QEMU command line parsing support, and just getting a bitmap into > > > cpu state one way or another. > > > > I'd ask for consistency with existing practice no matter how flawed if > > we had such consistency. > > > > If I understand your "[PATCH 00/13] target/arm/kvm: enable SVE in > > guests" correctly, the bitmap form of [1, 2, 4] is > > > > -cpu max,sve-vls-map=11 > > > > Observe bit#0 means 1; better document that clearly. > > > > If we used dotted keys to produce an intList, we'd do > > > > -cpu max,sve-vls-map.0=1,sve-vls-map.1=2,sve-vls-map.2=4 > > > > If the option argument is QAPIfied, we additionally get > > > > -cpu '{"type": "max", "sve-vls-map": [1, 2, 4]}' > > > > for free. > > > > If we did it like -numa (please don't), we'd get something like > > > > -cpu max,sve-vls-map=1-2,sve-vls-map=4 > > > > None of the above is exactly a pinnacle of user-friendliness. JSON is > > at least ugly in a regular way. Your bitmap encoded in a number is at > > least concise. > > > > If a numerically encoded bitmap is the least bad option here, I wonder > > why it's not the least bad option for -numa... Perhaps because there 64 > > isn't big enough. > > > > I'm afraid the numerically encoded bitmap will make its way into the > > QAPI schema sooner or later. This will create an unfortunate > > inconsistency with the [int] encoding already there. > > > > Who's going to use sve-vls-map? Humans, or pretty much only machines? > > My thought is primarily machines. If a human wants to use the command > line and SVE, then I'm assuming they'll be happy with sve-max-vq or > figuring out a map they like once and then sticking to it. maybe naive question, but why not use a property/bit as user facing interface, in line with what we do with CPUID bits. (that's assuming that bits have fixed meaning). Yes, it's verbose but follows current practice and works fine with -cpu and -device. (I really hate custom preprocessing of -cpu and we were working hard to remove that in favor of canonical properties at the expense of more verbose CLI). > > Thanks, > drew > ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? 2019-05-14 14:48 ` Igor Mammedov @ 2019-05-15 8:18 ` Andrew Jones 2019-05-15 10:52 ` Igor Mammedov 2019-05-15 11:00 ` Dave Martin 0 siblings, 2 replies; 45+ messages in thread From: Andrew Jones @ 2019-05-15 8:18 UTC (permalink / raw) To: Igor Mammedov Cc: peter.maydell, Daniel P. Berrangé, Markus Armbruster, qemu-devel, dgilbert, Dave.Martin On Tue, May 14, 2019 at 04:48:38PM +0200, Igor Mammedov wrote: > On Tue, 14 May 2019 11:02:25 +0200 > Andrew Jones <drjones@redhat.com> wrote: > > My thought is primarily machines. If a human wants to use the command > > line and SVE, then I'm assuming they'll be happy with sve-max-vq or > > figuring out a map they like once and then sticking to it. > > maybe naive question, but why not use a property/bit as user facing interface, > in line with what we do with CPUID bits. (that's assuming that bits have > fixed meaning). > Yes, it's verbose but follows current practice and works fine with -cpu and > -device. > (I really hate custom preprocessing of -cpu and we were working hard to remove > that in favor of canonical properties at the expense of more verbose CLI). > Are you asking if we should do something like the following? -cpu host,sve1=on,sve=2=on,sve3=off,sve4=on (Where the numbers represent the number of vector quadwords supported.) Naturally that would work, but it would be super verbose and require adding tons of properties. Or maybe you meant something else. If so, please provide an example. Thanks, drew ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? 2019-05-15 8:18 ` Andrew Jones @ 2019-05-15 10:52 ` Igor Mammedov 2019-05-15 11:54 ` Andrew Jones 2019-05-15 11:00 ` Dave Martin 1 sibling, 1 reply; 45+ messages in thread From: Igor Mammedov @ 2019-05-15 10:52 UTC (permalink / raw) To: Andrew Jones Cc: peter.maydell, Daniel P. Berrangé, Markus Armbruster, qemu-devel, dgilbert, Dave.Martin On Wed, 15 May 2019 10:18:54 +0200 Andrew Jones <drjones@redhat.com> wrote: > On Tue, May 14, 2019 at 04:48:38PM +0200, Igor Mammedov wrote: > > On Tue, 14 May 2019 11:02:25 +0200 > > Andrew Jones <drjones@redhat.com> wrote: > > > My thought is primarily machines. If a human wants to use the command > > > line and SVE, then I'm assuming they'll be happy with sve-max-vq or > > > figuring out a map they like once and then sticking to it. > > > > maybe naive question, but why not use a property/bit as user facing interface, > > in line with what we do with CPUID bits. (that's assuming that bits have > > fixed meaning). > > Yes, it's verbose but follows current practice and works fine with -cpu and > > -device. > > (I really hate custom preprocessing of -cpu and we were working hard to remove > > that in favor of canonical properties at the expense of more verbose CLI). > > > > Are you asking if we should do something like the following? > > -cpu host,sve1=on,sve=2=on,sve3=off,sve4=on > > (Where the numbers represent the number of vector quadwords supported.) something like this, but why not replace 1,2,... with more meaning full sve-encrypt/sve-something-else/... since using magic numbers is not very descriptive (but if there is some spec where they come from that we could point users to it might be acceptable too, but I'd reserve number approach for values only). > Naturally that would work, but it would be super verbose and require > adding tons of properties. yep, it's not very (human) user friendly but that's what mgmt applications are for and well users always can prepare a script for long CLI and use whatever abstractions they could think of. > Or maybe you meant something else. If so, > please provide an example. > > Thanks, > drew ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? 2019-05-15 10:52 ` Igor Mammedov @ 2019-05-15 11:54 ` Andrew Jones 2019-05-23 8:35 ` Andrea Bolognani 0 siblings, 1 reply; 45+ messages in thread From: Andrew Jones @ 2019-05-15 11:54 UTC (permalink / raw) To: Igor Mammedov Cc: peter.maydell, Daniel P. Berrangé, ehabkost, Markus Armbruster, qemu-devel, dgilbert, abologna, Dave.Martin On Wed, May 15, 2019 at 12:52:29PM +0200, Igor Mammedov wrote: > On Wed, 15 May 2019 10:18:54 +0200 > Andrew Jones <drjones@redhat.com> wrote: > > > On Tue, May 14, 2019 at 04:48:38PM +0200, Igor Mammedov wrote: > > > On Tue, 14 May 2019 11:02:25 +0200 > > > Andrew Jones <drjones@redhat.com> wrote: > > > > My thought is primarily machines. If a human wants to use the command > > > > line and SVE, then I'm assuming they'll be happy with sve-max-vq or > > > > figuring out a map they like once and then sticking to it. > > > > > > maybe naive question, but why not use a property/bit as user facing interface, > > > in line with what we do with CPUID bits. (that's assuming that bits have > > > fixed meaning). > > > Yes, it's verbose but follows current practice and works fine with -cpu and > > > -device. > > > (I really hate custom preprocessing of -cpu and we were working hard to remove > > > that in favor of canonical properties at the expense of more verbose CLI). > > > > > > > Are you asking if we should do something like the following? > > > > -cpu host,sve1=on,sve=2=on,sve3=off,sve4=on > > > > (Where the numbers represent the number of vector quadwords supported.) > > something like this, but why not replace 1,2,... with more meaning full > sve-encrypt/sve-something-else/... > since using magic numbers is not very descriptive > (but if there is some spec where they come from that we could point users to > it might be acceptable too, but I'd reserve number approach for values only). The numbers aren't magic, they're part of the name. '1' in the above 'sve1' means one quadword. It would probably have been better to use bits instead in the example, i.e. -cpu host,sve128=on,sve256=on,sve384=off,sve512=on where it's now clear that "sve512" has an analogy with x86's "avx512". > > > Naturally that would work, but it would be super verbose and require > > adding tons of properties. > yep, it's not very (human) user friendly but that's what mgmt applications > are for and well users always can prepare a script for long CLI and use > whatever abstractions they could think of. > So I set off to convince Igor of the wide word idea (he sits next to me, so I didn't have go far), but he has convinced me of the above property idea. He used the magic phrase: "less code would be needed". If we use the properties like above then we get introspection for free (cpu property listing which libvirt already knows how to do) - so no QMP query needed. The cost is adding several properties (16 to handle the current 2048-bit limit), but I guess that's cheap enough. The command line is verbose, but also much easier for a human to construct and read. I'm pretty sold on this path, but adding Andrea and Eduardo for their input as well. Thanks, drew ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? 2019-05-15 11:54 ` Andrew Jones @ 2019-05-23 8:35 ` Andrea Bolognani 2019-05-24 18:24 ` Eduardo Habkost 0 siblings, 1 reply; 45+ messages in thread From: Andrea Bolognani @ 2019-05-23 8:35 UTC (permalink / raw) To: Andrew Jones, Igor Mammedov Cc: peter.maydell, Daniel P. Berrangé, ehabkost, Markus Armbruster, qemu-devel, dgilbert, Dave.Martin On Wed, 2019-05-15 at 13:54 +0200, Andrew Jones wrote: > On Wed, May 15, 2019 at 12:52:29PM +0200, Igor Mammedov wrote: > > since using magic numbers is not very descriptive > > (but if there is some spec where they come from that we could point users to > > it might be acceptable too, but I'd reserve number approach for values only). > > The numbers aren't magic, they're part of the name. '1' in the above > 'sve1' means one quadword. It would probably have been better to use bits > instead in the example, i.e. > > -cpu host,sve128=on,sve256=on,sve384=off,sve512=on > > where it's now clear that "sve512" has an analogy with x86's "avx512". > [...] > > So I set off to convince Igor of the wide word idea (he sits next to me, > so I didn't have go far), but he has convinced me of the above property > idea. He used the magic phrase: "less code would be needed". If we use > the properties like above then we get introspection for free (cpu property > listing which libvirt already knows how to do) - so no QMP query needed. > The cost is adding several properties (16 to handle the current 2048-bit > limit), but I guess that's cheap enough. The command line is verbose, but > also much easier for a human to construct and read. I'm pretty sold on > this path, but adding Andrea and Eduardo for their input as well. Sorry for taking a while to respond. Anyway, the above looks good to me as a general direction, but note that you'll have to implement at the very least the query-cpu-model-expansion QMP command for the introspection to work. query-cpu-model-baseline and query-cpu-model-comparison are two more QMP command which, while perhaps not immediately applicabile, we will want to implement at some point; more in general, what s390x is doing with respect to CPU models is a good blueprint, according to the libvirt developer who's the most involved with that specific area of the project. -- Andrea Bolognani / Red Hat / Virtualization ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? 2019-05-23 8:35 ` Andrea Bolognani @ 2019-05-24 18:24 ` Eduardo Habkost 2019-05-27 16:29 ` Andrea Bolognani 0 siblings, 1 reply; 45+ messages in thread From: Eduardo Habkost @ 2019-05-24 18:24 UTC (permalink / raw) To: Andrea Bolognani Cc: peter.maydell, Andrew Jones, Daniel P. Berrangé, qemu-devel, Markus Armbruster, Igor Mammedov, Dave.Martin, dgilbert On Thu, May 23, 2019 at 10:35:24AM +0200, Andrea Bolognani wrote: > On Wed, 2019-05-15 at 13:54 +0200, Andrew Jones wrote: > > On Wed, May 15, 2019 at 12:52:29PM +0200, Igor Mammedov wrote: > > > since using magic numbers is not very descriptive > > > (but if there is some spec where they come from that we could point users to > > > it might be acceptable too, but I'd reserve number approach for values only). > > > > The numbers aren't magic, they're part of the name. '1' in the above > > 'sve1' means one quadword. It would probably have been better to use bits > > instead in the example, i.e. > > > > -cpu host,sve128=on,sve256=on,sve384=off,sve512=on > > > > where it's now clear that "sve512" has an analogy with x86's "avx512". > > > [...] > > > > So I set off to convince Igor of the wide word idea (he sits next to me, > > so I didn't have go far), but he has convinced me of the above property > > idea. He used the magic phrase: "less code would be needed". If we use > > the properties like above then we get introspection for free (cpu property > > listing which libvirt already knows how to do) - so no QMP query needed. > > The cost is adding several properties (16 to handle the current 2048-bit > > limit), but I guess that's cheap enough. The command line is verbose, but > > also much easier for a human to construct and read. I'm pretty sold on > > this path, but adding Andrea and Eduardo for their input as well. > > Sorry for taking a while to respond. Anyway, the above looks good to > me as a general direction, but note that you'll have to implement at > the very least the query-cpu-model-expansion QMP command for the > introspection to work. Why is query-cpu-model-expansion needed? Isn't device-list-properties enough? > > query-cpu-model-baseline and query-cpu-model-comparison are two more > QMP command which, while perhaps not immediately applicabile, we will > want to implement at some point; more in general, what s390x is doing > with respect to CPU models is a good blueprint, according to the > libvirt developer who's the most involved with that specific area of > the project. Agreed. Even if not necessary right now, query-cpu-model-* will probably be needed eventually. -- Eduardo ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? 2019-05-24 18:24 ` Eduardo Habkost @ 2019-05-27 16:29 ` Andrea Bolognani 2019-05-27 18:59 ` Eduardo Habkost 0 siblings, 1 reply; 45+ messages in thread From: Andrea Bolognani @ 2019-05-27 16:29 UTC (permalink / raw) To: Eduardo Habkost Cc: peter.maydell, Andrew Jones, Daniel P. Berrangé, qemu-devel, Markus Armbruster, Igor Mammedov, Dave.Martin, dgilbert On Fri, 2019-05-24 at 15:24 -0300, Eduardo Habkost wrote: > On Thu, May 23, 2019 at 10:35:24AM +0200, Andrea Bolognani wrote: > > [...] the above looks good to > > me as a general direction, but note that you'll have to implement at > > the very least the query-cpu-model-expansion QMP command for the > > introspection to work. > > Why is query-cpu-model-expansion needed? Isn't > device-list-properties enough? Good question. I'll have to check with Jirka, but from playing with both commands it looks like the latter returns a superset of what the former does, so for the purpose of figuring out which vector lengths the QEMU binary recognizes it should be good enough; I suspect, however, that query-cpu-model-expansion might be (made to be) smarter and for example not report vector lengths that the underlying hardware doesn't support, which would be valuable for the purpose of user friendly error reporting and allowing applications to decide which vector lengths to request when creating guests. I'll try to come back with more than theories soon :) -- Andrea Bolognani / Red Hat / Virtualization ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? 2019-05-27 16:29 ` Andrea Bolognani @ 2019-05-27 18:59 ` Eduardo Habkost 0 siblings, 0 replies; 45+ messages in thread From: Eduardo Habkost @ 2019-05-27 18:59 UTC (permalink / raw) To: Andrea Bolognani Cc: peter.maydell, Andrew Jones, Daniel P. Berrangé, qemu-devel, Markus Armbruster, Igor Mammedov, Dave.Martin, dgilbert On Mon, May 27, 2019 at 06:29:44PM +0200, Andrea Bolognani wrote: > On Fri, 2019-05-24 at 15:24 -0300, Eduardo Habkost wrote: > > On Thu, May 23, 2019 at 10:35:24AM +0200, Andrea Bolognani wrote: > > > [...] the above looks good to > > > me as a general direction, but note that you'll have to implement at > > > the very least the query-cpu-model-expansion QMP command for the > > > introspection to work. > > > > Why is query-cpu-model-expansion needed? Isn't > > device-list-properties enough? > > Good question. > > I'll have to check with Jirka, but from playing with both commands > it looks like the latter returns a superset of what the former does, > so for the purpose of figuring out which vector lengths the QEMU > binary recognizes it should be good enough; I suspect, however, that > query-cpu-model-expansion might be (made to be) smarter and for > example not report vector lengths that the underlying hardware > doesn't support, which would be valuable for the purpose of user > friendly error reporting and allowing applications to decide which > vector lengths to request when creating guests. Yes, query-cpu-model-expansion returns additional information, so it depends on what exactly you are looking for. If you want to know which properties a given QEMU binary supports in the command-line, `device-list-properties` is supposed to be enough. If you need to know which properties can be really enabled in a given host (based on QEMU+KVM+hardware capabilities), you'll need `query-cpu-model-expansion model=max`. -- Eduardo ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? 2019-05-15 8:18 ` Andrew Jones 2019-05-15 10:52 ` Igor Mammedov @ 2019-05-15 11:00 ` Dave Martin 2019-05-15 11:09 ` Dr. David Alan Gilbert 2019-05-15 11:42 ` Andrew Jones 1 sibling, 2 replies; 45+ messages in thread From: Dave Martin @ 2019-05-15 11:00 UTC (permalink / raw) To: Andrew Jones Cc: peter.maydell, Daniel P. Berrangé, Markus Armbruster, dgilbert, qemu-devel, Igor Mammedov On Wed, May 15, 2019 at 09:18:54AM +0100, Andrew Jones wrote: > On Tue, May 14, 2019 at 04:48:38PM +0200, Igor Mammedov wrote: > > On Tue, 14 May 2019 11:02:25 +0200 > > Andrew Jones <drjones@redhat.com> wrote: > > > My thought is primarily machines. If a human wants to use the command > > > line and SVE, then I'm assuming they'll be happy with sve-max-vq or > > > figuring out a map they like once and then sticking to it. > > > > maybe naive question, but why not use a property/bit as user facing interface, > > in line with what we do with CPUID bits. (that's assuming that bits have > > fixed meaning). > > Yes, it's verbose but follows current practice and works fine with -cpu and > > -device. > > (I really hate custom preprocessing of -cpu and we were working hard to remove > > that in favor of canonical properties at the expense of more verbose CLI). > > > > Are you asking if we should do something like the following? > > -cpu host,sve1=on,sve=2=on,sve3=off,sve4=on Note, there is nothing SVE-specific about this. Either enabling features on a per-vcpu basis is justified, or it isn't: if it's justified, then it would be better to have a general way of specifying per-vcpu properties, rather than it being reinvented per feature. Creating mismatched configurations is allowed by the architecture and so it's useful for testing the kernel, but probably less useful for real- world use cases today. So it may be a good idea to get the symmetric support sorted out first before thinking about whether and how to specify asymmetric configurations. Cheers ---Dave ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? 2019-05-15 11:00 ` Dave Martin @ 2019-05-15 11:09 ` Dr. David Alan Gilbert 2019-05-15 12:51 ` Dave Martin 2019-05-15 11:42 ` Andrew Jones 1 sibling, 1 reply; 45+ messages in thread From: Dr. David Alan Gilbert @ 2019-05-15 11:09 UTC (permalink / raw) To: Dave Martin Cc: peter.maydell, Andrew Jones, Daniel P. Berrangé, qemu-devel, Markus Armbruster, Igor Mammedov * Dave Martin (Dave.Martin@arm.com) wrote: > On Wed, May 15, 2019 at 09:18:54AM +0100, Andrew Jones wrote: > > On Tue, May 14, 2019 at 04:48:38PM +0200, Igor Mammedov wrote: > > > On Tue, 14 May 2019 11:02:25 +0200 > > > Andrew Jones <drjones@redhat.com> wrote: > > > > My thought is primarily machines. If a human wants to use the command > > > > line and SVE, then I'm assuming they'll be happy with sve-max-vq or > > > > figuring out a map they like once and then sticking to it. > > > > > > maybe naive question, but why not use a property/bit as user facing interface, > > > in line with what we do with CPUID bits. (that's assuming that bits have > > > fixed meaning). > > > Yes, it's verbose but follows current practice and works fine with -cpu and > > > -device. > > > (I really hate custom preprocessing of -cpu and we were working hard to remove > > > that in favor of canonical properties at the expense of more verbose CLI). > > > > > > > Are you asking if we should do something like the following? > > > > -cpu host,sve1=on,sve=2=on,sve3=off,sve4=on > > Note, there is nothing SVE-specific about this. > > Either enabling features on a per-vcpu basis is justified, or it isn't: > if it's justified, then it would be better to have a general way of > specifying per-vcpu properties, rather than it being reinvented per > feature. SVE *is* a bit unusual. In most CPU features they're actually features, they're on or off, so we have a big list of features that are enabled/disabled. We've had that type of thing (at least on x86) for years and it's OK. We've got one or two things where they're numerical (e.g. host-physbits) and we struggle a bit with how to handle them. SVE is somewhere in between - it's a list of numbers, apparently a fairly large arbitrarily set of numbers that could be chosen so you'd need lots of feature flags (sve1...sve64 say or more); so that doesn't fit the existing things we've had that have worked. Dave > Creating mismatched configurations is allowed by the architecture and so > it's useful for testing the kernel, but probably less useful for real- > world use cases today. > > So it may be a good idea to get the symmetric support sorted out first > before thinking about whether and how to specify asymmetric > configurations. > > Cheers > ---Dave -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? 2019-05-15 11:09 ` Dr. David Alan Gilbert @ 2019-05-15 12:51 ` Dave Martin 0 siblings, 0 replies; 45+ messages in thread From: Dave Martin @ 2019-05-15 12:51 UTC (permalink / raw) To: Dr. David Alan Gilbert Cc: peter.maydell, Andrew Jones, Daniel P. Berrangé, qemu-devel, Markus Armbruster, Igor Mammedov On Wed, May 15, 2019 at 12:09:04PM +0100, Dr. David Alan Gilbert wrote: > * Dave Martin (Dave.Martin@arm.com) wrote: > > On Wed, May 15, 2019 at 09:18:54AM +0100, Andrew Jones wrote: > > > On Tue, May 14, 2019 at 04:48:38PM +0200, Igor Mammedov wrote: > > > > On Tue, 14 May 2019 11:02:25 +0200 > > > > Andrew Jones <drjones@redhat.com> wrote: > > > > > My thought is primarily machines. If a human wants to use the command > > > > > line and SVE, then I'm assuming they'll be happy with sve-max-vq or > > > > > figuring out a map they like once and then sticking to it. > > > > > > > > maybe naive question, but why not use a property/bit as user facing interface, > > > > in line with what we do with CPUID bits. (that's assuming that bits have > > > > fixed meaning). > > > > Yes, it's verbose but follows current practice and works fine with -cpu and > > > > -device. > > > > (I really hate custom preprocessing of -cpu and we were working hard to remove > > > > that in favor of canonical properties at the expense of more verbose CLI). > > > > > > > > > > Are you asking if we should do something like the following? > > > > > > -cpu host,sve1=on,sve=2=on,sve3=off,sve4=on > > > > Note, there is nothing SVE-specific about this. > > > > Either enabling features on a per-vcpu basis is justified, or it isn't: > > if it's justified, then it would be better to have a general way of > > specifying per-vcpu properties, rather than it being reinvented per > > feature. > > SVE *is* a bit unusual. In most CPU features they're actually features, > they're on or off, so we have a big list of features that are > enabled/disabled. We've had that type of thing (at least on x86) for > years and it's OK. > We've got one or two things where they're numerical > (e.g. host-physbits) and we struggle a bit with how to handle them. > > SVE is somewhere in between - it's a list of numbers, apparently a > fairly large arbitrarily set of numbers that could be chosen so you'd > need lots of feature flags (sve1...sve64 say or more); so that doesn't > fit the existing things we've had that have worked. I see what you mean now. SVE configuration is indeed more than just a boolean flag. Cheers ---Dave ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? 2019-05-15 11:00 ` Dave Martin 2019-05-15 11:09 ` Dr. David Alan Gilbert @ 2019-05-15 11:42 ` Andrew Jones 2019-05-15 12:50 ` Dave Martin 1 sibling, 1 reply; 45+ messages in thread From: Andrew Jones @ 2019-05-15 11:42 UTC (permalink / raw) To: Dave Martin Cc: peter.maydell, Daniel P. Berrangé, Markus Armbruster, dgilbert, qemu-devel, Igor Mammedov On Wed, May 15, 2019 at 12:00:45PM +0100, Dave Martin wrote: > On Wed, May 15, 2019 at 09:18:54AM +0100, Andrew Jones wrote: > > On Tue, May 14, 2019 at 04:48:38PM +0200, Igor Mammedov wrote: > > > On Tue, 14 May 2019 11:02:25 +0200 > > > Andrew Jones <drjones@redhat.com> wrote: > > > > My thought is primarily machines. If a human wants to use the command > > > > line and SVE, then I'm assuming they'll be happy with sve-max-vq or > > > > figuring out a map they like once and then sticking to it. > > > > > > maybe naive question, but why not use a property/bit as user facing interface, > > > in line with what we do with CPUID bits. (that's assuming that bits have > > > fixed meaning). > > > Yes, it's verbose but follows current practice and works fine with -cpu and > > > -device. > > > (I really hate custom preprocessing of -cpu and we were working hard to remove > > > that in favor of canonical properties at the expense of more verbose CLI). > > > > > > > Are you asking if we should do something like the following? > > > > -cpu host,sve1=on,sve=2=on,sve3=off,sve4=on > > Note, there is nothing SVE-specific about this. In the above example there is some specific SVE stuff there. If the command line has sve4=on, then it must also have sve1=on and sve2=on, per the architecture requiring all smaller power-of-2 vector lengths. Only sve3 is optional, but because it's optional we have to explicitly state when it's on or off in order to ensure we can cleanly fail a migration to a host that doesn't support that option. > > Either enabling features on a per-vcpu basis is justified, or it isn't: > if it's justified, then it would be better to have a general way of > specifying per-vcpu properties, rather than it being reinvented per > feature. > > Creating mismatched configurations is allowed by the architecture and so > it's useful for testing the kernel, but probably less useful for real- > world use cases today. > > So it may be a good idea to get the symmetric support sorted out first > before thinking about whether and how to specify asymmetric > configurations. These properties are per-vcpu for KVM only. QEMU doesn't have a way to allow per-vcpu features to be described on the command line yet. With '-cpu host,...' The '...' applies to all vcpus. So we are "just" working on the symmetric support now. Thanks, drew ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? 2019-05-15 11:42 ` Andrew Jones @ 2019-05-15 12:50 ` Dave Martin 0 siblings, 0 replies; 45+ messages in thread From: Dave Martin @ 2019-05-15 12:50 UTC (permalink / raw) To: Andrew Jones Cc: peter.maydell, Daniel P. Berrangé, Markus Armbruster, dgilbert, qemu-devel, Igor Mammedov On Wed, May 15, 2019 at 12:42:44PM +0100, Andrew Jones wrote: > On Wed, May 15, 2019 at 12:00:45PM +0100, Dave Martin wrote: > > On Wed, May 15, 2019 at 09:18:54AM +0100, Andrew Jones wrote: > > > On Tue, May 14, 2019 at 04:48:38PM +0200, Igor Mammedov wrote: > > > > On Tue, 14 May 2019 11:02:25 +0200 > > > > Andrew Jones <drjones@redhat.com> wrote: > > > > > My thought is primarily machines. If a human wants to use the command > > > > > line and SVE, then I'm assuming they'll be happy with sve-max-vq or > > > > > figuring out a map they like once and then sticking to it. > > > > > > > > maybe naive question, but why not use a property/bit as user facing interface, > > > > in line with what we do with CPUID bits. (that's assuming that bits have > > > > fixed meaning). > > > > Yes, it's verbose but follows current practice and works fine with -cpu and > > > > -device. > > > > (I really hate custom preprocessing of -cpu and we were working hard to remove > > > > that in favor of canonical properties at the expense of more verbose CLI). > > > > > > > > > > Are you asking if we should do something like the following? > > > > > > -cpu host,sve1=on,sve=2=on,sve3=off,sve4=on > > > > Note, there is nothing SVE-specific about this. > > In the above example there is some specific SVE stuff there. If the > command line has sve4=on, then it must also have sve1=on and sve2=on, > per the architecture requiring all smaller power-of-2 vector lengths. > Only sve3 is optional, but because it's optional we have to explicitly > state when it's on or off in order to ensure we can cleanly fail a > migration to a host that doesn't support that option. > > > > > Either enabling features on a per-vcpu basis is justified, or it isn't: > > if it's justified, then it would be better to have a general way of > > specifying per-vcpu properties, rather than it being reinvented per > > feature. > > > > Creating mismatched configurations is allowed by the architecture and so > > it's useful for testing the kernel, but probably less useful for real- > > world use cases today. > > > > So it may be a good idea to get the symmetric support sorted out first > > before thinking about whether and how to specify asymmetric > > configurations. > > These properties are per-vcpu for KVM only. QEMU doesn't have a way > to allow per-vcpu features to be described on the command line yet. > With '-cpu host,...' The '...' applies to all vcpus. So we are "just" > working on the symmetric support now. OK, I think I misunderstood what was being proposed here. Until/unless someone comes up with a compelling use case, I think it's entirely reasonable for QEMU not to support asymmetry of this sort. Cheers ---Dave ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? 2019-05-14 4:54 ` Markus Armbruster 2019-05-14 9:02 ` Andrew Jones @ 2019-05-14 15:28 ` Dave Martin 1 sibling, 0 replies; 45+ messages in thread From: Dave Martin @ 2019-05-14 15:28 UTC (permalink / raw) To: Markus Armbruster Cc: peter.maydell, Andrew Jones, Daniel P.Berrangé, dgilbert, qemu-devel, imammedo On Tue, May 14, 2019 at 05:54:03AM +0100, Markus Armbruster wrote: > Andrew Jones <drjones@redhat.com> writes: > > > On Thu, Apr 18, 2019 at 07:48:09PM +0200, Markus Armbruster wrote: > >> Daniel P. Berrangé <berrange@redhat.com> writes: > >> > >> > On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote: > >> >> Hi all, > >> >> > >> >> First some background: > >> >> > >> >> For the userspace side of AArch64 guest SVE support we need to > >> >> expose KVM's allowed vector lengths bitmap to the user and allow > >> >> the user to choose a subset of that bitmap. Since bitmaps are a > >> >> bit awkward to work with then we'll likely want to expose it as > >> >> an array of vector lengths instead. Also, assuming we want to > >> >> expose the lengths as number-of-quadwords (quadword == 128 bits > >> >> for AArch64 and vector lengths must be multiples of quadwords) > >> >> rather than number-of-bits, then an example array (which will > >> >> always be a sequence) might be > >> >> > >> >> [ 8, 16, 32 ] > >> >> > >> >> The user may choose a subsequence, but only through truncation, > >> >> i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. > >> >> > >> >> Furthermore, different hosts may support different sequences > >> >> which have the same maximum. For example, if the above sequence > >> >> is for Host_A, then Host_B could be > >> >> > >> >> [ 8, 16, 24, 32 ] > >> >> > >> >> The host must support all lengths in the sequence, which means > >> >> that while Host_A supports 32, since it doesn't support 24 and > >> >> we can only truncate sequences, we must use either [ 8 ] or > >> >> [ 8, 16 ] for a compatible sequence if we intend to migrate > >> >> between the hosts. > >> >> > >> >> Now to the $SUBJECT question: > >> >> > >> >> My feeling is that we should require the sequence to be > >> >> provided on the command line as a cpu property. Something > >> >> like > >> >> > >> >> -cpu host,sve-vl-list=8:16 > >> >> > >> >> (I chose ':' for the delimiter because ',' can't work, but > >> >> if there's a better choice, then that's fine by me.) > >> >> > >> >> Afaict a property list like this will require a new parser, > >> > >> We had 20+ of those when I last counted. Among the more annoying > >> reasons CLI QAPIfication is hard[1]. > >> > >> >> which feels a bit funny since it seems we should already > >> >> have support for this type of thing somewhere in QEMU. So, > >> >> the question is: do we? I see we have array properties, but > >> >> I don't believe that works with the command line. Should we > >> >> only use QMP for this? We already want some QMP in order to > >> >> query the supported vector lengths. Maybe we should use QMP > >> >> to set the selection too? But then what about command line > >> >> support for developers? And if the property is on the command > >> >> line then we don't have to add it to the migration stream. > >> > > >> > You should be able to use arrays from the CLI with QemuOpts by repeating > >> > the same option name many times, though I can't say it is a very > >> > nice approach if you have many values to list as it gets very repetative. > >> > >> Yes, this is one of the ways the current CLI does lists. It's also one > >> of the more annoying reasons CLI QAPIfication is hard[2]. > >> > >> QemuOpts let the last param=value win the stupidest way that could > >> possibly work (I respect that): add to the front of the list, search it > >> front to back. > >> > >> Then somebody discovered that if you search the list manually, you can > >> see them all, and abuse that to get a list-valued param. I'm sure that > >> felt clever at the time. > >> > >> Another way to do lists the funky list feature of string input and opts > >> visitor. Yet another annoying reason CLI QAPIfication is hard[3]. > >> > >> We use the opts visitor's list feature for -numa node,cpus=... Hmm, > >> looks like we even combine it with the "multiple param=value build up a > >> list" technique: -smp node,cpus=0-1,cpus=4-5 denotes [0,1,4,5]. > >> > >> > That's the curse of not having a good CLI syntax for non-scalar data in > >> > QemuOpts & why Markus believes we should switch to JSON for the CLI too > >> > > >> > -cpu host,sve-vl=8,sve-vl=16 > >> > >> We actually have CLI syntax for non-scalar data: dotted keys. Dotted > >> keys are syntactic sugar for JSON. It looks friendlier than JSON for > >> simple cases, then gets uglier as things get more complex, and then it > >> falls apart: it can't quite express all of JSON. > >> > >> Example: sve-vl.0=8,sve-vl.1=16 > >> gets desugared into {"sve": [8, 16]} > >> if the QAPI schema has 'sve': ['int']. > >> > >> The comment at the beginning of util/keyval.c explains it in more > >> detail. > >> > >> It powers -blockdev and -display. Both options accept either JSON or > >> dotted keys. If the option argument starts with '{', it's JSON. > >> Management applications should stick to JSON. > >> > >> > >> [1] Towards a more expressive and introspectable QEMU command line > >> https://www.linux-kvm.org/images/f/f2/Armbru-qapi-cmdline_1.pdf > >> Slide 34 "Backward compatibility" item 1 > >> > >> [2] ibid, item 4 > >> > >> [3] ibid, item 3 > >> > > > > Sorry I forgot to follow up to this earlier. I looked at the examples > > provided and saw they were all for independent command line options, > > rather than command line options like '-cpu' that then accepts additional > > properties. I couldn't see how I could use ',' to separate array members > > when using properties or to use an array property input on the command > > line. > > The argument of -cpu is parsed ad hoc. Unlike QemuOpts and dotted keys, > parse_cpu_option() doesn't seem to support escaping ','. Not that > escaping would be a user-friendly solution. > > > In the end I opted to use a single uint64_t for a bitmap, as 64 is > > big enough for now, > > Do you think it'll remain big enough? The answer is "probably". I didn't want to have to redesign the kernel ABI again if the architecture busts this limit in future. 64 bits already goes beyond what the SVE architecture supports today. But it would be preferable for QEMUs interface to have the same arbitrary limit as the kernel, rather than a different arbitrary limit. Depends how painful it is to support, I guess. [...] Cheers ---Dave ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? @ 2019-04-19 0:07 ` Laszlo Ersek 0 siblings, 0 replies; 45+ messages in thread From: Laszlo Ersek @ 2019-04-19 0:07 UTC (permalink / raw) To: Andrew Jones, qemu-devel Cc: peter.maydell, imammedo, Dave.Martin, armbru, dgilbert On 04/18/19 11:28, Andrew Jones wrote: > Hi all, > > First some background: > > For the userspace side of AArch64 guest SVE support we need to > expose KVM's allowed vector lengths bitmap to the user and allow > the user to choose a subset of that bitmap. Since bitmaps are a > bit awkward to work with then we'll likely want to expose it as > an array of vector lengths instead. Also, assuming we want to > expose the lengths as number-of-quadwords (quadword == 128 bits > for AArch64 and vector lengths must be multiples of quadwords) > rather than number-of-bits, then an example array (which will > always be a sequence) might be > > [ 8, 16, 32 ] > > The user may choose a subsequence, but only through truncation, > i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. > > Furthermore, different hosts may support different sequences > which have the same maximum. For example, if the above sequence > is for Host_A, then Host_B could be > > [ 8, 16, 24, 32 ] > > The host must support all lengths in the sequence, which means > that while Host_A supports 32, since it doesn't support 24 and > we can only truncate sequences, we must use either [ 8 ] or > [ 8, 16 ] for a compatible sequence if we intend to migrate > between the hosts. > > Now to the $SUBJECT question: > > My feeling is that we should require the sequence to be > provided on the command line as a cpu property. Something > like > > -cpu host,sve-vl-list=8:16 > > (I chose ':' for the delimiter because ',' can't work, but > if there's a better choice, then that's fine by me.) > > Afaict a property list like this will require a new parser, > which feels a bit funny since it seems we should already > have support for this type of thing somewhere in QEMU. So, > the question is: do we? I think so. OptsVisitor can parse repeated properties from an option and turn them into a list that you define in QAPI schema. More precisely, you'd define the schema first, subject to "rules" and "restrictions", and then OptsVisitor would allow you to grab that schema from command line options. A list of integers qualifies. See for example: [Qemu-devel] [PATCH v2 0/8] OptsVisitor: support / flatten integer range https://lists.gnu.org/archive/html/qemu-devel/2013-08/msg02686.html (this is actually a feature on top of what you seem to need, but this is what I could find right now). Although I worked on this stuff in 2012-2013 (and so I remember basically nothing about it), it should still function, because the qtest suite exercises it with good coverage. So, suggestions: - git show -s eb7ee2cb - check how "-numa" is parsed, and steal it :) If OptsVisitor is hard to combine with existent options that are already parsed in a different manner (and I do think it would be difficult), then a new option could be introduced. For example: -sve-vector-lengths allow=8,allow=16 which should give you an ordered list of lengths, and you could distill that manually into a single bitmap. Thanks, Laszlo PS: I do mean I don't remember anything about OptsVisitor :/ > I see we have array properties, but > I don't believe that works with the command line. Should we > only use QMP for this? We already want some QMP in order to > query the supported vector lengths. Maybe we should use QMP > to set the selection too? But then what about command line > support for developers? And if the property is on the command > line then we don't have to add it to the migration stream. > > Thanks, > drew > ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Qemu-devel] How do we do user input bitmap properties? @ 2019-04-19 0:07 ` Laszlo Ersek 0 siblings, 0 replies; 45+ messages in thread From: Laszlo Ersek @ 2019-04-19 0:07 UTC (permalink / raw) To: Andrew Jones, qemu-devel Cc: peter.maydell, dgilbert, Dave.Martin, armbru, imammedo On 04/18/19 11:28, Andrew Jones wrote: > Hi all, > > First some background: > > For the userspace side of AArch64 guest SVE support we need to > expose KVM's allowed vector lengths bitmap to the user and allow > the user to choose a subset of that bitmap. Since bitmaps are a > bit awkward to work with then we'll likely want to expose it as > an array of vector lengths instead. Also, assuming we want to > expose the lengths as number-of-quadwords (quadword == 128 bits > for AArch64 and vector lengths must be multiples of quadwords) > rather than number-of-bits, then an example array (which will > always be a sequence) might be > > [ 8, 16, 32 ] > > The user may choose a subsequence, but only through truncation, > i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. > > Furthermore, different hosts may support different sequences > which have the same maximum. For example, if the above sequence > is for Host_A, then Host_B could be > > [ 8, 16, 24, 32 ] > > The host must support all lengths in the sequence, which means > that while Host_A supports 32, since it doesn't support 24 and > we can only truncate sequences, we must use either [ 8 ] or > [ 8, 16 ] for a compatible sequence if we intend to migrate > between the hosts. > > Now to the $SUBJECT question: > > My feeling is that we should require the sequence to be > provided on the command line as a cpu property. Something > like > > -cpu host,sve-vl-list=8:16 > > (I chose ':' for the delimiter because ',' can't work, but > if there's a better choice, then that's fine by me.) > > Afaict a property list like this will require a new parser, > which feels a bit funny since it seems we should already > have support for this type of thing somewhere in QEMU. So, > the question is: do we? I think so. OptsVisitor can parse repeated properties from an option and turn them into a list that you define in QAPI schema. More precisely, you'd define the schema first, subject to "rules" and "restrictions", and then OptsVisitor would allow you to grab that schema from command line options. A list of integers qualifies. See for example: [Qemu-devel] [PATCH v2 0/8] OptsVisitor: support / flatten integer range https://lists.gnu.org/archive/html/qemu-devel/2013-08/msg02686.html (this is actually a feature on top of what you seem to need, but this is what I could find right now). Although I worked on this stuff in 2012-2013 (and so I remember basically nothing about it), it should still function, because the qtest suite exercises it with good coverage. So, suggestions: - git show -s eb7ee2cb - check how "-numa" is parsed, and steal it :) If OptsVisitor is hard to combine with existent options that are already parsed in a different manner (and I do think it would be difficult), then a new option could be introduced. For example: -sve-vector-lengths allow=8,allow=16 which should give you an ordered list of lengths, and you could distill that manually into a single bitmap. Thanks, Laszlo PS: I do mean I don't remember anything about OptsVisitor :/ > I see we have array properties, but > I don't believe that works with the command line. Should we > only use QMP for this? We already want some QMP in order to > query the supported vector lengths. Maybe we should use QMP > to set the selection too? But then what about command line > support for developers? And if the property is on the command > line then we don't have to add it to the migration stream. > > Thanks, > drew > ^ permalink raw reply [flat|nested] 45+ messages in thread
end of thread, other threads:[~2019-05-27 19:00 UTC | newest] Thread overview: 45+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-04-18 9:28 [Qemu-devel] How do we do user input bitmap properties? Andrew Jones 2019-04-18 9:28 ` Andrew Jones 2019-04-18 9:46 ` Andrew Jones 2019-04-18 9:46 ` Andrew Jones 2019-04-18 10:52 ` Dave Martin 2019-04-18 10:52 ` Dave Martin 2019-04-18 11:28 ` Andrew Jones 2019-04-18 11:28 ` Andrew Jones 2019-04-18 14:03 ` Dave Martin 2019-04-18 14:03 ` Dave Martin 2019-04-18 14:43 ` Andrew Jones 2019-04-18 14:43 ` Andrew Jones 2019-04-18 14:46 ` Dave Martin 2019-04-18 14:46 ` Dave Martin 2019-04-18 14:57 ` Dr. David Alan Gilbert 2019-04-18 14:57 ` Dr. David Alan Gilbert 2019-04-18 11:26 ` Daniel P. Berrangé 2019-04-18 11:26 ` Daniel P. Berrangé 2019-04-18 15:09 ` Andrew Jones 2019-04-18 15:09 ` Andrew Jones 2019-04-18 17:48 ` Markus Armbruster 2019-04-18 17:48 ` Markus Armbruster 2019-05-13 18:42 ` Andrew Jones 2019-05-14 4:54 ` Markus Armbruster 2019-05-14 9:02 ` Andrew Jones 2019-05-14 13:32 ` Markus Armbruster 2019-05-15 8:15 ` Andrew Jones 2019-05-15 10:53 ` Dave Martin 2019-05-15 10:59 ` Dr. David Alan Gilbert 2019-05-14 14:48 ` Igor Mammedov 2019-05-15 8:18 ` Andrew Jones 2019-05-15 10:52 ` Igor Mammedov 2019-05-15 11:54 ` Andrew Jones 2019-05-23 8:35 ` Andrea Bolognani 2019-05-24 18:24 ` Eduardo Habkost 2019-05-27 16:29 ` Andrea Bolognani 2019-05-27 18:59 ` Eduardo Habkost 2019-05-15 11:00 ` Dave Martin 2019-05-15 11:09 ` Dr. David Alan Gilbert 2019-05-15 12:51 ` Dave Martin 2019-05-15 11:42 ` Andrew Jones 2019-05-15 12:50 ` Dave Martin 2019-05-14 15:28 ` Dave Martin 2019-04-19 0:07 ` Laszlo Ersek 2019-04-19 0:07 ` Laszlo Ersek
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.