[PATCH] iotests: Fix up python style in 300

[PATCH] iotests: Fix up python style in 300

[Eric Blake]

Break some long lines, and relax our type hints to be more generic to
any JSON, in order to more easily permit the additional JSON depth now
possible in migration parameters.  Detected by iotest 297.

Fixes: ca4bfec41d56
 (qemu-iotests: 300: Add test case for modifying persistence of bitmap)
Reported-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
---
 tests/qemu-iotests/300 | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/tests/qemu-iotests/300 b/tests/qemu-iotests/300
index 63036f6a6e13..adb927629747 100755
--- a/tests/qemu-iotests/300
+++ b/tests/qemu-iotests/300
@@ -22,7 +22,7 @@
 import os
 import random
 import re
-from typing import Dict, List, Optional, Union
+from typing import Dict, List, Optional

 import iotests

@@ -30,7 +30,7 @@ import iotests
 # pylint: disable=wrong-import-order
 import qemu

-BlockBitmapMapping = List[Dict[str, Union[str, List[Dict[str, str]]]]]
+BlockBitmapMapping = List[Dict[str, object]]

 mig_sock = os.path.join(iotests.sock_dir, 'mig_sock')

@@ -602,7 +602,8 @@ class TestCrossAliasMigration(TestDirtyBitmapMigration):

 class TestAliasTransformMigration(TestDirtyBitmapMigration):
     """
-    Tests the 'transform' option which modifies bitmap persistence on migration.
+    Tests the 'transform' option which modifies bitmap persistence on
+    migration.
     """

     src_node_name = 'node-a'
@@ -674,7 +675,8 @@ class TestAliasTransformMigration(TestDirtyBitmapMigration):
         bitmaps = self.vm_b.query_bitmaps()

         for node in bitmaps:
-            bitmaps[node] = sorted(((bmap['name'], bmap['persistent']) for bmap in bitmaps[node]))
+            bitmaps[node] = sorted(((bmap['name'], bmap['persistent'])
+                                    for bmap in bitmaps[node]))

         self.assertEqual(bitmaps,
                          {'node-a': [('bmap-a', True), ('bmap-b', False)],
-- 
2.30.1

Re: [PATCH] iotests: Fix up python style in 300

[John Snow]

On 2/15/21 5:05 PM, Eric Blake wrote:
> Break some long lines, and relax our type hints to be more generic to
> any JSON, in order to more easily permit the additional JSON depth now
> possible in migration parameters.  Detected by iotest 297.
> 
> Fixes: ca4bfec41d56
>   (qemu-iotests: 300: Add test case for modifying persistence of bitmap)
> Reported-by: Kevin Wolf <kwolf@redhat.com>
> Signed-off-by: Eric Blake <eblake@redhat.com>

Reviewed-by: John Snow <jsnow@redhat.com>

> ---
>   tests/qemu-iotests/300 | 10 ++++++----
>   1 file changed, 6 insertions(+), 4 deletions(-)
> 
> diff --git a/tests/qemu-iotests/300 b/tests/qemu-iotests/300
> index 63036f6a6e13..adb927629747 100755
> --- a/tests/qemu-iotests/300
> +++ b/tests/qemu-iotests/300
> @@ -22,7 +22,7 @@
>   import os
>   import random
>   import re
> -from typing import Dict, List, Optional, Union
> +from typing import Dict, List, Optional
> 
>   import iotests
> 
> @@ -30,7 +30,7 @@ import iotests
>   # pylint: disable=wrong-import-order
>   import qemu
> 
> -BlockBitmapMapping = List[Dict[str, Union[str, List[Dict[str, str]]]]]
> +BlockBitmapMapping = List[Dict[str, object]]
> 

Assuming iotest 297 didn't yap about this, I think this has the 
necessary power for this file and we don't have to work any harder.

If in the future you try to treat e.g. bmap['persistent'] as a 
particular kind of value (string? bool? int?) mypy will likely complain 
about that a little, saying it has no insight into the type beyond "object".

If *that* becomes annoying, you can degrade this type to use 'Any' 
instead of 'object' and even those checks will cease.

>   mig_sock = os.path.join(iotests.sock_dir, 'mig_sock')
> 
> @@ -602,7 +602,8 @@ class TestCrossAliasMigration(TestDirtyBitmapMigration):
> 
>   class TestAliasTransformMigration(TestDirtyBitmapMigration):
>       """
> -    Tests the 'transform' option which modifies bitmap persistence on migration.
> +    Tests the 'transform' option which modifies bitmap persistence on
> +    migration.
>       """
> 
>       src_node_name = 'node-a'
> @@ -674,7 +675,8 @@ class TestAliasTransformMigration(TestDirtyBitmapMigration):
>           bitmaps = self.vm_b.query_bitmaps()
> 
>           for node in bitmaps:
> -            bitmaps[node] = sorted(((bmap['name'], bmap['persistent']) for bmap in bitmaps[node]))
> +            bitmaps[node] = sorted(((bmap['name'], bmap['persistent'])
> +                                    for bmap in bitmaps[node]))
> 
>           self.assertEqual(bitmaps,
>                            {'node-a': [('bmap-a', True), ('bmap-b', False)],
> 

Re: [PATCH] iotests: Fix up python style in 300

[Vladimir Sementsov-Ogievskiy]

16.02.2021 01:05, Eric Blake wrote:
> Break some long lines, and relax our type hints to be more generic to
> any JSON, in order to more easily permit the additional JSON depth now
> possible in migration parameters.  Detected by iotest 297.
> 
> Fixes: ca4bfec41d56
>   (qemu-iotests: 300: Add test case for modifying persistence of bitmap)
> Reported-by: Kevin Wolf<kwolf@redhat.com>
> Signed-off-by: Eric Blake<eblake@redhat.com>


Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

-- 
Best regards,
Vladimir

Re: [PATCH] iotests: Fix up python style in 300

[Vladimir Sementsov-Ogievskiy]

16.02.2021 02:21, John Snow wrote:
> On 2/15/21 5:05 PM, Eric Blake wrote:
>> Break some long lines, and relax our type hints to be more generic to
>> any JSON, in order to more easily permit the additional JSON depth now
>> possible in migration parameters.  Detected by iotest 297.
>>
>> Fixes: ca4bfec41d56
>>   (qemu-iotests: 300: Add test case for modifying persistence of bitmap)
>> Reported-by: Kevin Wolf <kwolf@redhat.com>
>> Signed-off-by: Eric Blake <eblake@redhat.com>
> 
> Reviewed-by: John Snow <jsnow@redhat.com>
> 
>> ---
>>   tests/qemu-iotests/300 | 10 ++++++----
>>   1 file changed, 6 insertions(+), 4 deletions(-)
>>
>> diff --git a/tests/qemu-iotests/300 b/tests/qemu-iotests/300
>> index 63036f6a6e13..adb927629747 100755
>> --- a/tests/qemu-iotests/300
>> +++ b/tests/qemu-iotests/300
>> @@ -22,7 +22,7 @@
>>   import os
>>   import random
>>   import re
>> -from typing import Dict, List, Optional, Union
>> +from typing import Dict, List, Optional
>>
>>   import iotests
>>
>> @@ -30,7 +30,7 @@ import iotests
>>   # pylint: disable=wrong-import-order
>>   import qemu
>>
>> -BlockBitmapMapping = List[Dict[str, Union[str, List[Dict[str, str]]]]]
>> +BlockBitmapMapping = List[Dict[str, object]]
>>
> 
> Assuming iotest 297 didn't yap about this, I think this has the necessary power for this file and we don't have to work any harder.
> 
> If in the future you try to treat e.g. bmap['persistent'] as a particular kind of value (string? bool? int?) mypy will likely complain about that a little, saying it has no insight into the type beyond "object".
> 
> If *that* becomes annoying, you can degrade this type to use 'Any' instead of 'object' and even those checks will cease.

Probably at some future moment we'll have generated python types for QAPI structures ? :)

> 
>>   mig_sock = os.path.join(iotests.sock_dir, 'mig_sock')
>>
>> @@ -602,7 +602,8 @@ class TestCrossAliasMigration(TestDirtyBitmapMigration):
>>
>>   class TestAliasTransformMigration(TestDirtyBitmapMigration):
>>       """
>> -    Tests the 'transform' option which modifies bitmap persistence on migration.
>> +    Tests the 'transform' option which modifies bitmap persistence on
>> +    migration.
>>       """
>>
>>       src_node_name = 'node-a'
>> @@ -674,7 +675,8 @@ class TestAliasTransformMigration(TestDirtyBitmapMigration):
>>           bitmaps = self.vm_b.query_bitmaps()
>>
>>           for node in bitmaps:
>> -            bitmaps[node] = sorted(((bmap['name'], bmap['persistent']) for bmap in bitmaps[node]))
>> +            bitmaps[node] = sorted(((bmap['name'], bmap['persistent'])
>> +                                    for bmap in bitmaps[node]))
>>
>>           self.assertEqual(bitmaps,
>>                            {'node-a': [('bmap-a', True), ('bmap-b', False)],
>>
> 
> 


-- 
Best regards,
Vladimir

Re: [PATCH] iotests: Fix up python style in 300

[Markus Armbruster]

Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> writes:

> 16.02.2021 02:21, John Snow wrote:
>> On 2/15/21 5:05 PM, Eric Blake wrote:
>>> Break some long lines, and relax our type hints to be more generic to
>>> any JSON, in order to more easily permit the additional JSON depth now
>>> possible in migration parameters.  Detected by iotest 297.
>>>
>>> Fixes: ca4bfec41d56
>>>   (qemu-iotests: 300: Add test case for modifying persistence of bitmap)
>>> Reported-by: Kevin Wolf <kwolf@redhat.com>
>>> Signed-off-by: Eric Blake <eblake@redhat.com>
>> Reviewed-by: John Snow <jsnow@redhat.com>
>> 
>>> ---
>>>   tests/qemu-iotests/300 | 10 ++++++----
>>>   1 file changed, 6 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/tests/qemu-iotests/300 b/tests/qemu-iotests/300
>>> index 63036f6a6e13..adb927629747 100755
>>> --- a/tests/qemu-iotests/300
>>> +++ b/tests/qemu-iotests/300
>>> @@ -22,7 +22,7 @@
>>>   import os
>>>   import random
>>>   import re
>>> -from typing import Dict, List, Optional, Union
>>> +from typing import Dict, List, Optional
>>>
>>>   import iotests
>>>
>>> @@ -30,7 +30,7 @@ import iotests
>>>   # pylint: disable=wrong-import-order
>>>   import qemu
>>>
>>> -BlockBitmapMapping = List[Dict[str, Union[str, List[Dict[str, str]]]]]
>>> +BlockBitmapMapping = List[Dict[str, object]]
>>>
>> Assuming iotest 297 didn't yap about this, I think this has the
>> necessary power for this file and we don't have to work any harder.
>> If in the future you try to treat e.g. bmap['persistent'] as a
>> particular kind of value (string? bool? int?) mypy will likely
>> complain about that a little, saying it has no insight into the type
>> beyond "object".
>> If *that* becomes annoying, you can degrade this type to use 'Any'
>> instead of 'object' and even those checks will cease.
>
> Probably at some future moment we'll have generated python types for QAPI structures ? :)

Generating Python from the QAPI schema is possible.  I'm not aware of
anyone planning to work on it near term.

Re: [PATCH] iotests: Fix up python style in 300

[Kevin Wolf]

Am 15.02.2021 um 23:05 hat Eric Blake geschrieben:
> Break some long lines, and relax our type hints to be more generic to
> any JSON, in order to more easily permit the additional JSON depth now
> possible in migration parameters.  Detected by iotest 297.
> 
> Fixes: ca4bfec41d56
>  (qemu-iotests: 300: Add test case for modifying persistence of bitmap)
> Reported-by: Kevin Wolf <kwolf@redhat.com>
> Signed-off-by: Eric Blake <eblake@redhat.com>

Thanks, applied to the block branch.

Kevin

Re: [PATCH] iotests: Fix up python style in 300

[John Snow]

On 2/26/21 2:04 AM, Vladimir Sementsov-Ogievskiy wrote:
> 16.02.2021 02:21, John Snow wrote:
>> On 2/15/21 5:05 PM, Eric Blake wrote:
>>> Break some long lines, and relax our type hints to be more generic to
>>> any JSON, in order to more easily permit the additional JSON depth now
>>> possible in migration parameters.  Detected by iotest 297.
>>>
>>> Fixes: ca4bfec41d56
>>>   (qemu-iotests: 300: Add test case for modifying persistence of bitmap)
>>> Reported-by: Kevin Wolf <kwolf@redhat.com>
>>> Signed-off-by: Eric Blake <eblake@redhat.com>
>>
>> Reviewed-by: John Snow <jsnow@redhat.com>
>>
>>> ---
>>>   tests/qemu-iotests/300 | 10 ++++++----
>>>   1 file changed, 6 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/tests/qemu-iotests/300 b/tests/qemu-iotests/300
>>> index 63036f6a6e13..adb927629747 100755
>>> --- a/tests/qemu-iotests/300
>>> +++ b/tests/qemu-iotests/300
>>> @@ -22,7 +22,7 @@
>>>   import os
>>>   import random
>>>   import re
>>> -from typing import Dict, List, Optional, Union
>>> +from typing import Dict, List, Optional
>>>
>>>   import iotests
>>>
>>> @@ -30,7 +30,7 @@ import iotests
>>>   # pylint: disable=wrong-import-order
>>>   import qemu
>>>
>>> -BlockBitmapMapping = List[Dict[str, Union[str, List[Dict[str, str]]]]]
>>> +BlockBitmapMapping = List[Dict[str, object]]
>>>
>>
>> Assuming iotest 297 didn't yap about this, I think this has the 
>> necessary power for this file and we don't have to work any harder.
>>
>> If in the future you try to treat e.g. bmap['persistent'] as a 
>> particular kind of value (string? bool? int?) mypy will likely 
>> complain about that a little, saying it has no insight into the type 
>> beyond "object".
>>
>> If *that* becomes annoying, you can degrade this type to use 'Any' 
>> instead of 'object' and even those checks will cease.
> 
> Probably at some future moment we'll have generated python types for 
> QAPI structures ? :)
> 

That's my hope, yes!

Typing the QAPI generator is something I see as a step to doing this so 
that we can safely work on the QAPI generator a bit more vigorously.

Marc-Andre is adding rust backends, I'd like to add either a Python or a 
JSON-Schema backend to help generate a fully typed SDK for us in Python.

I don't know how suitable those tools will be to use in the test suite; 
I suspect that every last build of QEMU from the development tree will 
have to possibly re-generate such a Python module.

When I get a little closer to a prototype for this I will try to 
announce it. In the meantime I am very fastidiously trying to strictly 
type the QAPI generator and move it to ./python/qemu/qapi.

--js