* [Qemu-devel] [RFC PATCH qemu] qapi: Add query-memory-checksum
@ 2019-08-22 1:16 Alexey Kardashevskiy
2019-08-22 1:33 ` Eric Blake
2019-08-22 14:16 ` Markus Armbruster
0 siblings, 2 replies; 9+ messages in thread
From: Alexey Kardashevskiy @ 2019-08-22 1:16 UTC (permalink / raw)
To: qemu-devel; +Cc: Alexey Kardashevskiy, Paolo Bonzini
This returns MD5 checksum of all RAM blocks for migration debugging
as this is way faster than saving the entire RAM to a file and checking
that.
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
I am actually wondering if there is an easier way of getting these
checksums and I just do not see it, it cannot be that we fixed all
memory migration bugs :)
---
qapi/misc.json | 27 +++++++++++++++++++++++++++
include/exec/cpu-common.h | 1 +
exec.c | 16 ++++++++++++++++
monitor/qmp-cmds.c | 9 +++++++++
4 files changed, 53 insertions(+)
diff --git a/qapi/misc.json b/qapi/misc.json
index a7fba7230cfa..e7475f30a844 100644
--- a/qapi/misc.json
+++ b/qapi/misc.json
@@ -1194,6 +1194,33 @@
##
{ 'command': 'query-memory-size-summary', 'returns': 'MemoryInfo' }
+##
+# @MemoryChecksum:
+#
+# A string with MD5 checksum of all RAMBlocks.
+#
+# @checksum: the checksum.
+#
+# Since: 3.2.0
+##
+{ 'struct': 'MemoryChecksum',
+ 'data' : { 'checksum': 'str' } }
+
+##
+# @query-memory-checksum:
+#
+# Return the MD5 checksum of all RAMBlocks.
+#
+# Example:
+#
+# -> { "execute": "query-memory-checksum" }
+# <- { "return": { "checksum": "a0880304994f64cb2edad77b9a1cd58f" } }
+#
+# Since: 3.2.0
+##
+{ 'command': 'query-memory-checksum',
+ 'returns': 'MemoryChecksum' }
+
##
# @AddfdInfo:
diff --git a/include/exec/cpu-common.h b/include/exec/cpu-common.h
index f7dbe75fbc38..15dbf18c2d5d 100644
--- a/include/exec/cpu-common.h
+++ b/include/exec/cpu-common.h
@@ -57,6 +57,7 @@ void qemu_ram_set_idstr(RAMBlock *block, const char *name, DeviceState *dev);
void qemu_ram_unset_idstr(RAMBlock *block);
const char *qemu_ram_get_idstr(RAMBlock *rb);
void *qemu_ram_get_host_addr(RAMBlock *rb);
+gchar *qemu_ram_chksum(void);
ram_addr_t qemu_ram_get_offset(RAMBlock *rb);
ram_addr_t qemu_ram_get_used_length(RAMBlock *rb);
bool qemu_ram_is_shared(RAMBlock *rb);
diff --git a/exec.c b/exec.c
index 3e78de3b8f8b..76f7f63cf71b 100644
--- a/exec.c
+++ b/exec.c
@@ -2050,6 +2050,22 @@ void *qemu_ram_get_host_addr(RAMBlock *rb)
return rb->host;
}
+gchar *qemu_ram_chksum(void)
+{
+ struct RAMBlock *rb;
+ GChecksum *chksum = g_checksum_new(G_CHECKSUM_MD5);
+ gchar *ret;
+
+ RAMBLOCK_FOREACH(rb) {
+ g_checksum_update(chksum, qemu_ram_get_host_addr(rb),
+ qemu_ram_get_used_length(rb));
+ }
+ ret = g_strdup(g_checksum_get_string(chksum));
+ g_checksum_free(chksum);
+
+ return ret;
+}
+
ram_addr_t qemu_ram_get_offset(RAMBlock *rb)
{
return rb->offset;
diff --git a/monitor/qmp-cmds.c b/monitor/qmp-cmds.c
index b9ae40eec751..ec52bd82588e 100644
--- a/monitor/qmp-cmds.c
+++ b/monitor/qmp-cmds.c
@@ -413,3 +413,12 @@ MemoryInfo *qmp_query_memory_size_summary(Error **errp)
return mem_info;
}
+
+MemoryChecksum *qmp_query_memory_checksum(Error **errp)
+{
+ MemoryChecksum *chk = g_malloc0(sizeof(MemoryChecksum));
+
+ chk->checksum = qemu_ram_chksum();
+
+ return chk;
+}
--
2.17.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [RFC PATCH qemu] qapi: Add query-memory-checksum
2019-08-22 1:16 [Qemu-devel] [RFC PATCH qemu] qapi: Add query-memory-checksum Alexey Kardashevskiy
@ 2019-08-22 1:33 ` Eric Blake
2019-08-22 3:20 ` Alexey Kardashevskiy
2019-08-22 14:16 ` Markus Armbruster
1 sibling, 1 reply; 9+ messages in thread
From: Eric Blake @ 2019-08-22 1:33 UTC (permalink / raw)
To: Alexey Kardashevskiy, qemu-devel; +Cc: Paolo Bonzini
[-- Attachment #1.1: Type: text/plain, Size: 2200 bytes --]
On 8/21/19 8:16 PM, Alexey Kardashevskiy wrote:
> This returns MD5 checksum of all RAM blocks for migration debugging
> as this is way faster than saving the entire RAM to a file and checking
> that.
>
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> ---
>
>
> I am actually wondering if there is an easier way of getting these
> checksums and I just do not see it, it cannot be that we fixed all
> memory migration bugs :)
I'm not sure whether the command itself makes sense, but for the interface:
> +++ b/qapi/misc.json
> @@ -1194,6 +1194,33 @@
> ##
> { 'command': 'query-memory-size-summary', 'returns': 'MemoryInfo' }
>
> +##
> +# @MemoryChecksum:
> +#
> +# A string with MD5 checksum of all RAMBlocks.
> +#
> +# @checksum: the checksum.
> +#
> +# Since: 3.2.0
This should be 4.2, not 3.2.
> +##
> +{ 'struct': 'MemoryChecksum',
> + 'data' : { 'checksum': 'str' } }
> +
> +##
> +# @query-memory-checksum:
> +#
> +# Return the MD5 checksum of all RAMBlocks.
> +#
> +# Example:
> +#
> +# -> { "execute": "query-memory-checksum" }
> +# <- { "return": { "checksum": "a0880304994f64cb2edad77b9a1cd58f" } }
> +#
> +# Since: 3.2.0
and again
> +##
> +{ 'command': 'query-memory-checksum',
> + 'returns': 'MemoryChecksum' }
> +
>
> +++ b/exec.c
> @@ -2050,6 +2050,22 @@ void *qemu_ram_get_host_addr(RAMBlock *rb)
> return rb->host;
> }
>
> +gchar *qemu_ram_chksum(void)
gchar is a pointless glib type. Use 'char' instead.
> +{
> + struct RAMBlock *rb;
> + GChecksum *chksum = g_checksum_new(G_CHECKSUM_MD5);
> + gchar *ret;
> +
> + RAMBLOCK_FOREACH(rb) {
> + g_checksum_update(chksum, qemu_ram_get_host_addr(rb),
> + qemu_ram_get_used_length(rb));
> + }
> + ret = g_strdup(g_checksum_get_string(chksum));
> + g_checksum_free(chksum);
> +
> + return ret;
> +}
How long does this take to run? Is it something where you really want
to block the guest while chewing over the guest's entire memory?
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [RFC PATCH qemu] qapi: Add query-memory-checksum
2019-08-22 1:33 ` Eric Blake
@ 2019-08-22 3:20 ` Alexey Kardashevskiy
0 siblings, 0 replies; 9+ messages in thread
From: Alexey Kardashevskiy @ 2019-08-22 3:20 UTC (permalink / raw)
To: Eric Blake, qemu-devel; +Cc: Paolo Bonzini
On 22/08/2019 11:33, Eric Blake wrote:
> On 8/21/19 8:16 PM, Alexey Kardashevskiy wrote:
>> This returns MD5 checksum of all RAM blocks for migration debugging
>> as this is way faster than saving the entire RAM to a file and checking
>> that.
>>
>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>> ---
>>
>>
>> I am actually wondering if there is an easier way of getting these
>> checksums and I just do not see it, it cannot be that we fixed all
>> memory migration bugs :)
>
> I'm not sure whether the command itself makes sense, but for the interface:
>
>
>> +++ b/qapi/misc.json
>> @@ -1194,6 +1194,33 @@
>> ##
>> { 'command': 'query-memory-size-summary', 'returns': 'MemoryInfo' }
>>
>> +##
>> +# @MemoryChecksum:
>> +#
>> +# A string with MD5 checksum of all RAMBlocks.
>> +#
>> +# @checksum: the checksum.
>> +#
>> +# Since: 3.2.0
>
> This should be 4.2, not 3.2.
>
>> +##
>> +{ 'struct': 'MemoryChecksum',
>> + 'data' : { 'checksum': 'str' } }
>> +
>> +##
>> +# @query-memory-checksum:
>> +#
>> +# Return the MD5 checksum of all RAMBlocks.
>> +#
>> +# Example:
>> +#
>> +# -> { "execute": "query-memory-checksum" }
>> +# <- { "return": { "checksum": "a0880304994f64cb2edad77b9a1cd58f" } }
>> +#
>> +# Since: 3.2.0
>
> and again
>
>> +##
>> +{ 'command': 'query-memory-checksum',
>> + 'returns': 'MemoryChecksum' }
>> +
>>
>
>> +++ b/exec.c
>> @@ -2050,6 +2050,22 @@ void *qemu_ram_get_host_addr(RAMBlock *rb)
>> return rb->host;
>> }
>>
>> +gchar *qemu_ram_chksum(void)
>
> gchar is a pointless glib type. Use 'char' instead.
>
>> +{
>> + struct RAMBlock *rb;
>> + GChecksum *chksum = g_checksum_new(G_CHECKSUM_MD5);
>> + gchar *ret;
>> +
>> + RAMBLOCK_FOREACH(rb) {
>> + g_checksum_update(chksum, qemu_ram_get_host_addr(rb),
>> + qemu_ram_get_used_length(rb));
>> + }
>> + ret = g_strdup(g_checksum_get_string(chksum));
>> + g_checksum_free(chksum);
>> +
>> + return ret;
>> +}
>
> How long does this take to run? Is it something where you really want
> to block the guest while chewing over the guest's entire memory?
10-20 times faster than "pmemsave" and blocking the guest is not a
problem here as both - source and destination - guests are stopped
(otherwise the checksum does not make sense).
--
Alexey
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [RFC PATCH qemu] qapi: Add query-memory-checksum
2019-08-22 1:16 [Qemu-devel] [RFC PATCH qemu] qapi: Add query-memory-checksum Alexey Kardashevskiy
2019-08-22 1:33 ` Eric Blake
@ 2019-08-22 14:16 ` Markus Armbruster
2019-08-22 14:49 ` Daniel P. Berrangé
1 sibling, 1 reply; 9+ messages in thread
From: Markus Armbruster @ 2019-08-22 14:16 UTC (permalink / raw)
To: Alexey Kardashevskiy; +Cc: Paolo Bonzini, qemu-devel
Alexey Kardashevskiy <aik@ozlabs.ru> writes:
> This returns MD5 checksum of all RAM blocks for migration debugging
> as this is way faster than saving the entire RAM to a file and checking
> that.
>
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Any particular reason for MD5? Have you measured the other choices
offered by GLib?
I understand you don't need crypto-strength here. Both MD5 and SHA-1
would be bad choices then.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [RFC PATCH qemu] qapi: Add query-memory-checksum
2019-08-22 14:16 ` Markus Armbruster
@ 2019-08-22 14:49 ` Daniel P. Berrangé
2019-08-23 5:49 ` Markus Armbruster
0 siblings, 1 reply; 9+ messages in thread
From: Daniel P. Berrangé @ 2019-08-22 14:49 UTC (permalink / raw)
To: Markus Armbruster; +Cc: Alexey Kardashevskiy, Paolo Bonzini, qemu-devel
On Thu, Aug 22, 2019 at 04:16:53PM +0200, Markus Armbruster wrote:
> Alexey Kardashevskiy <aik@ozlabs.ru> writes:
>
> > This returns MD5 checksum of all RAM blocks for migration debugging
> > as this is way faster than saving the entire RAM to a file and checking
> > that.
> >
> > Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>
> Any particular reason for MD5? Have you measured the other choices
> offered by GLib?
>
> I understand you don't need crypto-strength here. Both MD5 and SHA-1
> would be bad choices then.
We have a tests/bench-crypto-hash test but its hardcoded for sha256.
I hacked it to report all algorithms and got these results for varying
input chunk sizes:
/crypto/hash/md5/speed-512: 519.12 MB/sec OK
/crypto/hash/md5/speed-1024: 560.39 MB/sec OK
/crypto/hash/md5/speed-4096: 591.39 MB/sec OK
/crypto/hash/md5/speed-16384: 576.46 MB/sec OK
/crypto/hash/sha1/speed-512: 443.12 MB/sec OK
/crypto/hash/sha1/speed-1024: 518.82 MB/sec OK
/crypto/hash/sha1/speed-4096: 555.60 MB/sec OK
/crypto/hash/sha1/speed-16384: 568.16 MB/sec OK
/crypto/hash/sha224/speed-512: 221.90 MB/sec OK
/crypto/hash/sha224/speed-1024: 239.79 MB/sec OK
/crypto/hash/sha224/speed-4096: 269.37 MB/sec OK
/crypto/hash/sha224/speed-16384: 274.87 MB/sec OK
/crypto/hash/sha256/speed-512: 222.75 MB/sec OK
/crypto/hash/sha256/speed-1024: 253.25 MB/sec OK
/crypto/hash/sha256/speed-4096: 272.80 MB/sec OK
/crypto/hash/sha256/speed-16384: 275.59 MB/sec OK
/crypto/hash/sha384/speed-512: 322.73 MB/sec OK
/crypto/hash/sha384/speed-1024: 369.84 MB/sec OK
/crypto/hash/sha384/speed-4096: 406.71 MB/sec OK
/crypto/hash/sha384/speed-16384: 417.87 MB/sec OK
/crypto/hash/sha512/speed-512: 320.62 MB/sec OK
/crypto/hash/sha512/speed-1024: 361.93 MB/sec OK
/crypto/hash/sha512/speed-4096: 404.91 MB/sec OK
/crypto/hash/sha512/speed-16384: 418.53 MB/sec OK
/crypto/hash/ripemd160/speed-512: 226.45 MB/sec OK
/crypto/hash/ripemd160/speed-1024: 239.25 MB/sec OK
/crypto/hash/ripemd160/speed-4096: 251.31 MB/sec OK
/crypto/hash/ripemd160/speed-16384: 255.01 MB/sec OK
IOW, md5 is clearly the quickest, by a considerable margin over
SHA256/512. SHA1 is slightly slower.
Assuming that we document that this command is intentionally
*not* trying to guarantee collision resistances we're ok.
In fact we should not document what kind of checksum is
reported by query-memory-checksum. The impl should be a black
box from user's POV.
If we're just aiming for debugging tool to detect accidental
corruption, could we even just ignore cryptographic hashs
entirely and do a crc32 - that'd be way faster than even
md5.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [RFC PATCH qemu] qapi: Add query-memory-checksum
2019-08-22 14:49 ` Daniel P. Berrangé
@ 2019-08-23 5:49 ` Markus Armbruster
2019-08-23 9:23 ` Daniel P. Berrangé
0 siblings, 1 reply; 9+ messages in thread
From: Markus Armbruster @ 2019-08-23 5:49 UTC (permalink / raw)
To: Daniel P. Berrangé; +Cc: Alexey Kardashevskiy, Paolo Bonzini, qemu-devel
Daniel P. Berrangé <berrange@redhat.com> writes:
> On Thu, Aug 22, 2019 at 04:16:53PM +0200, Markus Armbruster wrote:
>> Alexey Kardashevskiy <aik@ozlabs.ru> writes:
>>
>> > This returns MD5 checksum of all RAM blocks for migration debugging
>> > as this is way faster than saving the entire RAM to a file and checking
>> > that.
>> >
>> > Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>>
>> Any particular reason for MD5? Have you measured the other choices
>> offered by GLib?
>>
>> I understand you don't need crypto-strength here. Both MD5 and SHA-1
>> would be bad choices then.
>
> We have a tests/bench-crypto-hash test but its hardcoded for sha256.
> I hacked it to report all algorithms and got these results for varying
> input chunk sizes:
>
> /crypto/hash/md5/speed-512: 519.12 MB/sec OK
> /crypto/hash/md5/speed-1024: 560.39 MB/sec OK
> /crypto/hash/md5/speed-4096: 591.39 MB/sec OK
> /crypto/hash/md5/speed-16384: 576.46 MB/sec OK
> /crypto/hash/sha1/speed-512: 443.12 MB/sec OK
> /crypto/hash/sha1/speed-1024: 518.82 MB/sec OK
> /crypto/hash/sha1/speed-4096: 555.60 MB/sec OK
> /crypto/hash/sha1/speed-16384: 568.16 MB/sec OK
> /crypto/hash/sha224/speed-512: 221.90 MB/sec OK
> /crypto/hash/sha224/speed-1024: 239.79 MB/sec OK
> /crypto/hash/sha224/speed-4096: 269.37 MB/sec OK
> /crypto/hash/sha224/speed-16384: 274.87 MB/sec OK
> /crypto/hash/sha256/speed-512: 222.75 MB/sec OK
> /crypto/hash/sha256/speed-1024: 253.25 MB/sec OK
> /crypto/hash/sha256/speed-4096: 272.80 MB/sec OK
> /crypto/hash/sha256/speed-16384: 275.59 MB/sec OK
> /crypto/hash/sha384/speed-512: 322.73 MB/sec OK
> /crypto/hash/sha384/speed-1024: 369.84 MB/sec OK
> /crypto/hash/sha384/speed-4096: 406.71 MB/sec OK
> /crypto/hash/sha384/speed-16384: 417.87 MB/sec OK
> /crypto/hash/sha512/speed-512: 320.62 MB/sec OK
> /crypto/hash/sha512/speed-1024: 361.93 MB/sec OK
> /crypto/hash/sha512/speed-4096: 404.91 MB/sec OK
> /crypto/hash/sha512/speed-16384: 418.53 MB/sec OK
> /crypto/hash/ripemd160/speed-512: 226.45 MB/sec OK
> /crypto/hash/ripemd160/speed-1024: 239.25 MB/sec OK
> /crypto/hash/ripemd160/speed-4096: 251.31 MB/sec OK
> /crypto/hash/ripemd160/speed-16384: 255.01 MB/sec OK
>
>
> IOW, md5 is clearly the quickest, by a considerable margin over
> SHA256/512. SHA1 is slightly slower.
>
> Assuming that we document that this command is intentionally
> *not* trying to guarantee collision resistances we're ok.
>
> In fact we should not document what kind of checksum is
> reported by query-memory-checksum. The impl should be a black
> box from user's POV.
>
> If we're just aiming for debugging tool to detect accidental
> corruption, could we even just ignore cryptographic hashs
> entirely and do a crc32 - that'd be way faster than even
> md5.
Good points.
The doc strings should spell out "for debugging", like the commit
message does, and both should spell out "weak collision resistance".
I can't find CRC-32 in GLib, but zlib appears to provide it:
http://refspecs.linuxbase.org/LSB_3.0.0/LSB-Core-generic/LSB-Core-generic/zlib-crc32-1.html
Care to compare its speed to MD5?
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [RFC PATCH qemu] qapi: Add query-memory-checksum
2019-08-23 5:49 ` Markus Armbruster
@ 2019-08-23 9:23 ` Daniel P. Berrangé
2019-08-23 11:41 ` Markus Armbruster
0 siblings, 1 reply; 9+ messages in thread
From: Daniel P. Berrangé @ 2019-08-23 9:23 UTC (permalink / raw)
To: Markus Armbruster; +Cc: Alexey Kardashevskiy, Paolo Bonzini, qemu-devel
On Fri, Aug 23, 2019 at 07:49:31AM +0200, Markus Armbruster wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
>
> > On Thu, Aug 22, 2019 at 04:16:53PM +0200, Markus Armbruster wrote:
> >> Alexey Kardashevskiy <aik@ozlabs.ru> writes:
> >>
> >> > This returns MD5 checksum of all RAM blocks for migration debugging
> >> > as this is way faster than saving the entire RAM to a file and checking
> >> > that.
> >> >
> >> > Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> >>
> >> Any particular reason for MD5? Have you measured the other choices
> >> offered by GLib?
> >>
> >> I understand you don't need crypto-strength here. Both MD5 and SHA-1
> >> would be bad choices then.
> >
> > We have a tests/bench-crypto-hash test but its hardcoded for sha256.
> > I hacked it to report all algorithms and got these results for varying
> > input chunk sizes:
> >
> > /crypto/hash/md5/speed-512: 519.12 MB/sec OK
> > /crypto/hash/md5/speed-1024: 560.39 MB/sec OK
> > /crypto/hash/md5/speed-4096: 591.39 MB/sec OK
> > /crypto/hash/md5/speed-16384: 576.46 MB/sec OK
> > /crypto/hash/sha1/speed-512: 443.12 MB/sec OK
> > /crypto/hash/sha1/speed-1024: 518.82 MB/sec OK
> > /crypto/hash/sha1/speed-4096: 555.60 MB/sec OK
> > /crypto/hash/sha1/speed-16384: 568.16 MB/sec OK
> > /crypto/hash/sha224/speed-512: 221.90 MB/sec OK
> > /crypto/hash/sha224/speed-1024: 239.79 MB/sec OK
> > /crypto/hash/sha224/speed-4096: 269.37 MB/sec OK
> > /crypto/hash/sha224/speed-16384: 274.87 MB/sec OK
> > /crypto/hash/sha256/speed-512: 222.75 MB/sec OK
> > /crypto/hash/sha256/speed-1024: 253.25 MB/sec OK
> > /crypto/hash/sha256/speed-4096: 272.80 MB/sec OK
> > /crypto/hash/sha256/speed-16384: 275.59 MB/sec OK
> > /crypto/hash/sha384/speed-512: 322.73 MB/sec OK
> > /crypto/hash/sha384/speed-1024: 369.84 MB/sec OK
> > /crypto/hash/sha384/speed-4096: 406.71 MB/sec OK
> > /crypto/hash/sha384/speed-16384: 417.87 MB/sec OK
> > /crypto/hash/sha512/speed-512: 320.62 MB/sec OK
> > /crypto/hash/sha512/speed-1024: 361.93 MB/sec OK
> > /crypto/hash/sha512/speed-4096: 404.91 MB/sec OK
> > /crypto/hash/sha512/speed-16384: 418.53 MB/sec OK
> > /crypto/hash/ripemd160/speed-512: 226.45 MB/sec OK
> > /crypto/hash/ripemd160/speed-1024: 239.25 MB/sec OK
> > /crypto/hash/ripemd160/speed-4096: 251.31 MB/sec OK
> > /crypto/hash/ripemd160/speed-16384: 255.01 MB/sec OK
> >
> >
> > IOW, md5 is clearly the quickest, by a considerable margin over
> > SHA256/512. SHA1 is slightly slower.
> >
> > Assuming that we document that this command is intentionally
> > *not* trying to guarantee collision resistances we're ok.
> >
> > In fact we should not document what kind of checksum is
> > reported by query-memory-checksum. The impl should be a black
> > box from user's POV.
> >
> > If we're just aiming for debugging tool to detect accidental
> > corruption, could we even just ignore cryptographic hashs
> > entirely and do a crc32 - that'd be way faster than even
> > md5.
>
> Good points.
>
> The doc strings should spell out "for debugging", like the commit
> message does, and both should spell out "weak collision resistance".
>
> I can't find CRC-32 in GLib, but zlib appears to provide it:
> http://refspecs.linuxbase.org/LSB_3.0.0/LSB-Core-generic/LSB-Core-generic/zlib-crc32-1.html
>
> Care to compare its speed to MD5?
I hacked the code to use zlib's crc32 impl and got these for comparison:
/crypto/hash/crc32/speed-512: 1089.18 MB/sec OK
/crypto/hash/crc32/speed-1024: 1124.63 MB/sec OK
/crypto/hash/crc32/speed-4096: 1162.73 MB/sec OK
/crypto/hash/crc32/speed-16384: 1171.58 MB/sec OK
/crypto/hash/crc32/speed-1048576: 1165.68 MB/sec OK
/crypto/hash/md5/speed-512: 476.27 MB/sec OK
/crypto/hash/md5/speed-1024: 517.16 MB/sec OK
/crypto/hash/md5/speed-4096: 554.70 MB/sec OK
/crypto/hash/md5/speed-16384: 564.44 MB/sec OK
/crypto/hash/md5/speed-1048576: 566.78 MB/sec OK
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [RFC PATCH qemu] qapi: Add query-memory-checksum
2019-08-23 9:23 ` Daniel P. Berrangé
@ 2019-08-23 11:41 ` Markus Armbruster
2019-08-26 4:28 ` Alexey Kardashevskiy
0 siblings, 1 reply; 9+ messages in thread
From: Markus Armbruster @ 2019-08-23 11:41 UTC (permalink / raw)
To: Daniel P. Berrangé; +Cc: Alexey Kardashevskiy, Paolo Bonzini, qemu-devel
Daniel P. Berrangé <berrange@redhat.com> writes:
> On Fri, Aug 23, 2019 at 07:49:31AM +0200, Markus Armbruster wrote:
>> Daniel P. Berrangé <berrange@redhat.com> writes:
>>
>> > On Thu, Aug 22, 2019 at 04:16:53PM +0200, Markus Armbruster wrote:
>> >> Alexey Kardashevskiy <aik@ozlabs.ru> writes:
>> >>
>> >> > This returns MD5 checksum of all RAM blocks for migration debugging
>> >> > as this is way faster than saving the entire RAM to a file and checking
>> >> > that.
>> >> >
>> >> > Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>> >>
>> >> Any particular reason for MD5? Have you measured the other choices
>> >> offered by GLib?
>> >>
>> >> I understand you don't need crypto-strength here. Both MD5 and SHA-1
>> >> would be bad choices then.
>> >
>> > We have a tests/bench-crypto-hash test but its hardcoded for sha256.
>> > I hacked it to report all algorithms and got these results for varying
>> > input chunk sizes:
>> >
>> > /crypto/hash/md5/speed-512: 519.12 MB/sec OK
>> > /crypto/hash/md5/speed-1024: 560.39 MB/sec OK
>> > /crypto/hash/md5/speed-4096: 591.39 MB/sec OK
>> > /crypto/hash/md5/speed-16384: 576.46 MB/sec OK
>> > /crypto/hash/sha1/speed-512: 443.12 MB/sec OK
>> > /crypto/hash/sha1/speed-1024: 518.82 MB/sec OK
>> > /crypto/hash/sha1/speed-4096: 555.60 MB/sec OK
>> > /crypto/hash/sha1/speed-16384: 568.16 MB/sec OK
>> > /crypto/hash/sha224/speed-512: 221.90 MB/sec OK
>> > /crypto/hash/sha224/speed-1024: 239.79 MB/sec OK
>> > /crypto/hash/sha224/speed-4096: 269.37 MB/sec OK
>> > /crypto/hash/sha224/speed-16384: 274.87 MB/sec OK
>> > /crypto/hash/sha256/speed-512: 222.75 MB/sec OK
>> > /crypto/hash/sha256/speed-1024: 253.25 MB/sec OK
>> > /crypto/hash/sha256/speed-4096: 272.80 MB/sec OK
>> > /crypto/hash/sha256/speed-16384: 275.59 MB/sec OK
>> > /crypto/hash/sha384/speed-512: 322.73 MB/sec OK
>> > /crypto/hash/sha384/speed-1024: 369.84 MB/sec OK
>> > /crypto/hash/sha384/speed-4096: 406.71 MB/sec OK
>> > /crypto/hash/sha384/speed-16384: 417.87 MB/sec OK
>> > /crypto/hash/sha512/speed-512: 320.62 MB/sec OK
>> > /crypto/hash/sha512/speed-1024: 361.93 MB/sec OK
>> > /crypto/hash/sha512/speed-4096: 404.91 MB/sec OK
>> > /crypto/hash/sha512/speed-16384: 418.53 MB/sec OK
>> > /crypto/hash/ripemd160/speed-512: 226.45 MB/sec OK
>> > /crypto/hash/ripemd160/speed-1024: 239.25 MB/sec OK
>> > /crypto/hash/ripemd160/speed-4096: 251.31 MB/sec OK
>> > /crypto/hash/ripemd160/speed-16384: 255.01 MB/sec OK
>> >
>> >
>> > IOW, md5 is clearly the quickest, by a considerable margin over
>> > SHA256/512. SHA1 is slightly slower.
>> >
>> > Assuming that we document that this command is intentionally
>> > *not* trying to guarantee collision resistances we're ok.
>> >
>> > In fact we should not document what kind of checksum is
>> > reported by query-memory-checksum. The impl should be a black
>> > box from user's POV.
>> >
>> > If we're just aiming for debugging tool to detect accidental
>> > corruption, could we even just ignore cryptographic hashs
>> > entirely and do a crc32 - that'd be way faster than even
>> > md5.
>>
>> Good points.
>>
>> The doc strings should spell out "for debugging", like the commit
>> message does, and both should spell out "weak collision resistance".
>>
>> I can't find CRC-32 in GLib, but zlib appears to provide it:
>> http://refspecs.linuxbase.org/LSB_3.0.0/LSB-Core-generic/LSB-Core-generic/zlib-crc32-1.html
>>
>> Care to compare its speed to MD5?
>
> I hacked the code to use zlib's crc32 impl and got these for comparison:
>
> /crypto/hash/crc32/speed-512: 1089.18 MB/sec OK
> /crypto/hash/crc32/speed-1024: 1124.63 MB/sec OK
> /crypto/hash/crc32/speed-4096: 1162.73 MB/sec OK
> /crypto/hash/crc32/speed-16384: 1171.58 MB/sec OK
> /crypto/hash/crc32/speed-1048576: 1165.68 MB/sec OK
> /crypto/hash/md5/speed-512: 476.27 MB/sec OK
> /crypto/hash/md5/speed-1024: 517.16 MB/sec OK
> /crypto/hash/md5/speed-4096: 554.70 MB/sec OK
> /crypto/hash/md5/speed-16384: 564.44 MB/sec OK
> /crypto/hash/md5/speed-1048576: 566.78 MB/sec OK
Twice as fast. Alexey, what do you think?
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [RFC PATCH qemu] qapi: Add query-memory-checksum
2019-08-23 11:41 ` Markus Armbruster
@ 2019-08-26 4:28 ` Alexey Kardashevskiy
0 siblings, 0 replies; 9+ messages in thread
From: Alexey Kardashevskiy @ 2019-08-26 4:28 UTC (permalink / raw)
To: Markus Armbruster, Daniel P. Berrangé; +Cc: Paolo Bonzini, qemu-devel
On 23/08/2019 21:41, Markus Armbruster wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
>
>> On Fri, Aug 23, 2019 at 07:49:31AM +0200, Markus Armbruster wrote:
>>> Daniel P. Berrangé <berrange@redhat.com> writes:
>>>
>>>> On Thu, Aug 22, 2019 at 04:16:53PM +0200, Markus Armbruster wrote:
>>>>> Alexey Kardashevskiy <aik@ozlabs.ru> writes:
>>>>>
>>>>>> This returns MD5 checksum of all RAM blocks for migration debugging
>>>>>> as this is way faster than saving the entire RAM to a file and checking
>>>>>> that.
>>>>>>
>>>>>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>>>>>
>>>>> Any particular reason for MD5? Have you measured the other choices
>>>>> offered by GLib?
>>>>>
>>>>> I understand you don't need crypto-strength here. Both MD5 and SHA-1
>>>>> would be bad choices then.
>>>>
>>>> We have a tests/bench-crypto-hash test but its hardcoded for sha256.
>>>> I hacked it to report all algorithms and got these results for varying
>>>> input chunk sizes:
>>>>
>>>> /crypto/hash/md5/speed-512: 519.12 MB/sec OK
>>>> /crypto/hash/md5/speed-1024: 560.39 MB/sec OK
>>>> /crypto/hash/md5/speed-4096: 591.39 MB/sec OK
>>>> /crypto/hash/md5/speed-16384: 576.46 MB/sec OK
>>>> /crypto/hash/sha1/speed-512: 443.12 MB/sec OK
>>>> /crypto/hash/sha1/speed-1024: 518.82 MB/sec OK
>>>> /crypto/hash/sha1/speed-4096: 555.60 MB/sec OK
>>>> /crypto/hash/sha1/speed-16384: 568.16 MB/sec OK
>>>> /crypto/hash/sha224/speed-512: 221.90 MB/sec OK
>>>> /crypto/hash/sha224/speed-1024: 239.79 MB/sec OK
>>>> /crypto/hash/sha224/speed-4096: 269.37 MB/sec OK
>>>> /crypto/hash/sha224/speed-16384: 274.87 MB/sec OK
>>>> /crypto/hash/sha256/speed-512: 222.75 MB/sec OK
>>>> /crypto/hash/sha256/speed-1024: 253.25 MB/sec OK
>>>> /crypto/hash/sha256/speed-4096: 272.80 MB/sec OK
>>>> /crypto/hash/sha256/speed-16384: 275.59 MB/sec OK
>>>> /crypto/hash/sha384/speed-512: 322.73 MB/sec OK
>>>> /crypto/hash/sha384/speed-1024: 369.84 MB/sec OK
>>>> /crypto/hash/sha384/speed-4096: 406.71 MB/sec OK
>>>> /crypto/hash/sha384/speed-16384: 417.87 MB/sec OK
>>>> /crypto/hash/sha512/speed-512: 320.62 MB/sec OK
>>>> /crypto/hash/sha512/speed-1024: 361.93 MB/sec OK
>>>> /crypto/hash/sha512/speed-4096: 404.91 MB/sec OK
>>>> /crypto/hash/sha512/speed-16384: 418.53 MB/sec OK
>>>> /crypto/hash/ripemd160/speed-512: 226.45 MB/sec OK
>>>> /crypto/hash/ripemd160/speed-1024: 239.25 MB/sec OK
>>>> /crypto/hash/ripemd160/speed-4096: 251.31 MB/sec OK
>>>> /crypto/hash/ripemd160/speed-16384: 255.01 MB/sec OK
>>>>
>>>>
>>>> IOW, md5 is clearly the quickest, by a considerable margin over
>>>> SHA256/512. SHA1 is slightly slower.
>>>>
>>>> Assuming that we document that this command is intentionally
>>>> *not* trying to guarantee collision resistances we're ok.
>>>>
>>>> In fact we should not document what kind of checksum is
>>>> reported by query-memory-checksum. The impl should be a black
>>>> box from user's POV.
>>>>
>>>> If we're just aiming for debugging tool to detect accidental
>>>> corruption, could we even just ignore cryptographic hashs
>>>> entirely and do a crc32 - that'd be way faster than even
>>>> md5.
>>>
>>> Good points.
>>>
>>> The doc strings should spell out "for debugging", like the commit
>>> message does, and both should spell out "weak collision resistance".
>>>
>>> I can't find CRC-32 in GLib, but zlib appears to provide it:
>>> http://refspecs.linuxbase.org/LSB_3.0.0/LSB-Core-generic/LSB-Core-generic/zlib-crc32-1.html
>>>
>>> Care to compare its speed to MD5?
>>
>> I hacked the code to use zlib's crc32 impl and got these for comparison:
>>
>> /crypto/hash/crc32/speed-512: 1089.18 MB/sec OK
>> /crypto/hash/crc32/speed-1024: 1124.63 MB/sec OK
>> /crypto/hash/crc32/speed-4096: 1162.73 MB/sec OK
>> /crypto/hash/crc32/speed-16384: 1171.58 MB/sec OK
>> /crypto/hash/crc32/speed-1048576: 1165.68 MB/sec OK
>> /crypto/hash/md5/speed-512: 476.27 MB/sec OK
>> /crypto/hash/md5/speed-1024: 517.16 MB/sec OK
>> /crypto/hash/md5/speed-4096: 554.70 MB/sec OK
>> /crypto/hash/md5/speed-16384: 564.44 MB/sec OK
>> /crypto/hash/md5/speed-1048576: 566.78 MB/sec OK
>
> Twice as fast. Alexey, what do you think?
This is even better. TBH I picked md5 as I could not spot crc32 helper
in the first minute (I can see it now) and MD5 felt the fastest
available from glibc :) I'll probably add start..end range(s) and
repost. Thanks for all these numbers and reviews.
--
Alexey
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2019-08-26 4:30 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-08-22 1:16 [Qemu-devel] [RFC PATCH qemu] qapi: Add query-memory-checksum Alexey Kardashevskiy
2019-08-22 1:33 ` Eric Blake
2019-08-22 3:20 ` Alexey Kardashevskiy
2019-08-22 14:16 ` Markus Armbruster
2019-08-22 14:49 ` Daniel P. Berrangé
2019-08-23 5:49 ` Markus Armbruster
2019-08-23 9:23 ` Daniel P. Berrangé
2019-08-23 11:41 ` Markus Armbruster
2019-08-26 4:28 ` Alexey Kardashevskiy
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.