[PATCH v2 0/2] vnc: fix VNC artifacts

[PATCH v2 0/2] vnc: fix VNC artifacts

[Cameron Esfahani via]

Remove VNC optimization to reencode framebuffer update as raw if it's
smaller than the default encoding.  QEMU's implementation was naive and
didn't account for the ZLIB z_stream mutating with each compression.  Just
saving and restoring the output buffer offset wasn't sufficient to "rewind"
the previous encoding.  Considering that ZRLE is never larger than raw and
even though ZLIB can occasionally be fractionally larger than raw, the
overhead of implementing this optimization correctly isn't worth it.

While debugging this, I noticed ZRLE always compresses better than ZLIB.
Prioritize ZRLE over ZLIB, even if the client hints that ZLIB is preferred.

Cameron Esfahani (2):
  vnc: fix VNC artifacts
  vnc: prioritize ZRLE compression over ZLIB

 ui/vnc-enc-zrle.c |  4 ++--
 ui/vnc.c          | 31 +++++++++++--------------------
 2 files changed, 13 insertions(+), 22 deletions(-)

-- 
2.24.0

[PATCH v2 1/2] vnc: fix VNC artifacts

[Cameron Esfahani via]

Patch de3f7de7f4e257ce44cdabb90f5f17ee99624557 was too simplistic in its
implementation: it didn't account for the ZLIB z_stream mutating with
each compression.  Because of the mutation, simply resetting the output
buffer's offset wasn't sufficient to "rewind" the operation.  The mutated
z_stream would generate future zlib blocks which referred to symbols in
past blocks which weren't sent.  This would lead to artifacting.

This reverts commit de3f7de7f4e257ce44cdabb90f5f17ee99624557.

Fixes: <de3f7de7f4e257> ("vnc: allow fall back to RAW encoding")
Signed-off-by: Cameron Esfahani <dirty@apple.com>
---
 ui/vnc.c | 20 ++------------------
 1 file changed, 2 insertions(+), 18 deletions(-)

diff --git a/ui/vnc.c b/ui/vnc.c
index 4100d6e404..3e8d1f1207 100644
--- a/ui/vnc.c
+++ b/ui/vnc.c
@@ -898,8 +898,6 @@ int vnc_raw_send_framebuffer_update(VncState *vs, int x, int y, int w, int h)
 int vnc_send_framebuffer_update(VncState *vs, int x, int y, int w, int h)
 {
     int n = 0;
-    bool encode_raw = false;
-    size_t saved_offs = vs->output.offset;
 
     switch(vs->vnc_encoding) {
         case VNC_ENCODING_ZLIB:
@@ -922,24 +920,10 @@ int vnc_send_framebuffer_update(VncState *vs, int x, int y, int w, int h)
             n = vnc_zywrle_send_framebuffer_update(vs, x, y, w, h);
             break;
         default:
-            encode_raw = true;
+            vnc_framebuffer_update(vs, x, y, w, h, VNC_ENCODING_RAW);
+            n = vnc_raw_send_framebuffer_update(vs, x, y, w, h);
             break;
     }
-
-    /* If the client has the same pixel format as our internal buffer and
-     * a RAW encoding would need less space fall back to RAW encoding to
-     * save bandwidth and processing power in the client. */
-    if (!encode_raw && vs->write_pixels == vnc_write_pixels_copy &&
-        12 + h * w * VNC_SERVER_FB_BYTES <= (vs->output.offset - saved_offs)) {
-        vs->output.offset = saved_offs;
-        encode_raw = true;
-    }
-
-    if (encode_raw) {
-        vnc_framebuffer_update(vs, x, y, w, h, VNC_ENCODING_RAW);
-        n = vnc_raw_send_framebuffer_update(vs, x, y, w, h);
-    }
-
     return n;
 }
 
-- 
2.24.0

[PATCH v2 2/2] vnc: prioritize ZRLE compression over ZLIB

[Cameron Esfahani via]

In my investigation, ZRLE always compresses better than ZLIB so
prioritize ZRLE over ZLIB, even if the client hints that ZLIB is
preferred.

zlib buffer is always reset in zrle_compress_data(), so using offset to
calculate next_out and avail_out is useless.

Signed-off-by: Cameron Esfahani <dirty@apple.com>
---
 ui/vnc-enc-zrle.c |  4 ++--
 ui/vnc.c          | 11 +++++++++--
 2 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/ui/vnc-enc-zrle.c b/ui/vnc-enc-zrle.c
index 17fd28a2e2..b4f71e32cf 100644
--- a/ui/vnc-enc-zrle.c
+++ b/ui/vnc-enc-zrle.c
@@ -98,8 +98,8 @@ static int zrle_compress_data(VncState *vs, int level)
     /* set pointers */
     zstream->next_in = vs->zrle->zrle.buffer;
     zstream->avail_in = vs->zrle->zrle.offset;
-    zstream->next_out = vs->zrle->zlib.buffer + vs->zrle->zlib.offset;
-    zstream->avail_out = vs->zrle->zlib.capacity - vs->zrle->zlib.offset;
+    zstream->next_out = vs->zrle->zlib.buffer;
+    zstream->avail_out = vs->zrle->zlib.capacity;
     zstream->data_type = Z_BINARY;
 
     /* start encoding */
diff --git a/ui/vnc.c b/ui/vnc.c
index 3e8d1f1207..1d7138a3a0 100644
--- a/ui/vnc.c
+++ b/ui/vnc.c
@@ -2071,8 +2071,15 @@ static void set_encodings(VncState *vs, int32_t *encodings, size_t n_encodings)
             break;
 #endif
         case VNC_ENCODING_ZLIB:
-            vs->features |= VNC_FEATURE_ZLIB_MASK;
-            vs->vnc_encoding = enc;
+            /*
+             * VNC_ENCODING_ZRLE compresses better than VNC_ENCODING_ZLIB.
+             * So prioritize ZRLE, even if the client hints that it prefers
+             * ZLIB.
+             */
+            if ((vs->features & VNC_FEATURE_ZRLE_MASK) == 0) {
+                vs->features |= VNC_FEATURE_ZLIB_MASK;
+                vs->vnc_encoding = enc;
+            }
             break;
         case VNC_ENCODING_ZRLE:
             vs->features |= VNC_FEATURE_ZRLE_MASK;
-- 
2.24.0

Re: [PATCH v2 1/2] vnc: fix VNC artifacts

[Gerd Hoffmann]

On Mon, Jan 20, 2020 at 09:00:51PM -0800, Cameron Esfahani wrote:
> Patch de3f7de7f4e257ce44cdabb90f5f17ee99624557 was too simplistic in its
> implementation: it didn't account for the ZLIB z_stream mutating with
> each compression.  Because of the mutation, simply resetting the output
> buffer's offset wasn't sufficient to "rewind" the operation.  The mutated
> z_stream would generate future zlib blocks which referred to symbols in
> past blocks which weren't sent.  This would lead to artifacting.
> 
> This reverts commit de3f7de7f4e257ce44cdabb90f5f17ee99624557.
> 
> Fixes: <de3f7de7f4e257> ("vnc: allow fall back to RAW encoding")
> Signed-off-by: Cameron Esfahani <dirty@apple.com>

Looks like you didn't realize that "revert" was meant literally.  Git has a
revert subcommand, i.e. you can simply run "git revert de3f7de7f4e257" to
create a commit undoing the changes, with a commit message saying so.  The
generated text should be left intact, to make the job for tools analyzing git
commits easier.  The commit message (for reverts typically explaining why the
reverted commit was buggy) can go below the generated text.

Also note that only the patch commit messages end up in the commit log, the
cover letter text doesn't.  So any important details should (also) be in the
commit messages so they are recorded in the log.

Reworked the commit message, looks like this now:

-----------------------------------------------------------------
commit 0780ec7be82dd4781e9fd216b5d99a125882ff5a (HEAD -> queue/ui)
Author: Gerd Hoffmann <kraxel@redhat.com>
Date:   Tue Jan 21 07:02:10 2020 +0100

    Revert "vnc: allow fall back to RAW encoding"
    
    This reverts commit de3f7de7f4e257ce44cdabb90f5f17ee99624557.
    
    Remove VNC optimization to reencode framebuffer update as raw if it's
    smaller than the default encoding.
    
    QEMU's implementation was naive and didn't account for the ZLIB z_stream
    mutating with each compression.  Because of the mutation, simply
    resetting the output buffer's offset wasn't sufficient to "rewind" the
    operation.  The mutated z_stream would generate future zlib blocks which
    referred to symbols in past blocks which weren't sent.  This would lead
    to artifacting.
    
    Considering that ZRLE is never larger than raw and even though ZLIB can
    occasionally be fractionally larger than raw, the overhead of
    implementing this optimization correctly isn't worth it.
    
    Signed-off-by: Cameron Esfahani <dirty@apple.com>
    Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
-----------------------------------------------------------------

Modified patch queued up.
Patch 2/2 is fine as-is.

thanks,
  Gerd