[Qemu-devel] PATCH: Secure TLS encrypted authentication for VNC

* [Qemu-devel] PATCH: Secure TLS encrypted authentication for VNC
@ 2007-02-24 16:54 Daniel P. Berrange
  2007-02-24 18:57 ` Luke-Jr
  2007-02-28 21:27 ` [Qemu-devel] " S. I. Becker
  0 siblings, 2 replies; 6+ messages in thread
From: Daniel P. Berrange @ 2007-02-24 16:54 UTC (permalink / raw)
  To: qemu-devel

[-- Attachment #1: Type: text/plain, Size: 6326 bytes --]

Having repeatedly said that we should be doing TLS encryption for VNC, I 
figured I ought to get down & implement it. So, in the spirit of 'release
early, release often', here is the very first cut of my patch for QEMU.
This isn't suitable for inclusion in CVS yet - I just want to put it out
for people to see / experiment with.

 - The event loop in vl.c does not handle removal of file descriptors
   in a safe way, so if you remove a filedescriptor during a read 
   event, there is a pretty good chance the event loop will SEGV when
   checking the write condition. The way I've got around this is that
   in the qemu_set_fd_handler2() method, instead of removing the 
   IOHandlerRecord object from the list, we simply set the fd to -1.
   In the main_loop_wait() method, when processing the results from
   select() we skip over any fd's set to -1, and then have a 2nd
   loop over the IOHandlerRecord records to finally remove any objects
   whose fd is set to -1. This lets us safely remove filedescriptors
   from within callbacks.

 - The VNC code now negotiates any protocol version out of 3.3, 3.7, 3.8

 - There is support for the current 'None' auth type, the standard 'VNC'
   challenge/response auth type, and finally the VeNCrypt extension which
   implements a TLS layer with several sub-auth types. Since it can now
   do any protocol version, and negotiate auth types, we should be able
   to easily add more auth types if we want compatability with other 
   RFB auth extensions from projects like UltraVNC/TightVNC/etc. 

 - When choosing the VeNCrypt auth type, the client/server then negotiate
   which sub-auth type they want to use. Then they perform a standard
   TLS handshake, and if this is successful move on to do the actual
   authentication. So the actual auth data exchange, and all subsequent
   RFB protocol traffic is TLS encrypted.

 - The sub-auth types supported by VeNCrypt are:

     - Plain  - username & password - no TLS at all
     - TLS Anon + None - TLS anonymous credential exchange, followed
                         by standard 'None' auth type.
     - TLS Anon + VNC  - TLS anonymous credential exchange, followed
                         by standard 'VNC' auth type.
     - TLS Anon + Plain - TLS anonymous credential exchange, followed
                          by customer 'Plain' username/password auth type.
     - TLS X509 + None - TLS x509 cert credential exchange, followed
                         by standard 'None' auth type.
     - TLS X509 + VNC  - TLS x509 cert credential exchange, followed
                         by standard 'VNC' auth type.
     - TLS X509 + Plain - TLS x509 cert credential exchange, followed
                          by customer 'Plain' username/password auth type.

 - I did not implement any of the 'Plain' sub-auth types above. I may
   add the TLS encrtyped Plain auth types, but certainly not the clear
   text version.

 - The 3 TLS Anon subauth types use the basic diffie-hellman anonymous
   credential exchange. Since there is no apriori trust relationship, 
   this is subject to MITM attacks, so only marginally more useful than
   the existing clear text wire format.

 - The 3 TLS x509 subauth types do a full x509 certificate exchange.
   This is exactly the same top security model as used in the most 
   recent HTTPS protocol implementationss, so the mode I'd recommend.
   The server needs to be configured with a CA cert, a CA revocation
   list (to block revoked clients), and its own server cert & key.
   The server is currently setup to request a client cert and will
   verify the cert against the CA cert & CRL. I need to make it 
   possible to turn this client cert verification on/off. If you used
   TLS X509 + None, then a whitelist of client CNAMEs and client 
   cert verification could be your primary auth. If you use the TLS
   X509 + VNC / Plain auth schemes, then client cert verification 
   should be optional. So client programs connecting at very least
   need access to the CA Cert, and if the server does cert verification
   client programs will need to supply their own certificate too.

 - Currently I've just hardcode which auth scheme the server will
   activate in vnc_display_init() method - this is one of the reasons
   the patch isn't ready to commit to CVS yet. I need to tie in the
   auth setup into either extra command line args, or extra monitor
   commands (or probably a mix of both). eg I need to hook up Anthony
   Ligouri's monitor command for changing the password to this.
   I also need to figure out to best specify the paths to the server
   key/key and CA cert/ CRL files. Currently they're hardcoded to
   be in the files  cert.pem, key.pem, ca-cert.pem ca-crl.pem in
   the current working directory.

 - I hardcoded the makefile to call pkg-config to determine gnutls
   compiler/linker flags. This obviously needs to be fixed to be
   done in the regular configure script. Also all the TLS support
   needs to be #ifdef HAVE_TLS  so that people can still build
   QEMU without gnutls if desired.

 - The separately attached d3des.h, d3des.c files are the standard
   DES implementation shared across every single VNC client / server
   impl out there. I've just included them 'as-is' with no change.

 - If configured to use the None, or VNC auth types, any of the 
   standard VNC viewer programs will connect and if neccessary 
   do the challenge/response authentication just fine. If the TLS
   VeNCrypt authentication type is activated, then you will obviously
   need a client program which supports this - the VeNCrypt project
   on sourceforge supplies a vncviewer implementing this scheme.
   I am also working with Anthony Ligouri to extend his awesome
   GTK VNC widget to support all the different authentication types. 
   This widget will provide a very easy way for people who want to
   build GUI frontends around QEMU to drop in secure console support.
   I intend to integrate it in virt-manager for example.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

[-- Attachment #2: qemu-vnc-tls.patch --]
[-- Type: text/plain, Size: 25288 bytes --]

Index: Makefile.target
===================================================================
RCS file: /sources/qemu/qemu/Makefile.target,v
retrieving revision 1.145
diff -u -p -u -p -r1.145 Makefile.target

--- Makefile.target	10 Feb 2007 21:52:52 -0000	1.145
+++ Makefile.target	24 Feb 2007 15:45:43 -0000
@@ -193,6 +193,8 @@ endif
 ifdef CONFIG_SOLARIS
 LIBS+=-lsocket -lnsl -lresolv
 endif
+LIBS+=$(shell pkg-config --libs gnutls)
+CPPFLAGS+=$(shell pkg-config --cflags gnutls)
 
 # profiling code
 ifdef TARGET_GPROF
@@ -414,7 +416,7 @@ endif
 ifdef CONFIG_SDL
 VL_OBJS+=sdl.o x_keymap.o
 endif
-VL_OBJS+=vnc.o
+VL_OBJS+=vnc.o d3des.o
 ifdef CONFIG_COCOA
 VL_OBJS+=cocoa.o
 COCOA_LIBS=-F/System/Library/Frameworks -framework Cocoa -framework IOKit
@@ -472,7 +474,7 @@ cocoa.o: cocoa.m
 sdl.o: sdl.c keymaps.c sdl_keysym.h
 	$(CC) $(CFLAGS) $(CPPFLAGS) $(SDL_CFLAGS) $(BASE_CFLAGS) -c -o $@ $<
 
-vnc.o: vnc.c keymaps.c sdl_keysym.h vnchextile.h
+vnc.o: vnc.c keymaps.c sdl_keysym.h vnchextile.h d3des.h
 	$(CC) $(CFLAGS) $(CPPFLAGS) $(BASE_CFLAGS) -c -o $@ $<
 
 sdlaudio.o: sdlaudio.c
Index: vl.c
===================================================================
RCS file: /sources/qemu/qemu/vl.c,v
retrieving revision 1.259
diff -u -p -u -p -r1.259 vl.c
--- vl.c	22 Feb 2007 01:48:01 -0000	1.259
+++ vl.c	24 Feb 2007 15:45:44 -0000
@@ -4487,8 +4487,7 @@ int qemu_set_fd_handler2(int fd, 
             if (ioh == NULL)
                 break;
             if (ioh->fd == fd) {
-                *pioh = ioh->next;
-                qemu_free(ioh);
+	        ioh->fd = -1;
                 break;
             }
             pioh = &ioh->next;
@@ -6157,7 +6156,7 @@ void qemu_system_powerdown_request(void)
 
 void main_loop_wait(int timeout)
 {
-    IOHandlerRecord *ioh, *ioh_next;
+  IOHandlerRecord *ioh, *ioh_next, *ioh_prev;
     fd_set rfds, wfds, xfds;
     int ret, nfds;
     struct timeval tv;
@@ -6219,16 +6218,31 @@ void main_loop_wait(int timeout)
 #endif
     ret = select(nfds + 1, &rfds, &wfds, &xfds, &tv);
     if (ret > 0) {
-        /* XXX: better handling of removal */
         for(ioh = first_io_handler; ioh != NULL; ioh = ioh_next) {
             ioh_next = ioh->next;
             if (FD_ISSET(ioh->fd, &rfds)) {
                 ioh->fd_read(ioh->opaque);
             }
-            if (FD_ISSET(ioh->fd, &wfds)) {
+            if (ioh->fd != -1 && FD_ISSET(ioh->fd, &wfds)) {
                 ioh->fd_write(ioh->opaque);
             }
         }
+	ioh_prev = NULL;
+	ioh = first_io_handler;
+	while (ioh) {
+	    ioh_next = ioh->next;
+	    if (ioh->fd == -1) {
+	        if (ioh_prev == NULL) {
+		  first_io_handler = ioh_next;
+		} else {
+		  ioh_prev->next = ioh_next;
+		}
+	        qemu_free(ioh);
+            } else {
+	      ioh_prev = ioh;
+	    }
+	    ioh = ioh_next;
+        }
     }
 #if defined(CONFIG_SLIRP)
     if (slirp_inited) {
Index: vnc.c
===================================================================
RCS file: /sources/qemu/qemu/vnc.c,v
retrieving revision 1.12
diff -u -p -u -p -r1.12 vnc.c
--- vnc.c	5 Feb 2007 20:20:30 -0000	1.12
+++ vnc.c	24 Feb 2007 15:45:44 -0000
@@ -30,6 +30,10 @@
 
 #include "vnc_keysym.h"
 #include "keymaps.c"
+#include "d3des.h"
+
+#include <gnutls/gnutls.h>
+#include <gnutls/x509.h>
 
 typedef struct Buffer
 {
@@ -54,12 +58,66 @@ typedef void VncSendHextileTile(VncState
 #define VNC_MAX_HEIGHT 2048
 #define VNC_DIRTY_WORDS (VNC_MAX_WIDTH / (16 * 32))
 
+enum {
+    VNC_WIREMODE_CLEAR,
+    VNC_WIREMODE_TLS_HANDSHAKE,
+    VNC_WIREMODE_TLS_SESSION,
+};
+
+enum {
+    VNC_WIRE_AUTH_CLEAR,
+    VNC_WIRE_AUTH_TLS_ANON,
+    VNC_WIRE_AUTH_TLS_X509,
+};
+
+enum {
+    VNC_AUTH_INVALID = 0,
+    VNC_AUTH_NONE = 1,
+    VNC_AUTH_VNC = 2,
+    VNC_AUTH_RA2 = 5,
+    VNC_AUTH_RA2NE = 6,
+    VNC_AUTH_TIGHT = 16,
+    VNC_AUTH_ULTRA = 17,
+    VNC_AUTH_TLS = 18,
+    VNC_AUTH_VENCRYPT = 19
+};
+
+enum {
+    VNC_AUTH_VENCRYPT_PLAIN = 256,
+    VNC_AUTH_VENCRYPT_TLSNONE = 257,
+    VNC_AUTH_VENCRYPT_TLSVNC = 258,
+    VNC_AUTH_VENCRYPT_TLSPLAIN = 259,
+    VNC_AUTH_VENCRYPT_X509NONE = 260,
+    VNC_AUTH_VENCRYPT_X509VNC = 261,
+    VNC_AUTH_VENCRYPT_X509PLAIN = 262,
+};
+
+#define CA_FILE "ca-cert.pem"
+#define CRL_FILE "ca-crl.pem"
+#define KEY_FILE "key.pem"
+#define CERT_FILE "cert.pem"
+
+#define MAX_AUTH 20
+#define AUTHCHALLENGESIZE 16
+
 struct VncState
 {
     QEMUTimer *timer;
     int lsock;
     int csock;
     DisplayState *ds;
+
+    int major;
+    int minor;
+
+    int auth;
+    int subauth;
+
+    int wiremode;
+    gnutls_session_t tls_session;
+    char *password;
+    char challenge[AUTHCHALLENGESIZE];
+
     int need_update;
     int width;
     int height;
@@ -94,6 +152,19 @@ struct VncState
 
 static VncState *vnc_state; /* needed for info vnc */
 
+#if 1
+#define VNC_DEBUG(fmt, ...) do { fprintf(stderr, fmt, ## __VA_ARGS__); } while (0)
+#else
+#define VNC_DEBUG(fmt, ...) do { } while (0)
+#endif
+
+static void debug_log(int level, const char* str) {
+#if 0
+    VNC_DEBUG("%d %s", level, str);
+#endif
+}
+
+
 void do_info_vnc(void)
 {
     if (vnc_state == NULL)
@@ -546,16 +617,21 @@ static void buffer_append(Buffer *buffer
 
 static int vnc_client_io_error(VncState *vs, int ret, int last_errno)
 {
-    if (ret == 0 || ret == -1) {
-	if (ret == -1 && (last_errno == EINTR || last_errno == EAGAIN))
+    if (ret <= 0) {
+	if (ret < 0 && (last_errno == EINTR || last_errno == EAGAIN))
 	    return 0;
 
+	VNC_DEBUG("Closing down client sock\n");
 	qemu_set_fd_handler2(vs->csock, NULL, NULL, NULL, NULL);
 	closesocket(vs->csock);
 	vs->csock = -1;
 	buffer_reset(&vs->input);
 	buffer_reset(&vs->output);
 	vs->need_update = 0;
+	if (vs->tls_session) {
+	    gnutls_deinit(vs->tls_session);
+	    vs->tls_session = NULL;
+	}
 	return 0;
     }
     return ret;
@@ -571,7 +647,10 @@ static void vnc_client_write(void *opaqu
     long ret;
     VncState *vs = opaque;
 
-    ret = send(vs->csock, vs->output.buffer, vs->output.offset, 0);
+    if (vs->tls_session) {
+	ret = gnutls_write(vs->tls_session, vs->output.buffer, vs->output.offset);
+    } else
+	ret = send(vs->csock, vs->output.buffer, vs->output.offset, 0);
     ret = vnc_client_io_error(vs, ret, socket_error());
     if (!ret)
 	return;
@@ -597,7 +676,10 @@ static void vnc_client_read(void *opaque
 
     buffer_reserve(&vs->input, 4096);
 
-    ret = recv(vs->csock, buffer_end(&vs->input), 4096, 0);
+    if (vs->tls_session)
+	ret = gnutls_read(vs->tls_session, buffer_end(&vs->input), 4096);
+    else
+	ret = recv(vs->csock, buffer_end(&vs->input), 4096, 0);
     ret = vnc_client_io_error(vs, ret, socket_error());
     if (!ret)
 	return;
@@ -692,6 +774,44 @@ static uint32_t read_u32(uint8_t *data, 
 	    (data[offset + 2] << 8) | data[offset + 3]);
 }
 
+ssize_t vnc_tls_push(gnutls_transport_ptr_t transport,
+		     const void *data,
+		     size_t len) {
+    struct VncState *vs = (struct VncState *)transport;
+    int ret, lastErrno;
+
+    ret = send(vs->csock, data, len, 0);
+    lastErrno = socket_error();
+    ret = vnc_client_io_error(vs, ret, lastErrno);
+    errno = lastErrno;
+    if (!ret) {
+	VNC_DEBUG("Failed errno %d\n", errno);
+	return -1;
+    }
+
+    return ret;
+}
+
+
+ssize_t vnc_tls_pull(gnutls_transport_ptr_t transport,
+		     void *data,
+		     size_t len) {
+    struct VncState *vs = (struct VncState *)transport;
+    int ret, lastErrno;
+
+    ret = recv(vs->csock, data, len, 0);
+    lastErrno = socket_error();
+    ret = vnc_client_io_error(vs, ret, lastErrno);
+    errno = lastErrno;
+    if (!ret) {
+	VNC_DEBUG("Failed errno %d\n", errno);
+	return -1;
+    }
+
+    return ret;
+}
+
+
 static void client_cut_text(VncState *vs, size_t len, char *text)
 {
 }
@@ -1109,23 +1229,523 @@ static int protocol_client_init(VncState
     return 0;
 }
 
+
+
+static void make_challenge(VncState *vs)
+{
+    int i;
+
+    srand(time(NULL)+getpid()+getpid()*987654+rand());
+
+    for (i = 0 ; i < sizeof(vs->challenge) ; i++)
+        vs->challenge[i] = (int) (256.0*rand()/(RAND_MAX+1.0));
+}
+
+static int protocol_client_auth_vnc(VncState *vs, char *data, size_t len)
+{
+    char response[AUTHCHALLENGESIZE];
+    int i, j, pwlen;
+    char key[8];
+
+    memcpy(response, vs->challenge, AUTHCHALLENGESIZE);
+
+    /* Calculate the sent challenge */
+    pwlen = strlen(vs->password);
+    for (i=0; i<sizeof(key); i++)
+        key[i] = i<pwlen ? vs->password[i] : 0;
+    deskey(key, EN0);
+    for (j = 0; j < AUTHCHALLENGESIZE; j += 8)
+        des(response+j, response+j);
+    
+    if (memcmp(response, data, AUTHCHALLENGESIZE) != 0) {
+	VNC_DEBUG("Client challenge reponse did not match\n");
+	vnc_write_u32(vs, 1); /* Reject auth */
+	if (vs->minor >= 8) {
+	    static const char err[] = "Authentication failed";
+	    vnc_write_u32(vs, sizeof(err));
+	    vnc_write(vs, err, sizeof(err));
+	}
+	vnc_flush(vs);
+	vnc_client_error(vs);
+    } else {
+	VNC_DEBUG("Accepting VNC challenge response\n");
+	vnc_write_u32(vs, 0); /* Accept auth */
+	vnc_flush(vs);
+
+	vnc_read_when(vs, protocol_client_init, 1);
+    }
+    return 0;
+}
+
+static int start_auth_vnc(VncState *vs)
+{
+    make_challenge(vs);
+    vnc_write(vs, vs->challenge, sizeof(vs->challenge));
+    vnc_flush(vs);
+
+    vnc_read_when(vs, protocol_client_auth_vnc, sizeof(vs->challenge));
+    return 0;
+}
+
+#define DH_BITS 1024
+static gnutls_dh_params_t dh_params;
+
+static int vnc_tls_initialize(void)
+{
+    static int tlsinitialized = 0;
+
+    if (tlsinitialized)
+	return 1;
+
+    if (gnutls_global_init () < 0)
+	return 0;
+
+    if (gnutls_dh_params_init (&dh_params) < 0)
+	return 0;
+    if (gnutls_dh_params_generate2 (dh_params, DH_BITS) < 0)
+	return 0;
+
+    gnutls_global_set_log_level(10);
+    gnutls_global_set_log_function(debug_log);
+
+    tlsinitialized = 1;
+
+    return 1;
+}
+
+static gnutls_anon_server_credentials vnc_tls_initialize_anon_cred(void)
+{
+    gnutls_anon_server_credentials anon_cred;
+    int ret;
+
+    if ((ret = gnutls_anon_allocate_server_credentials(&anon_cred)) < 0) {
+	VNC_DEBUG("Cannot allocate credentials %s\n", gnutls_strerror(ret));
+	return NULL;
+    }
+
+    gnutls_anon_set_server_dh_params(anon_cred, dh_params);
+
+    return anon_cred;
+}
+
+static gnutls_certificate_credentials_t vnc_tls_initialize_cert_cred(void)
+{
+    gnutls_certificate_credentials_t x509_cred;
+    int ret;
+    struct stat st;
+
+    if ((ret = gnutls_certificate_allocate_credentials(&x509_cred)) < 0) {
+	VNC_DEBUG("Cannot allocate credentials %s\n", gnutls_strerror(ret));
+	return NULL;
+    }
+    if ((ret = gnutls_certificate_set_x509_trust_file(x509_cred, CA_FILE, GNUTLS_X509_FMT_PEM)) < 0) {
+	VNC_DEBUG("Cannot load CA certificate %s\n", gnutls_strerror(ret));
+	return NULL;
+    }
+
+    if ((ret = gnutls_certificate_set_x509_key_file (x509_cred, CERT_FILE, KEY_FILE,
+						     GNUTLS_X509_FMT_PEM)) < 0) {
+	VNC_DEBUG("Cannot load certificate & key %s\n", gnutls_strerror(ret));
+	return NULL;
+    }
+
+    if (stat(CRL_FILE, &st) < 0) {
+	if (errno != ENOENT) {
+	    return NULL;
+	}
+    } else {
+	if ((ret = gnutls_certificate_set_x509_crl_file(x509_cred, CRL_FILE, GNUTLS_X509_FMT_PEM)) < 0) {
+	    VNC_DEBUG("Cannot load CRL %s\n", gnutls_strerror(ret));
+	    return NULL;
+	}
+    }
+
+    gnutls_certificate_set_dh_params (x509_cred, dh_params);
+
+    return x509_cred;
+}
+
+static int vnc_validate_certificate(struct VncState *vnc)
+{
+    int ret;
+    unsigned int status;
+    const gnutls_datum_t *certs;
+    unsigned int nCerts, i;
+    time_t now;
+
+    VNC_DEBUG("Validating\n");
+    if ((ret = gnutls_certificate_verify_peers2 (vnc->tls_session, &status)) < 0) {
+	VNC_DEBUG("Verify failed %s\n", gnutls_strerror(ret));
+	return 0;
+    }
+
+    if ((now = time(NULL)) == ((time_t)-1)) {
+	return 0;
+    }
+
+    if (status != 0) {
+	if (status & GNUTLS_CERT_INVALID)
+	    VNC_DEBUG ("The certificate is not trusted.\n");
+
+	if (status & GNUTLS_CERT_SIGNER_NOT_FOUND)
+	    VNC_DEBUG ("The certificate hasn't got a known issuer.\n");
+
+	if (status & GNUTLS_CERT_REVOKED)
+	    VNC_DEBUG ("The certificate has been revoked.\n");
+
+	if (status & GNUTLS_CERT_INSECURE_ALGORITHM)
+	    VNC_DEBUG ("The certificate uses an insecure algorithm\n");
+
+	return 0;
+    } else {
+	VNC_DEBUG("Certificate is valid!\n");
+    }
+
+    if (gnutls_certificate_type_get(vnc->tls_session) != GNUTLS_CRT_X509)
+	return 0;
+
+    if (!(certs = gnutls_certificate_get_peers(vnc->tls_session, &nCerts)))
+	return 0;
+
+    for (i = 0 ; i < nCerts ; i++) {
+	gnutls_x509_crt_t cert;
+	VNC_DEBUG ("Checking chain %d\n", i);
+	if (gnutls_x509_crt_init (&cert) < 0)
+	    return 0;
+
+	if (gnutls_x509_crt_import(cert, &certs[i], GNUTLS_X509_FMT_DER) < 0) {
+	    gnutls_x509_crt_deinit (cert);
+	    return 0;
+	}
+
+	if (gnutls_x509_crt_get_expiration_time (cert) < now) {
+	    VNC_DEBUG("The certificate has expired\n");
+	    gnutls_x509_crt_deinit (cert);
+	    return 0;
+	}
+
+	if (gnutls_x509_crt_get_activation_time (cert) > now) {
+	    VNC_DEBUG("The certificate is not yet activated\n");
+	    gnutls_x509_crt_deinit (cert);
+	    return 0;
+	}
+
+	if (gnutls_x509_crt_get_activation_time (cert) > now) {
+	    VNC_DEBUG("The certificate is not yet activated\n");
+	    gnutls_x509_crt_deinit (cert);
+	    return 0;
+	}
+    }
+
+    return 1;
+}
+
+static int start_auth_vencrypt_subauth(VncState *vs)
+{
+    switch (vs->subauth) {
+    case VNC_AUTH_VENCRYPT_TLSNONE:
+    case VNC_AUTH_VENCRYPT_X509NONE:
+	VNC_DEBUG("Accept auth none\n");
+	vnc_write_u32(vs, 0); /* Accept auth completion */
+	vnc_read_when(vs, protocol_client_init, 1);
+	break;
+	
+    case VNC_AUTH_VENCRYPT_TLSVNC:
+    case VNC_AUTH_VENCRYPT_X509VNC:
+	VNC_DEBUG("Start VNC auth\n");
+	return start_auth_vnc(vs);
+	
+    default: /* Should not be possible, but just in case */
+	VNC_DEBUG("Reject auth %d\n", vs->auth);
+	vnc_write_u8(vs, 1);
+	if (vs->minor >= 8) {
+	    static const char err[] = "Unsupported authentication type";
+	    vnc_write_u32(vs, sizeof(err));
+	    vnc_write(vs, err, sizeof(err));
+	}
+	vnc_client_error(vs);
+    }
+
+    return 0;
+}
+
+static void vnc_handshake_io(void *opaque);
+
+static int vnc_continue_handshake(struct VncState *vs, int anonTLS) {
+    int ret;
+
+    if ((ret = gnutls_handshake(vs->tls_session)) < 0) {
+	if (!gnutls_error_is_fatal(ret)) {
+	    if (!gnutls_record_get_direction(vs->tls_session))
+		qemu_set_fd_handler(vs->csock, vnc_handshake_io, NULL, vs);
+	    else
+		qemu_set_fd_handler(vs->csock, NULL, vnc_handshake_io, vs);
+	    VNC_DEBUG("Handshake interrupted (blocking)\n");
+	    return 0;
+	}
+	VNC_DEBUG("Handshake failed %s\n", gnutls_strerror(ret));
+	vnc_client_error(vs);
+	return -1;
+    }
+
+    VNC_DEBUG("Handshake done\n");
+
+    if (anonTLS) {
+	VNC_DEBUG("Anonymous TLS, no certificate checks\n");
+    } else {
+	if (!vnc_validate_certificate(vs)) {
+	    VNC_DEBUG("Certificate validation failed\n");
+	    vnc_client_error(vs);
+	    return -1;
+	}
+    }
+
+    VNC_DEBUG("Switching to TLS data mode\n");
+    vs->wiremode = VNC_WIREMODE_TLS_SESSION;
+    qemu_set_fd_handler2(vs->csock, NULL, vnc_client_read, vnc_client_write, vs);
+
+    return start_auth_vencrypt_subauth(vs);
+}
+
+static void vnc_handshake_io(void *opaque) {
+    struct VncState *vs = (struct VncState *)opaque;
+    int anon = 0;
+
+    if (vs->subauth == VNC_AUTH_VENCRYPT_TLSNONE ||
+	vs->subauth == VNC_AUTH_VENCRYPT_TLSVNC ||
+	vs->subauth == VNC_AUTH_VENCRYPT_TLSPLAIN)
+	anon = 1;
+
+    vnc_continue_handshake(vs, anon);
+}
+
+static int vnc_start_tls(struct VncState *vs, int anonTLS) {
+    static const int cert_type_priority[] = { GNUTLS_CRT_X509, 0 };
+    static const int protocol_priority[]= { GNUTLS_TLS1_1, GNUTLS_TLS1_0, GNUTLS_SSL3, 0 };
+    static const int kx_priority[] = {GNUTLS_KX_DHE_DSS, GNUTLS_KX_RSA, GNUTLS_KX_DHE_RSA, GNUTLS_KX_SRP, 0};
+    static const int kx_anon[] = {GNUTLS_KX_ANON_DH, 0};
+
+    VNC_DEBUG("Do TLS handshake, anon ? %d\n", anonTLS);
+    if (vnc_tls_initialize() < 0) {
+	VNC_DEBUG("Failed to init TLS\n");
+	vnc_client_error(vs);
+	return -1;
+    }
+    if (vs->tls_session == NULL) {
+	if (gnutls_init(&vs->tls_session, GNUTLS_SERVER) < 0) {
+	    vnc_client_error(vs);
+	    return -1;
+	}
+
+	if (gnutls_set_default_priority(vs->tls_session) < 0) {
+	    vnc_client_error(vs);
+	    return -1;
+	}
+
+	if (gnutls_kx_set_priority(vs->tls_session, anonTLS ? kx_anon : kx_priority) < 0) {
+	    vnc_client_error(vs);
+	    return -1;
+	}
+
+	if (gnutls_certificate_type_set_priority(vs->tls_session, cert_type_priority) < 0) {
+	    vnc_client_error(vs);
+	    return -1;
+	}
+
+	if (gnutls_protocol_set_priority(vs->tls_session, protocol_priority) < 0) {
+	    vnc_client_error(vs);
+	    return -1;
+	}
+
+	if (anonTLS) {
+	    gnutls_anon_server_credentials anon_cred = vnc_tls_initialize_anon_cred();
+	    if (!anon_cred) {
+		vnc_client_error(vs);
+		return -1;
+	    }
+	    if (gnutls_credentials_set(vs->tls_session, GNUTLS_CRD_ANON, anon_cred) < 0) {
+		vnc_client_error(vs);
+		return -1;
+	    }
+	} else {
+	    gnutls_certificate_credentials_t x509_cred = vnc_tls_initialize_cert_cred();
+	    if (!x509_cred) {
+		vnc_client_error(vs);
+		return -1;
+	    }
+	    if (gnutls_credentials_set(vs->tls_session, GNUTLS_CRD_CERTIFICATE, x509_cred) < 0) {
+		vnc_client_error(vs);
+		return -1;
+	    }
+	    gnutls_certificate_server_set_request(vs->tls_session, GNUTLS_CERT_REQUEST);
+	}
+
+	gnutls_transport_set_ptr(vs->tls_session, (gnutls_transport_ptr_t)vs);
+	gnutls_transport_set_push_function(vs->tls_session, vnc_tls_push);
+	gnutls_transport_set_pull_function(vs->tls_session, vnc_tls_pull);
+    }
+
+    return vnc_continue_handshake(vs, anonTLS);
+}
+
+
+static int protocol_client_vencrypt_auth(VncState *vs, char *data, size_t len)
+{
+    int auth = read_u32(data, 0);
+
+    if (auth != vs->subauth) {
+	VNC_DEBUG("Rejecting auth %d\n", auth);
+	vnc_write_u8(vs, 0); /* Reject auth */
+	vnc_flush(vs);
+	vnc_client_error(vs);
+    } else {
+	int anon = 0;
+	VNC_DEBUG("Accepting auth %d, starting handshake\n", auth);
+	vnc_write_u8(vs, 1); /* Accept auth */
+	vnc_flush(vs);
+
+	if (vs->subauth == VNC_AUTH_VENCRYPT_TLSNONE ||
+	    vs->subauth == VNC_AUTH_VENCRYPT_TLSVNC ||
+	    vs->subauth == VNC_AUTH_VENCRYPT_TLSPLAIN)
+	    anon = 1;
+
+	if (vnc_start_tls(vs, anon) < 0) {
+	    VNC_DEBUG("Failed to complete TLS\n");
+	    return 0;
+	}
+
+	if (vs->wiremode == VNC_WIREMODE_TLS_SESSION) {
+	    VNC_DEBUG("Starting VeNCrypt subauth\n");
+	    return start_auth_vencrypt_subauth(vs);
+	} else {
+	    VNC_DEBUG("TLS handshake blocked\n");
+	    return 0;
+	}
+    }
+    return 0;
+}
+
+static int protocol_client_vencrypt_init(VncState *vs, char *data, size_t len)
+{
+    if (data[0] != 0 ||
+	data[1] != 2) {
+	VNC_DEBUG("Unsupported VeNCrypt protocol %d.%d\n", (int)data[0], (int)data[1]);
+	vnc_write_u8(vs, 1); /* Reject version */
+	vnc_flush(vs);
+	vnc_client_error(vs);
+    } else {
+	VNC_DEBUG("Sending allowed auth %d\n", vs->subauth);
+	vnc_write_u8(vs, 0); /* Accept version */
+	vnc_write_u8(vs, 1); /* Number of sub-auths */
+	vnc_write_u32(vs, vs->subauth); /* The supported auth */
+	vnc_flush(vs);
+	vnc_read_when(vs, protocol_client_vencrypt_auth, 4);
+    }
+    return 0;
+}
+
+static int start_auth_vencrypt(VncState *vs)
+{
+    /* Send VeNCrypt version 0.2 */
+    vnc_write_u8(vs, 0);
+    vnc_write_u8(vs, 2);
+
+    vnc_read_when(vs, protocol_client_vencrypt_init, 2);
+    return 0;
+}
+
+static int protocol_client_auth(VncState *vs, char *data, size_t len)
+{
+    if (data[0] != vs->auth) { /* Reject auth */
+	VNC_DEBUG("Reject auth %d\n", (int)data[0]);
+	vnc_write_u32(vs, 1);
+	if (vs->minor >= 8) {
+	    static const char err[] = "Unsupported authentication type";
+	    vnc_write_u32(vs, sizeof(err));
+	    vnc_write(vs, err, sizeof(err));
+	}
+	vnc_client_error(vs);
+    } else { /* Accept requested auth */
+	VNC_DEBUG("Client requested auth %d\n", (int)data[0]);
+	switch (vs->auth) {
+	case VNC_AUTH_NONE:
+	    VNC_DEBUG("Accept auth none\n");
+	    vnc_write_u32(vs, 0); /* Accept auth completion */
+	    vnc_read_when(vs, protocol_client_init, 1);
+	    break;
+
+	case VNC_AUTH_VNC:
+	    VNC_DEBUG("Start VNC auth\n");
+	    return start_auth_vnc(vs);
+
+	case VNC_AUTH_VENCRYPT:
+	    VNC_DEBUG("Start VeNCrypt auth %d\n", vs->auth);
+	    return start_auth_vencrypt(vs);
+
+	default: /* Should not be possible, but just in case */
+	    VNC_DEBUG("Reject auth %d\n", vs->auth);
+	    vnc_write_u8(vs, 1);
+	    if (vs->minor >= 8) {
+		static const char err[] = "Unsupported authentication type";
+		vnc_write_u32(vs, sizeof(err));
+		vnc_write(vs, err, sizeof(err));
+	    }
+	    vnc_client_error(vs);
+	}
+    }
+    return 0;
+}
+
 static int protocol_version(VncState *vs, char *version, size_t len)
 {
     char local[13];
-    int maj, min;
 
     memcpy(local, version, 12);
     local[12] = 0;
 
-    if (sscanf(local, "RFB %03d.%03d\n", &maj, &min) != 2) {
-	vnc_client_error(vs);
-	return 0;
+    if (sscanf(local, "RFB %03d.%03d\n", &vs->major, &vs->minor) != 2) {
+	VNC_DEBUG("Malformed version response\n");
+        vnc_client_error(vs);
+        return 0;
     }
 
-    vnc_write_u32(vs, 1); /* None */
-    vnc_flush(vs);
+    if (vs->major != 3 ||
+        (vs->minor != 3 &&
+         vs->minor != 7 &&
+         vs->minor != 8)) {
+	VNC_DEBUG("Unsupported version %d.%d\n", vs->major, vs->minor);
+	vnc_write_u32(vs, VNC_AUTH_INVALID);
+	vnc_flush(vs);
+        vnc_client_error(vs);
+        return 0;
+    }
 
-    vnc_read_when(vs, protocol_client_init, 1);
+    VNC_DEBUG("Using protocol %d.%d\n", vs->major, vs->minor);
+
+    if (vs->minor == 3) {
+	if (vs->auth == VNC_AUTH_NONE) {
+	    VNC_DEBUG("Accept auth none\n");
+            vnc_write_u32(vs, vs->auth);
+	    vnc_flush(vs);
+	    vnc_read_when(vs, protocol_client_init, 1);
+	} else if (vs->auth == VNC_AUTH_VNC) {
+	    VNC_DEBUG("Start VNC auth\n");
+            vnc_write_u32(vs, vs->auth);
+	    vnc_flush(vs);
+	    start_auth_vnc(vs);
+	} else {
+	    VNC_DEBUG("Unsupported auth %d in protocol 3.3\n", vs->auth);
+	    vnc_write_u32(vs, VNC_AUTH_INVALID);
+	    vnc_flush(vs);
+	    vnc_client_error(vs);
+	}
+    } else {
+	VNC_DEBUG("Telling client we support auth %d\n", vs->auth);
+	vnc_write_u8(vs, 1); /* num auth */
+	vnc_write_u8(vs, vs->auth);
+	vnc_read_when(vs, protocol_client_auth, 1);
+	vnc_flush(vs);
+    }
 
     return 0;
 }
@@ -1139,20 +1759,60 @@ static void vnc_listen_read(void *opaque
     vs->csock = accept(vs->lsock, (struct sockaddr *)&addr, &addrlen);
     if (vs->csock != -1) {
         socket_set_nonblock(vs->csock);
-	qemu_set_fd_handler2(vs->csock, NULL, vnc_client_read, NULL, opaque);
-	vnc_write(vs, "RFB 003.003\n", 12);
-	vnc_flush(vs);
-	vnc_read_when(vs, protocol_version, 12);
-	memset(vs->old_data, 0, vs->ds->linesize * vs->ds->height);
-	memset(vs->dirty_row, 0xFF, sizeof(vs->dirty_row));
-	vs->has_resize = 0;
-	vs->has_hextile = 0;
-	vs->ds->dpy_copy = NULL;
+        qemu_set_fd_handler2(vs->csock, NULL, vnc_client_read, NULL, opaque);
+        vnc_write(vs, "RFB 003.008\n", 12);
+        vnc_flush(vs);
+        vnc_read_when(vs, protocol_version, 12);
+        memset(vs->old_data, 0, vs->ds->linesize * vs->ds->height);
+        memset(vs->dirty_row, 0xFF, sizeof(vs->dirty_row));
+        vs->has_resize = 0;
+        vs->has_hextile = 0;
+        vs->ds->dpy_copy = NULL;
+        vs->wiremode = VNC_WIREMODE_CLEAR;
+	vs->tls_session = NULL;
     }
 }
 
 extern int parse_host_port(struct sockaddr_in *saddr, const char *str);
 
+void vnc_init_auth(VncState *vs, int mode, const char *password)
+{
+    if (vs->password)
+	free(vs->password);
+    if (password)
+	vs->password = strdup(password);
+    else
+	vs->password = NULL;
+
+    switch (mode) {
+    default:
+    case VNC_WIRE_AUTH_CLEAR:
+	if (vs->password)
+	    vs->auth = VNC_AUTH_VNC;
+	else
+	    vs->auth = VNC_AUTH_NONE;
+	break;
+
+    case VNC_WIRE_AUTH_TLS_ANON:
+	vs->auth = VNC_AUTH_VENCRYPT;
+	if (vs->password)
+	    vs->subauth = VNC_AUTH_VENCRYPT_TLSVNC;
+	else
+	    vs->subauth = VNC_AUTH_VENCRYPT_TLSNONE;
+	break;
+
+    case VNC_WIRE_AUTH_TLS_X509:
+	vs->auth = VNC_AUTH_VENCRYPT;
+	if (vs->password)
+	    vs->subauth = VNC_AUTH_VENCRYPT_X509VNC;
+	else
+	    vs->subauth = VNC_AUTH_VENCRYPT_X509NONE;
+	break;
+    }
+
+    VNC_DEBUG("Selected auth mode %d password '%s'\n", vs->auth, vs->password);
+}
+
 void vnc_display_init(DisplayState *ds, const char *arg)
 {
     struct sockaddr *addr;
@@ -1173,6 +1833,12 @@ void vnc_display_init(DisplayState *ds, 
     vnc_state = vs;
     vs->display = arg;
 
+    /* XXX Settable from command line or monitor */
+    //vnc_init_auth(vs, VNC_WIRE_AUTH_CLEAR, NULL);
+    //vnc_init_auth(vs, VNC_WIRE_AUTH_CLEAR, "123456");
+    vnc_init_auth(vs, VNC_WIRE_AUTH_TLS_ANON, "123456");
+    //vnc_init_auth(vs, VNC_WIRE_AUTH_TLS_X509, "123456");
+
     vs->lsock = -1;
     vs->csock = -1;
     vs->depth = 4;
@@ -1256,3 +1922,11 @@ void vnc_display_init(DisplayState *ds, 
 	exit(1);
     }
 }
+
+/*
+ * Local variables:
+ *  c-indent-level: 4
+ *  c-basic-offset: 4
+ *  tab-width: 8
+ * End:
+ */

[-- Attachment #3: d3des.c --]
[-- Type: text/plain, Size: 15629 bytes --]

/*
 * This is D3DES (V5.09) by Richard Outerbridge with the double and
 * triple-length support removed for use in VNC.  Also the bytebit[] array
 * has been reversed so that the most significant bit in each byte of the
 * key is ignored, not the least significant.
 *
 * These changes are:
 *  Copyright (C) 1999 AT&T Laboratories Cambridge.  All Rights Reserved.
 *
 * This software is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 */

/* D3DES (V5.09) -
 *
 * A portable, public domain, version of the Data Encryption Standard.
 *
 * Written with Symantec's THINK (Lightspeed) C by Richard Outerbridge.
 * Thanks to: Dan Hoey for his excellent Initial and Inverse permutation
 * code;  Jim Gillogly & Phil Karn for the DES key schedule code; Dennis
 * Ferguson, Eric Young and Dana How for comparing notes; and Ray Lau,
 * for humouring me on.
 *
 * Copyright (c) 1988,1989,1990,1991,1992 by Richard Outerbridge.
 * (GEnie : OUTER; CIS : [71755,204]) Graven Imagery, 1992.
 */

#include "d3des.h"

static void scrunch(unsigned char *, unsigned long *);
static void unscrun(unsigned long *, unsigned char *);
static void desfunc(unsigned long *, unsigned long *);
static void cookey(unsigned long *);

static unsigned long KnL[32] = { 0L };
/*
static unsigned long KnR[32] = { 0L };
static unsigned long Kn3[32] = { 0L };
static unsigned char Df_Key[24] = {
	0x01,0x23,0x45,0x67,0x89,0xab,0xcd,0xef,
	0xfe,0xdc,0xba,0x98,0x76,0x54,0x32,0x10,
	0x89,0xab,0xcd,0xef,0x01,0x23,0x45,0x67 };
*/

static unsigned short bytebit[8]	= {
	01, 02, 04, 010, 020, 040, 0100, 0200 };

static unsigned long bigbyte[24] = {
	0x800000L,	0x400000L,	0x200000L,	0x100000L,
	0x80000L,	0x40000L,	0x20000L,	0x10000L,
	0x8000L,	0x4000L,	0x2000L,	0x1000L,
	0x800L, 	0x400L, 	0x200L, 	0x100L,
	0x80L,		0x40L,		0x20L,		0x10L,
	0x8L,		0x4L,		0x2L,		0x1L	};

/* Use the key schedule specified in the Standard (ANSI X3.92-1981). */

static unsigned char pc1[56] = {
	56, 48, 40, 32, 24, 16,  8,	 0, 57, 49, 41, 33, 25, 17,
	 9,  1, 58, 50, 42, 34, 26,	18, 10,  2, 59, 51, 43, 35,
	62, 54, 46, 38, 30, 22, 14,	 6, 61, 53, 45, 37, 29, 21,
	13,  5, 60, 52, 44, 36, 28,	20, 12,  4, 27, 19, 11,  3 };

static unsigned char totrot[16] = {
	1,2,4,6,8,10,12,14,15,17,19,21,23,25,27,28 };

static unsigned char pc2[48] = {
	13, 16, 10, 23,  0,  4,  2, 27, 14,  5, 20,  9,
	22, 18, 11,  3, 25,  7, 15,  6, 26, 19, 12,  1,
	40, 51, 30, 36, 46, 54, 29, 39, 50, 44, 32, 47,
	43, 48, 38, 55, 33, 52, 45, 41, 49, 35, 28, 31 };

void deskey(key, edf)	/* Thanks to James Gillogly & Phil Karn! */
unsigned char *key;
int edf;
{
	register int i, j, l, m, n;
	unsigned char pc1m[56], pcr[56];
	unsigned long kn[32];

	for ( j = 0; j < 56; j++ ) {
		l = pc1[j];
		m = l & 07;
		pc1m[j] = (key[l >> 3] & bytebit[m]) ? 1 : 0;
		}
	for( i = 0; i < 16; i++ ) {
		if( edf == DE1 ) m = (15 - i) << 1;
		else m = i << 1;
		n = m + 1;
		kn[m] = kn[n] = 0L;
		for( j = 0; j < 28; j++ ) {
			l = j + totrot[i];
			if( l < 28 ) pcr[j] = pc1m[l];
			else pcr[j] = pc1m[l - 28];
			}
		for( j = 28; j < 56; j++ ) {
		    l = j + totrot[i];
		    if( l < 56 ) pcr[j] = pc1m[l];
		    else pcr[j] = pc1m[l - 28];
		    }
		for( j = 0; j < 24; j++ ) {
			if( pcr[pc2[j]] ) kn[m] |= bigbyte[j];
			if( pcr[pc2[j+24]] ) kn[n] |= bigbyte[j];
			}
		}
	cookey(kn);
	return;
	}

static void cookey(raw1)
register unsigned long *raw1;
{
	register unsigned long *cook, *raw0;
	unsigned long dough[32];
	register int i;

	cook = dough;
	for( i = 0; i < 16; i++, raw1++ ) {
		raw0 = raw1++;
		*cook	 = (*raw0 & 0x00fc0000L) << 6;
		*cook	|= (*raw0 & 0x00000fc0L) << 10;
		*cook	|= (*raw1 & 0x00fc0000L) >> 10;
		*cook++ |= (*raw1 & 0x00000fc0L) >> 6;
		*cook	 = (*raw0 & 0x0003f000L) << 12;
		*cook	|= (*raw0 & 0x0000003fL) << 16;
		*cook	|= (*raw1 & 0x0003f000L) >> 4;
		*cook++ |= (*raw1 & 0x0000003fL);
		}
	usekey(dough);
	return;
	}

void cpkey(into)
register unsigned long *into;
{
	register unsigned long *from, *endp;

	from = KnL, endp = &KnL[32];
	while( from < endp ) *into++ = *from++;
	return;
	}

void usekey(from)
register unsigned long *from;
{
	register unsigned long *to, *endp;

	to = KnL, endp = &KnL[32];
	while( to < endp ) *to++ = *from++;
	return;
	}

void des(inblock, outblock)
unsigned char *inblock, *outblock;
{
	unsigned long work[2];

	scrunch(inblock, work);
	desfunc(work, KnL);
	unscrun(work, outblock);
	return;
	}

static void scrunch(outof, into)
register unsigned char *outof;
register unsigned long *into;
{
	*into	 = (*outof++ & 0xffL) << 24;
	*into	|= (*outof++ & 0xffL) << 16;
	*into	|= (*outof++ & 0xffL) << 8;
	*into++ |= (*outof++ & 0xffL);
	*into	 = (*outof++ & 0xffL) << 24;
	*into	|= (*outof++ & 0xffL) << 16;
	*into	|= (*outof++ & 0xffL) << 8;
	*into	|= (*outof   & 0xffL);
	return;
	}

static void unscrun(outof, into)
register unsigned long *outof;
register unsigned char *into;
{
	*into++ = (unsigned char)((*outof >> 24) & 0xffL);
	*into++ = (unsigned char)((*outof >> 16) & 0xffL);
	*into++ = (unsigned char)((*outof >>  8) & 0xffL);
	*into++ = (unsigned char)( *outof++	 & 0xffL);
	*into++ = (unsigned char)((*outof >> 24) & 0xffL);
	*into++ = (unsigned char)((*outof >> 16) & 0xffL);
	*into++ = (unsigned char)((*outof >>  8) & 0xffL);
	*into	= (unsigned char)( *outof	 & 0xffL);
	return;
	}

static unsigned long SP1[64] = {
	0x01010400L, 0x00000000L, 0x00010000L, 0x01010404L,
	0x01010004L, 0x00010404L, 0x00000004L, 0x00010000L,
	0x00000400L, 0x01010400L, 0x01010404L, 0x00000400L,
	0x01000404L, 0x01010004L, 0x01000000L, 0x00000004L,
	0x00000404L, 0x01000400L, 0x01000400L, 0x00010400L,
	0x00010400L, 0x01010000L, 0x01010000L, 0x01000404L,
	0x00010004L, 0x01000004L, 0x01000004L, 0x00010004L,
	0x00000000L, 0x00000404L, 0x00010404L, 0x01000000L,
	0x00010000L, 0x01010404L, 0x00000004L, 0x01010000L,
	0x01010400L, 0x01000000L, 0x01000000L, 0x00000400L,
	0x01010004L, 0x00010000L, 0x00010400L, 0x01000004L,
	0x00000400L, 0x00000004L, 0x01000404L, 0x00010404L,
	0x01010404L, 0x00010004L, 0x01010000L, 0x01000404L,
	0x01000004L, 0x00000404L, 0x00010404L, 0x01010400L,
	0x00000404L, 0x01000400L, 0x01000400L, 0x00000000L,
	0x00010004L, 0x00010400L, 0x00000000L, 0x01010004L };

static unsigned long SP2[64] = {
	0x80108020L, 0x80008000L, 0x00008000L, 0x00108020L,
	0x00100000L, 0x00000020L, 0x80100020L, 0x80008020L,
	0x80000020L, 0x80108020L, 0x80108000L, 0x80000000L,
	0x80008000L, 0x00100000L, 0x00000020L, 0x80100020L,
	0x00108000L, 0x00100020L, 0x80008020L, 0x00000000L,
	0x80000000L, 0x00008000L, 0x00108020L, 0x80100000L,
	0x00100020L, 0x80000020L, 0x00000000L, 0x00108000L,
	0x00008020L, 0x80108000L, 0x80100000L, 0x00008020L,
	0x00000000L, 0x00108020L, 0x80100020L, 0x00100000L,
	0x80008020L, 0x80100000L, 0x80108000L, 0x00008000L,
	0x80100000L, 0x80008000L, 0x00000020L, 0x80108020L,
	0x00108020L, 0x00000020L, 0x00008000L, 0x80000000L,
	0x00008020L, 0x80108000L, 0x00100000L, 0x80000020L,
	0x00100020L, 0x80008020L, 0x80000020L, 0x00100020L,
	0x00108000L, 0x00000000L, 0x80008000L, 0x00008020L,
	0x80000000L, 0x80100020L, 0x80108020L, 0x00108000L };

static unsigned long SP3[64] = {
	0x00000208L, 0x08020200L, 0x00000000L, 0x08020008L,
	0x08000200L, 0x00000000L, 0x00020208L, 0x08000200L,
	0x00020008L, 0x08000008L, 0x08000008L, 0x00020000L,
	0x08020208L, 0x00020008L, 0x08020000L, 0x00000208L,
	0x08000000L, 0x00000008L, 0x08020200L, 0x00000200L,
	0x00020200L, 0x08020000L, 0x08020008L, 0x00020208L,
	0x08000208L, 0x00020200L, 0x00020000L, 0x08000208L,
	0x00000008L, 0x08020208L, 0x00000200L, 0x08000000L,
	0x08020200L, 0x08000000L, 0x00020008L, 0x00000208L,
	0x00020000L, 0x08020200L, 0x08000200L, 0x00000000L,
	0x00000200L, 0x00020008L, 0x08020208L, 0x08000200L,
	0x08000008L, 0x00000200L, 0x00000000L, 0x08020008L,
	0x08000208L, 0x00020000L, 0x08000000L, 0x08020208L,
	0x00000008L, 0x00020208L, 0x00020200L, 0x08000008L,
	0x08020000L, 0x08000208L, 0x00000208L, 0x08020000L,
	0x00020208L, 0x00000008L, 0x08020008L, 0x00020200L };

static unsigned long SP4[64] = {
	0x00802001L, 0x00002081L, 0x00002081L, 0x00000080L,
	0x00802080L, 0x00800081L, 0x00800001L, 0x00002001L,
	0x00000000L, 0x00802000L, 0x00802000L, 0x00802081L,
	0x00000081L, 0x00000000L, 0x00800080L, 0x00800001L,
	0x00000001L, 0x00002000L, 0x00800000L, 0x00802001L,
	0x00000080L, 0x00800000L, 0x00002001L, 0x00002080L,
	0x00800081L, 0x00000001L, 0x00002080L, 0x00800080L,
	0x00002000L, 0x00802080L, 0x00802081L, 0x00000081L,
	0x00800080L, 0x00800001L, 0x00802000L, 0x00802081L,
	0x00000081L, 0x00000000L, 0x00000000L, 0x00802000L,
	0x00002080L, 0x00800080L, 0x00800081L, 0x00000001L,
	0x00802001L, 0x00002081L, 0x00002081L, 0x00000080L,
	0x00802081L, 0x00000081L, 0x00000001L, 0x00002000L,
	0x00800001L, 0x00002001L, 0x00802080L, 0x00800081L,
	0x00002001L, 0x00002080L, 0x00800000L, 0x00802001L,
	0x00000080L, 0x00800000L, 0x00002000L, 0x00802080L };

static unsigned long SP5[64] = {
	0x00000100L, 0x02080100L, 0x02080000L, 0x42000100L,
	0x00080000L, 0x00000100L, 0x40000000L, 0x02080000L,
	0x40080100L, 0x00080000L, 0x02000100L, 0x40080100L,
	0x42000100L, 0x42080000L, 0x00080100L, 0x40000000L,
	0x02000000L, 0x40080000L, 0x40080000L, 0x00000000L,
	0x40000100L, 0x42080100L, 0x42080100L, 0x02000100L,
	0x42080000L, 0x40000100L, 0x00000000L, 0x42000000L,
	0x02080100L, 0x02000000L, 0x42000000L, 0x00080100L,
	0x00080000L, 0x42000100L, 0x00000100L, 0x02000000L,
	0x40000000L, 0x02080000L, 0x42000100L, 0x40080100L,
	0x02000100L, 0x40000000L, 0x42080000L, 0x02080100L,
	0x40080100L, 0x00000100L, 0x02000000L, 0x42080000L,
	0x42080100L, 0x00080100L, 0x42000000L, 0x42080100L,
	0x02080000L, 0x00000000L, 0x40080000L, 0x42000000L,
	0x00080100L, 0x02000100L, 0x40000100L, 0x00080000L,
	0x00000000L, 0x40080000L, 0x02080100L, 0x40000100L };

static unsigned long SP6[64] = {
	0x20000010L, 0x20400000L, 0x00004000L, 0x20404010L,
	0x20400000L, 0x00000010L, 0x20404010L, 0x00400000L,
	0x20004000L, 0x00404010L, 0x00400000L, 0x20000010L,
	0x00400010L, 0x20004000L, 0x20000000L, 0x00004010L,
	0x00000000L, 0x00400010L, 0x20004010L, 0x00004000L,
	0x00404000L, 0x20004010L, 0x00000010L, 0x20400010L,
	0x20400010L, 0x00000000L, 0x00404010L, 0x20404000L,
	0x00004010L, 0x00404000L, 0x20404000L, 0x20000000L,
	0x20004000L, 0x00000010L, 0x20400010L, 0x00404000L,
	0x20404010L, 0x00400000L, 0x00004010L, 0x20000010L,
	0x00400000L, 0x20004000L, 0x20000000L, 0x00004010L,
	0x20000010L, 0x20404010L, 0x00404000L, 0x20400000L,
	0x00404010L, 0x20404000L, 0x00000000L, 0x20400010L,
	0x00000010L, 0x00004000L, 0x20400000L, 0x00404010L,
	0x00004000L, 0x00400010L, 0x20004010L, 0x00000000L,
	0x20404000L, 0x20000000L, 0x00400010L, 0x20004010L };

static unsigned long SP7[64] = {
	0x00200000L, 0x04200002L, 0x04000802L, 0x00000000L,
	0x00000800L, 0x04000802L, 0x00200802L, 0x04200800L,
	0x04200802L, 0x00200000L, 0x00000000L, 0x04000002L,
	0x00000002L, 0x04000000L, 0x04200002L, 0x00000802L,
	0x04000800L, 0x00200802L, 0x00200002L, 0x04000800L,
	0x04000002L, 0x04200000L, 0x04200800L, 0x00200002L,
	0x04200000L, 0x00000800L, 0x00000802L, 0x04200802L,
	0x00200800L, 0x00000002L, 0x04000000L, 0x00200800L,
	0x04000000L, 0x00200800L, 0x00200000L, 0x04000802L,
	0x04000802L, 0x04200002L, 0x04200002L, 0x00000002L,
	0x00200002L, 0x04000000L, 0x04000800L, 0x00200000L,
	0x04200800L, 0x00000802L, 0x00200802L, 0x04200800L,
	0x00000802L, 0x04000002L, 0x04200802L, 0x04200000L,
	0x00200800L, 0x00000000L, 0x00000002L, 0x04200802L,
	0x00000000L, 0x00200802L, 0x04200000L, 0x00000800L,
	0x04000002L, 0x04000800L, 0x00000800L, 0x00200002L };

static unsigned long SP8[64] = {
	0x10001040L, 0x00001000L, 0x00040000L, 0x10041040L,
	0x10000000L, 0x10001040L, 0x00000040L, 0x10000000L,
	0x00040040L, 0x10040000L, 0x10041040L, 0x00041000L,
	0x10041000L, 0x00041040L, 0x00001000L, 0x00000040L,
	0x10040000L, 0x10000040L, 0x10001000L, 0x00001040L,
	0x00041000L, 0x00040040L, 0x10040040L, 0x10041000L,
	0x00001040L, 0x00000000L, 0x00000000L, 0x10040040L,
	0x10000040L, 0x10001000L, 0x00041040L, 0x00040000L,
	0x00041040L, 0x00040000L, 0x10041000L, 0x00001000L,
	0x00000040L, 0x10040040L, 0x00001000L, 0x00041040L,
	0x10001000L, 0x00000040L, 0x10000040L, 0x10040000L,
	0x10040040L, 0x10000000L, 0x00040000L, 0x10001040L,
	0x00000000L, 0x10041040L, 0x00040040L, 0x10000040L,
	0x10040000L, 0x10001000L, 0x10001040L, 0x00000000L,
	0x10041040L, 0x00041000L, 0x00041000L, 0x00001040L,
	0x00001040L, 0x00040040L, 0x10000000L, 0x10041000L };

static void desfunc(block, keys)
register unsigned long *block, *keys;
{
	register unsigned long fval, work, right, leftt;
	register int round;

	leftt = block[0];
	right = block[1];
	work = ((leftt >> 4) ^ right) & 0x0f0f0f0fL;
	right ^= work;
	leftt ^= (work << 4);
	work = ((leftt >> 16) ^ right) & 0x0000ffffL;
	right ^= work;
	leftt ^= (work << 16);
	work = ((right >> 2) ^ leftt) & 0x33333333L;
	leftt ^= work;
	right ^= (work << 2);
	work = ((right >> 8) ^ leftt) & 0x00ff00ffL;
	leftt ^= work;
	right ^= (work << 8);
	right = ((right << 1) | ((right >> 31) & 1L)) & 0xffffffffL;
	work = (leftt ^ right) & 0xaaaaaaaaL;
	leftt ^= work;
	right ^= work;
	leftt = ((leftt << 1) | ((leftt >> 31) & 1L)) & 0xffffffffL;

	for( round = 0; round < 8; round++ ) {
		work  = (right << 28) | (right >> 4);
		work ^= *keys++;
		fval  = SP7[ work		 & 0x3fL];
		fval |= SP5[(work >>  8) & 0x3fL];
		fval |= SP3[(work >> 16) & 0x3fL];
		fval |= SP1[(work >> 24) & 0x3fL];
		work  = right ^ *keys++;
		fval |= SP8[ work		 & 0x3fL];
		fval |= SP6[(work >>  8) & 0x3fL];
		fval |= SP4[(work >> 16) & 0x3fL];
		fval |= SP2[(work >> 24) & 0x3fL];
		leftt ^= fval;
		work  = (leftt << 28) | (leftt >> 4);
		work ^= *keys++;
		fval  = SP7[ work		 & 0x3fL];
		fval |= SP5[(work >>  8) & 0x3fL];
		fval |= SP3[(work >> 16) & 0x3fL];
		fval |= SP1[(work >> 24) & 0x3fL];
		work  = leftt ^ *keys++;
		fval |= SP8[ work		 & 0x3fL];
		fval |= SP6[(work >>  8) & 0x3fL];
		fval |= SP4[(work >> 16) & 0x3fL];
		fval |= SP2[(work >> 24) & 0x3fL];
		right ^= fval;
		}

	right = (right << 31) | (right >> 1);
	work = (leftt ^ right) & 0xaaaaaaaaL;
	leftt ^= work;
	right ^= work;
	leftt = (leftt << 31) | (leftt >> 1);
	work = ((leftt >> 8) ^ right) & 0x00ff00ffL;
	right ^= work;
	leftt ^= (work << 8);
	work = ((leftt >> 2) ^ right) & 0x33333333L;
	right ^= work;
	leftt ^= (work << 2);
	work = ((right >> 16) ^ leftt) & 0x0000ffffL;
	leftt ^= work;
	right ^= (work << 16);
	work = ((right >> 4) ^ leftt) & 0x0f0f0f0fL;
	leftt ^= work;
	right ^= (work << 4);
	*block++ = right;
	*block = leftt;
	return;
	}

/* Validation sets:
 *
 * Single-length key, single-length plaintext -
 * Key	  : 0123 4567 89ab cdef
 * Plain  : 0123 4567 89ab cde7
 * Cipher : c957 4425 6a5e d31d
 *
 * Double-length key, single-length plaintext -
 * Key	  : 0123 4567 89ab cdef fedc ba98 7654 3210
 * Plain  : 0123 4567 89ab cde7
 * Cipher : 7f1d 0a77 826b 8aff
 *
 * Double-length key, double-length plaintext -
 * Key	  : 0123 4567 89ab cdef fedc ba98 7654 3210
 * Plain  : 0123 4567 89ab cdef 0123 4567 89ab cdff
 * Cipher : 27a0 8440 406a df60 278f 47cf 42d6 15d7
 *
 * Triple-length key, single-length plaintext -
 * Key	  : 0123 4567 89ab cdef fedc ba98 7654 3210 89ab cdef 0123 4567
 * Plain  : 0123 4567 89ab cde7
 * Cipher : de0b 7c06 ae5e 0ed5
 *
 * Triple-length key, double-length plaintext -
 * Key	  : 0123 4567 89ab cdef fedc ba98 7654 3210 89ab cdef 0123 4567
 * Plain  : 0123 4567 89ab cdef 0123 4567 89ab cdff
 * Cipher : ad0d 1b30 ac17 cf07 0ed1 1c63 81e4 4de5
 *
 * d3des V5.0a rwo 9208.07 18:44 Graven Imagery
 **********************************************************************/

[-- Attachment #4: d3des.h --]
[-- Type: text/plain, Size: 1659 bytes --]

#ifndef D3DES_H
#define D3DES_H

/*
 * This is D3DES (V5.09) by Richard Outerbridge with the double and
 * triple-length support removed for use in VNC.
 *
 * These changes are:
 *  Copyright (C) 1999 AT&T Laboratories Cambridge.  All Rights Reserved.
 *
 * This software is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 */

/* d3des.h -
 *
 *	Headers and defines for d3des.c
 *	Graven Imagery, 1992.
 *
 * Copyright (c) 1988,1989,1990,1991,1992 by Richard Outerbridge
 *	(GEnie : OUTER; CIS : [71755,204])
 */

#define EN0	0	/* MODE == encrypt */
#define DE1	1	/* MODE == decrypt */

extern void deskey(unsigned char *, int);
/*		      hexkey[8]     MODE
 * Sets the internal key register according to the hexadecimal
 * key contained in the 8 bytes of hexkey, according to the DES,
 * for encryption or decryption according to MODE.
 */

extern void usekey(unsigned long *);
/*		    cookedkey[32]
 * Loads the internal key register with the data in cookedkey.
 */

extern void cpkey(unsigned long *);
/*		   cookedkey[32]
 * Copies the contents of the internal key register into the storage
 * located at &cookedkey[0].
 */

extern void des(unsigned char *, unsigned char *);
/*		    from[8]	      to[8]
 * Encrypts/Decrypts (according to the key currently loaded in the
 * internal key register) one block of eight bytes at address 'from'
 * into the block at address 'to'.  They can be the same.
 */

/* d3des.h V5.09 rwo 9208.04 15:06 Graven Imagery
 ********************************************************************/

#endif

^ permalink raw reply	[flat|nested] 6+ messages in thread