Qemu supports smart cards and xen 4.3 can be compiled with upstream
qemu-xen and it allows smart card support
http://wiki.qemu.org/Features/Smartcard
Maybe another option is spice protocol as it supports usb redirection and
smart cards
http://docs.cslabs.clarkson.edu/wiki/SPICE