From mboxrd@z Thu Jan 1 00:00:00 1970 From: Asias He Subject: Re: [RFC 0/5] Introduce VM Sockets virtio transport Date: Fri, 28 Jun 2013 14:26:58 +0800 Message-ID: <20130628062658.GC30441@hj.localdomain> References: <1372320004-20502-1-git-send-email-asias@redhat.com> <51CC8C65.10409@oracle.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Cc: Andy King , kvm@vger.kernel.org, Dmitry Torokhov , netdev@vger.kernel.org, Reilly Grant , virtualization@lists.linux-foundation.org, Pekka Enberg , "Michael S. Tsirkin" , "David S. Miller" To: Sasha Levin Return-path: Content-Disposition: inline In-Reply-To: <51CC8C65.10409@oracle.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: virtualization-bounces@lists.linux-foundation.org Errors-To: virtualization-bounces@lists.linux-foundation.org List-Id: netdev.vger.kernel.org On Thu, Jun 27, 2013 at 03:03:01PM -0400, Sasha Levin wrote: > Hi Asias, > > Looks nice! Some comments inline below (I've removed anything that mst already > commented on). Thanks. > On 06/27/2013 03:59 AM, Asias He wrote: > >Hello guys, > > > >In commit d021c344051af91 (VSOCK: Introduce VM Sockets), VMware added VM > >Sockets support. VM Sockets allows communication between virtual > >machines and the hypervisor. VM Sockets is able to use different > >hyervisor neutral transport to transfer data. Currently, only VMware > >VMCI transport is supported. > > > >This series introduces virtio transport for VM Sockets. > > > >Any comments are appreciated! Thanks! > > > >Code: > >========================= > >1) kernel bits > > git://github.com/asias/linux.git vsock > > > >2) userspace bits: > > git://github.com/asias/linux-kvm.git vsock > > > >Howto: > >========================= > >Make sure you have these kernel options: > > > > CONFIG_VSOCKETS=y > > CONFIG_VIRTIO_VSOCKETS=y > > CONFIG_VIRTIO_VSOCKETS_COMMON=y > > CONFIG_VHOST_VSOCK=m > > > >$ git clone git://github.com/asias/linux-kvm.git > >$ cd linux-kvm/tools/kvm > >$ co -b vsock origin/vsock > >$ make > >$ modprobe vhost_vsock > >$ ./lkvm run -d os.img -k bzImage --vsock guest_cid > > > >Test: > >========================= > >I hacked busybox's http server and wget to run over vsock. Start http > >server in host and guest, download a 512MB file in guest and host > >simultaneously for 6000 times. Manged to run the http stress test. > > > >Also, I wrote a small libvsock.so to play the LD_PRELOAD trick and > >managed to make sshd and ssh work over virito-vsock without modifying > >the source code. > > Why did it require hacking in the first place? Does running with kvmtool > and just doing regular networking over virtio-net running on top of vsock > achieves the same goal? VM Sockets introduces a new address family. We need to modify the source code of the application to use the new address family. Using vsock there is no need to use virtio-net at all. We can not run virtio-net on top of vsock. > >Draft VM Sockets Virtio Device spec: > >========================= > >Appendix K: VM Sockets Device > > > >The virtio VM sockets device is a virtio transport device for VM Sockets. VM > >Sockets allows communication between virtual machines and the hypervisor. > > > >Configuration: > > > >Subsystem Device ID 13 > > > >Virtqueues: > > 0:controlq; 1:receiveq0; 2:transmitq0 ... 2N+1:receivqN; 2N+2:transmitqN > > controlq is "defined but not used", is there something in mind for it? if not, > does it make sense keeping it here? we can always re-add it to the end just > like in virtio-net. In virtio-net, each queue pairs contains a controlq, I think it is not necessary for virtio-vsock. e.g. receiveq0 transmitq0 controlq0 In virtio-scsi, we have ctrlq evetq and N requestq. I want the similar layout here, to reserve a ctrl queue in the front. > >Feature bits: > > Currently, no feature bits are defined. > > > >Device configuration layout: > > > >Two configuration fields are currently defined. > > > > struct virtio_vsock_config { > > __u32 guest_cid; > > __u32 max_virtqueue_pairs; > > } __packed; > > > >The guest_cid field specifies the guest context id which likes the guest IP > >address. The max_virtqueue_pairs field specifies the maximum number of receive > >and transmit virtqueue pairs (receiveq0 ... receiveqN and transmitq0 ... > >transmitqN respectively; N = max_virtqueue_pairs - 1 ) that can be configured. > >The driver is free to use only one virtqueue pairs, or it can use more to > >achieve better performance. > > How does the driver tell the device how many vqs it's planning on actually using? > or is it assumed that all of them are in use? I want the driver to use all of them. > > > >Device Initialization: > >The initialization routine should discover the device's virtqueues. > > > >Device Operation: > >Packets are transmitted by placing them in the transmitq0..transmitqN, and > >buffers for incoming packets are placed in the receiveq0..receiveqN. In each > >case, the packet itself is preceded by a header: > > > > struct virtio_vsock_hdr { > > __u32 src_cid; > > __u32 src_port; > > __u32 dst_cid; > > __u32 dst_port; > > __u32 len; > > __u8 type; > > __u8 op; > > __u8 shut; > > __u64 fwd_cnt; > > __u64 buf_alloc; > > } __packed; > > > >src_cid and dst_cid: specify the source and destination context id. > >src_port and dst_port: specify the source and destination port. > >len: specifies the size of the data payload, it could be zero if no data > >payload is transferred. > >type: specifies the type of the packet, it can be SOCK_STREAM or SOCK_DGRAM. > >op: specifies the operation of the packet, it is defined as follows. > > > > enum { > > VIRTIO_VSOCK_OP_INVALID = 0, > > VIRTIO_VSOCK_OP_REQUEST = 1, > > VIRTIO_VSOCK_OP_NEGOTIATE = 2, > > VIRTIO_VSOCK_OP_OFFER = 3, > > VIRTIO_VSOCK_OP_ATTACH = 4, > > VIRTIO_VSOCK_OP_RW = 5, > > VIRTIO_VSOCK_OP_CREDIT = 6, > > VIRTIO_VSOCK_OP_RST = 7, > > VIRTIO_VSOCK_OP_SHUTDOWN = 8, > > }; > > > >shut: specifies the shutdown mode when the socket is being shutdown. 1 is for > >receive shutdown, 2 is for transmit shutdown, 3 is for both receive and transmit > >shutdown. > >fwd_cnt: specifies the the number of bytes the receiver has forwarded to userspace. > > For the previous packet? For the entire session? Reading ahead makes it clear but > it's worth mentioning here the context just to make it easy for implementers. Thanks. I will make this more clearer. > >buf_alloc: specifies the size of the receiver's recieve buffer in bytes. > receive > > >Virtio VM socket connection creation: > >1) Client sends VIRTIO_VSOCK_OP_REQUEST to server > >2) Server reponses with VIRTIO_VSOCK_OP_NEGOTIATE to client > >3) Client sends VIRTIO_VSOCK_OP_OFFER to server > >4) Server responses with VIRTIO_VSOCK_OP_ATTACH to client > > > >Virtio VM socket credit update: > >Virtio VM socket uses credit-based flow control. The sender maintains tx_cnt > >which counts the totoal number of bytes it has sent out, peer_fwd_cnt which > total > >counts the totoal number of byes the receiver has forwarded, and peer_buf_alloc > total > >which is the size of the receiver's receive buffer. The sender can send no more > >than the credit the receiver gives to the sender: credit = peer_buf_alloc - > > > Thanks, > Sasha > -- Asias