From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([140.186.70.92]:59695) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RAl2q-00046E-Pd for qemu-devel@nongnu.org; Mon, 03 Oct 2011 12:07:12 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RAl2p-0006Ry-KT for qemu-devel@nongnu.org; Mon, 03 Oct 2011 12:07:08 -0400 Received: from e35.co.us.ibm.com ([32.97.110.153]:36520) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RAl2p-0006Rt-D5 for qemu-devel@nongnu.org; Mon, 03 Oct 2011 12:07:07 -0400 Received: from d03relay05.boulder.ibm.com (d03relay05.boulder.ibm.com [9.17.195.107]) by e35.co.us.ibm.com (8.14.4/8.13.1) with ESMTP id p93Fjo3x001256 for ; Mon, 3 Oct 2011 09:45:50 -0600 Received: from d03av03.boulder.ibm.com (d03av03.boulder.ibm.com [9.17.195.169]) by d03relay05.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id p93FFdWS100036 for ; Mon, 3 Oct 2011 10:06:43 -0600 Received: from d03av03.boulder.ibm.com (loopback [127.0.0.1]) by d03av03.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id p93Eu0dS030607 for ; Mon, 3 Oct 2011 08:56:00 -0600 Message-ID: <4E89CCF1.3030603@us.ibm.com> Date: Mon, 03 Oct 2011 09:55:45 -0500 From: Anthony Liguori MIME-Version: 1.0 References: <1316443309-23843-1-git-send-email-mdroth@linux.vnet.ibm.com> <20111003064653.GA15380@redhat.com> <4E89AFB4.8000103@us.ibm.com> <20111003132445.GB18920@redhat.com> <4E89BC1A.30208@codemonkey.ws> <20111003141539.GC19689@redhat.com> In-Reply-To: <20111003141539.GC19689@redhat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [RFC] New Migration Protocol using Visitor Interface List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Michael S. Tsirkin" Cc: Michael Roth , qemu-devel@nongnu.org On 10/03/2011 09:15 AM, Michael S. Tsirkin wrote: > On Mon, Oct 03, 2011 at 08:43:54AM -0500, Anthony Liguori wrote: >> Having the ability to ignore some fields is not enough. > > But it is also really required. I agree. It's the principle of being conservative in what you send and liberal in what you accept. >> We need to >> also be able to split a single field into multiple fields, and event >> split a single device into multiple devices. If we're dealing with >> a tree data structure in memory, we have limitless ability to fudge >> things. >> >> Visitors buy us this. Right now, we're talking about: >> >> void virtio_marshal(QEMUFile *f, VirtioDevice *vdev); >> >> If you just do BER encoding, that doesn't help us. We still can >> only read from the wire to a VirtioDevice. If we do: >> >> void virtio_marshal(Visitor *v, VirtioDevice *vdev); >> >> We can still do BER encoding. We can also do QEMUFile encoding in >> the current format. >> >> But we also can Visit from a QObject. That means we can write >> something that reads in BER off the wire and generates QObjects >> corresponding to whatever is sent. We can then manipulate those >> QObjects in memory as much as we'd like before handing a QObject >> Visitor to virtio_marshal. >> >> Separating parsing off the wire from handing something to the device >> models and allowing transformation in between is the key to >> addressing compatibility in a robust fashion. > > Yes. I agree it is required, and doing this with visitors > does looks like a good way to handle that. > > However the ability to skip fields at the protocol level > is also required, and it needs to be handled at the > protocol level. For example, I am worried about > invalid input with a huge array size making us allocate > and read a gigabyte into memory on the assumption that we will want to > manipulate it later. Yes we could introduce artificial > buffer size limits but these might then come to bite us. > If instead we can see that no one needs the array > up front, it will be easy to discard. Yes, we just were covering in the other parts of the thread. When you end_struct, end_array, or end_list, the visitor can discard whatevers left. How I see this all evolving in the future is that we would have a formal protocol specification. From that spec, we would generate Visitors. This would handle taking what's on the wire and building an in-memory tree. If an ASN.1 sequence that isn't part of the spec is passed, we should just silently discard it. I think the scenario above is that we have an array in the spec, but the implementation no longer uses that array. Your concern is that its inefficient to keep it in memory? I think there are two cases. The first is that that array is required in the spec, but optional in our implementation and that decision is made at run time. In this case, it needs to get entered into memory. If the array is no longer needed at all in our implementation, we can certainly annotate the protocol spec to skip it and not bring it into memory. Maybe it's an internal spec annotation or something. All that said, this doesn't seem to be a terribly important problem to solve IMHO. Regards, Anthony Liguori >