Re: [PATCH v3 13/22] libqtest: make qtest_bufwrite send "atomic"

From: Alexander Oleinik <alxndr@bu.edu>
To: John Snow <jsnow@redhat.com>, Stefan Hajnoczi <stefanha@redhat.com>
Cc: Thomas Huth <thuth@redhat.com>,
	"pbonzini@redhat.com" <pbonzini@redhat.com>,
	"bsd@redhat.com" <bsd@redhat.com>,
	"qemu-devel@nongnu.org" <qemu-devel@nongnu.org>,
	Laurent Vivier <lvivier@redhat.com>
Subject: Re: [PATCH v3 13/22] libqtest: make qtest_bufwrite send "atomic"
Date: Thu, 19 Sep 2019 15:50:47 -0400	[thread overview]
Message-ID: <3e6476e5-7632-5e2d-5270-b14599ffba35@bu.edu> (raw)
In-Reply-To: <da63050e-73b2-d9ac-112b-75b9a7caa54d@redhat.com>

On 9/19/19 2:56 PM, John Snow wrote:
> 
> 
> On 9/19/19 6:37 AM, Stefan Hajnoczi wrote:
>> On Wed, Sep 18, 2019 at 11:19:40PM +0000, Oleinik, Alexander wrote:
>>> When using qtest "in-process" communication, qtest_sendf directly calls
>>> a function in the server (qtest.c). Combining the contents of the
>>> subsequent socket_sends into the qtest_sendf, makes it so the server can
>>> immediately handle the command, without building a local buffer and
>>> waiting for a newline.
>>>
>>> Signed-off-by: Alexander Oleinik <alxndr@bu.edu>
>>> ---
>>>   tests/libqtest.c | 4 +---
>>>   1 file changed, 1 insertion(+), 3 deletions(-)
>>>
>>> diff --git a/tests/libqtest.c b/tests/libqtest.c
>>> index 19feea9e17..d770462869 100644
>>> --- a/tests/libqtest.c
>>> +++ b/tests/libqtest.c
>>> @@ -1086,9 +1086,7 @@ void qtest_bufwrite(QTestState *s, uint64_t addr, const void *data, size_t size)
>>>       gchar *bdata;
>>>   
>>>       bdata = g_base64_encode(data, size);
>>> -    qtest_sendf(s, "b64write 0x%" PRIx64 " 0x%zx ", addr, size);
>>> -    socket_send(s->fd, bdata, strlen(bdata));
>>> -    socket_send(s->fd, "\n", 1);
>>> +    qtest_sendf(s, "b64write 0x%" PRIx64 " 0x%zx %s\n", addr, size, bdata);
>>>       qtest_rsp(s, 0);
>>>       g_free(bdata);
>>>   }
>>> -- 
>>> 2.23.0
>>
>> Cc John Snow, who added the b64write command.
>>
>> The downside to doing this is that sprintf-formatting needs to be
>> performed on the entire base64 buffer.  This slows things down slightly
>> and a larger temporary buffer needs to be allocated, but I'm not sure it
>> matters.
>>
> 
> *struggles to remember*
> 
> I guess I wanted something that had some space savings while maintaining
> some semblance of debuggability. This is almost certainly meant for AHCI
> tests where it's writing various patterns to large blocks of memory.
> 
> I doubt I really measured the performance of it, but it seemed like the
> way to go for transferring medium amounts of data at the time via the
> qtest protocol.
> 
> Looks like I am the only user of it, still:
> 
> tests/ahci-test.c:    qtest_bufwrite(ahci->parent->qts, ptr, tx, bufsize);
> tests/ahci-test.c:    qtest_bufwrite(ahci->parent->qts, ptr, tx, bufsize);
> tests/libqos/ahci.c:        qtest_bufwrite(ahci->parent->qts, ptr,
> buffer, bufsize);
> 
> The buffers can be quite large, so you might be re-buffering a decent
> amount of data from the sender now.
> 
> 1, Are large transfers like this guaranteed to be atomic anyway? What
> kind of socket is it? we're probably eclipsing frame and packet sizes here.
> 
> 2, I am not sure what being "atomic" affords us in terms of allowing the
> server to not wait for newlines, how does this change help?
> 
> --js
> 
I'm modifying qtest to allow the server and client to co-exist within
the same process (facilitating coverage-guided fuzzing). One of the
modifications is making qtest_sendf directly call a function in
qtest.c. All the other qtest commands are sent with a single
qtest_sendf call, so the qtest.c function could immediately call
qtest_process_command. This breaks if the command is sent with
different qtest_send/socket_send calls, as in b64write.

It should be simple to change qtest_server_inproc_recv (the qtest.c 
receiver) to
wait for an "\n" prior to qtest_process_command, so I will probably do 
that and
then normal(socket) qtest will keep the memory-reduction benefits of the
non-"atomic" approach.

As a side note, would qtest_memwrite, also benefit from splitting up the 
send
command?

      for (i = 0; i < size; i++) {
          sprintf(&enc[i * 2], "%02x", ptr[i]);
      }

      qtest_sendf(s, "write 0x%" PRIx64 " 0x%zx 0x%s\n", addr, size, enc);
      qtest_rsp(s, 0);
      g_free(enc);