Outreachy bite-sized tasks

Outreachy bite-sized tasks

[Roger Pau Monné]

Hello,

First of all, thanks for your interest in the Xen Project, and for wanting 
to participate in Outreachy.

Both of you have expressed interest in the "QEMU xen-blkback performance 
analysis and improvements" Outreachy project, and AFAIK both of you still 
need to perform your initial contribution.

I've found a couple of small tasks that you can perform and should allow 
you to complete your initial contribution to the project:

 - The first one is related to xenalyze, and it consist in creating a 
   header file that can be shared by both the Xen kernel and xenalyze.
   You will need to move the TRC_ defines found in sched_credit.c and 
   sched_credit2.c to a header that's shared with xenalyze and then 
   replace the usage of TRC_SCHED_CLASS_EVT with the defines in the header 
   file [0].

 - The second one consist in fixing the return codes of certain xl 
   commands. There are commands in xl that will return 0 (SUCCESS) even 
   when failing, which makes it very hard to write scripts that make use 
   of xl. A list of those commands can be found in [1], together with some 
   preliminary patches. Please note that those patches have comments that 
   you will need to address, and that you should also need to preserve the 
   original authorship of the patches plus yours.

I encourage you to read the wiki page about sending patches to xen-devel 
[2], it should guide you through your first steps on using git and 
creating suitable patches.

Also, please note that this is an open source project, so you will need to 
coordinate in order to figure out which task are you going to take, in 
order to avoid clashes or duplication of efforts.

If you have further questions, either reply to this thread (keeping 
the xen-devel mailing list on the Cc), or feel free to start another one 
if you think it's more suited.

Roger.

[0] http://lists.xenproject.org/archives/html/xen-devel/2016-02/msg02624.html
[1] http://lists.xenproject.org/archives/html/xen-devel/2015-12/msg02246.html
[2] http://wiki.xenproject.org/wiki/Submitting_Xen_Project_Patches

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Outreachy bite-sized tasks

[Paulina Szubarczyk]

[-- Attachment #1.1: Type: text/plain, Size: 2416 bytes --]

Hi,

Thank you for the proposed tasks. I would like to work on the second one,
fixing the return codes in xl.

Regards,
Paulina Szubarczyk

On 23 March 2016 at 16:32, Roger Pau Monné <roger.pau@citrix.com> wrote:

> Hello,
>
> First of all, thanks for your interest in the Xen Project, and for wanting
> to participate in Outreachy.
>
> Both of you have expressed interest in the "QEMU xen-blkback performance
> analysis and improvements" Outreachy project, and AFAIK both of you still
> need to perform your initial contribution.
>
> I've found a couple of small tasks that you can perform and should allow
> you to complete your initial contribution to the project:
>
>  - The first one is related to xenalyze, and it consist in creating a
>    header file that can be shared by both the Xen kernel and xenalyze.
>    You will need to move the TRC_ defines found in sched_credit.c and
>    sched_credit2.c to a header that's shared with xenalyze and then
>    replace the usage of TRC_SCHED_CLASS_EVT with the defines in the header
>    file [0].
>
>  - The second one consist in fixing the return codes of certain xl
>    commands. There are commands in xl that will return 0 (SUCCESS) even
>    when failing, which makes it very hard to write scripts that make use
>    of xl. A list of those commands can be found in [1], together with some
>    preliminary patches. Please note that those patches have comments that
>    you will need to address, and that you should also need to preserve the
>    original authorship of the patches plus yours.
>
> I encourage you to read the wiki page about sending patches to xen-devel
> [2], it should guide you through your first steps on using git and
> creating suitable patches.
>
> Also, please note that this is an open source project, so you will need to
> coordinate in order to figure out which task are you going to take, in
> order to avoid clashes or duplication of efforts.
>
> If you have further questions, either reply to this thread (keeping
> the xen-devel mailing list on the Cc), or feel free to start another one
> if you think it's more suited.
>
> Roger.
>
> [0]
> http://lists.xenproject.org/archives/html/xen-devel/2016-02/msg02624.html
> [1]
> http://lists.xenproject.org/archives/html/xen-devel/2015-12/msg02246.html
> [2] http://wiki.xenproject.org/wiki/Submitting_Xen_Project_Patches
>

[-- Attachment #1.2: Type: text/html, Size: 3307 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Outreachy bite-sized tasks

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 712 bytes --]

Hey,

please, do not use HTML for emails to this list.

On Wed, 2016-03-23 at 17:38 +0100, Paulina Szubarczyk wrote:
> Hi, 
> 
> Thank you for the proposed tasks. I would like to work on the second
> one, 
> fixing the return codes in xl.
> 
I just wanted to say that, since I've done (and mentored) some similar
activity before, so, if you go for this, feel free to ask and/or to Cc
me to the patches as well. :-)

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Outreachy bite-sized tasks

[Paulina Szubarczyk]

[-- Attachment #1.1: Type: text/plain, Size: 2200 bytes --]

Hi Roger,

An another point that needs to be filled up in the application is timeline.
I made the proposition of it, I am not sure if it should be more complex.
Could you look at it in your free time?

Timeline:

22 April - 22 May [Before the intership]
 * Elaborate performance measurement methodology. That may include putting
trace
    points using TRDCS and performing read operation with varying
block sizes on
    different storage devices as in [3].
 * Gain a profound knowledge about implementation of xen-blkback in
Qemu and Linux kernel.

23 May - 6 June [Begin of the intership - up to two weeks]
 * Prepare and improve performance measurement test.
 * Gather and document the results.

6 June - 23 August
 * Base on the analysis and the task description implement performance
improvement future
   inside of QEMU's xen-blkback.
 * Prepare and run regression tests and continue performance measurement
for
   each implemented feature.
 * Document the results.

15 August - 23 August [Close to the end]
 * Sum up the work:
   - indicate the obstacles
   - form the conclusion and
   - indicate future work.

References:
[1] http://lxr.free-electrons.com/source/drivers/block/xen-blkback/blkback.c
[2]
http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/include/public/io/blkif.h
[3]
https://events.linuxfoundation.org/sites/events/files/slides/20131025%20-%20Storage%20Performance%20PDF.pdf


On 23 March 2016 at 19:21, Dario Faggioli <dario.faggioli@citrix.com> wrote:
> Hey,
>
> please, do not use HTML for emails to this list.
>
> On Wed, 2016-03-23 at 17:38 +0100, Paulina Szubarczyk wrote:
>> Hi,
>>
>> Thank you for the proposed tasks. I would like to work on the second
>> one,
>> fixing the return codes in xl.
>>
> I just wanted to say that, since I've done (and mentored) some similar
> activity before, so, if you go for this, feel free to ask and/or to Cc
> me to the patches as well. :-)
>
> Regards,
> Dario
> --
> <<This happens because I choose it to happen!>> (Raistlin Majere)
> -----------------------------------------------------------------
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
>

[-- Attachment #1.2: Type: text/html, Size: 2991 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Outreachy bite-sized tasks

[Roger Pau Monné]

[-- Attachment #1: Type: text/plain, Size: 2581 bytes --]

Please don't top post, it breaks the flow of the conversation.

I'm also adding Anthony (one of the QEMU/Xen maintainers).
On Fri, 1 Apr 2016, Paulina Szubarczyk wrote:

> Hi Roger,
> 
> An another point that needs to be filled up in the application is timeline.
> I made the proposition of it, I am not sure if it should be more complex. Could you look at it in your free time?
> 
> Timeline:
> 
> 22 April - 22 May [Before the intership]
>  * Elaborate performance measurement methodology. That may include putting trace
>     points using TRDCS and performing read operation with varying block sizes on 
>     different storage devices as in [3].

Could you elaborate on TRDCS? I've tried googling for it, but nothing came 
up.

Also, I think it would be good to write down which tool are you going to 
use in order to perform those measurements, and how/which block sizes are 
you going to test. Also keep into account that apart from block sizes you 
also have to take into account the number of parallel workers and the 
queue depth of each one.

Regarding the different storage devices, we usually did benchmarks using a 
ramdisk, but Linux intorduced a null-block device [0] not long ago that I 
think could be used to model different storage devices without actually 
having to buy them.

I don't expect that you do all this now, but it is important to think 
about the different scenarios/benchmarks and plan ahead, so that you 
don't end up running benchmarks that are not useful.

>  * Gain a profound knowledge about implementation of xen-blkback in Qemu and Linux kernel.
> 
> 23 May - 6 June [Begin of the intership - up to two weeks]
>  * Prepare and improve performance measurement test.
>  * Gather and document the results.
> 
> 6 June - 23 August
>  * Base on the analysis and the task description implement performance improvement future 
>    inside of QEMU's xen-blkback.
>  * Prepare and run regression tests and continue performance measurement for 
>    each implemented feature.
>  * Document the results.
> 
> 15 August - 23 August [Close to the end]
>  * Sum up the work:
>    - indicate the obstacles
>    - form the conclusion and
>    - indicate future work.
> 
> References:
> [1] http://lxr.free-electrons.com/source/drivers/block/xen-blkback/blkback.c
> [2] http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/include/public/io/blkif.h
> [3] https://events.linuxfoundation.org/sites/events/files/slides/20131025%20-%20Storage%20Performance%20PDF.pdf

[0] https://www.kernel.org/doc/Documentation/block/null_blk.txt

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Outreachy bite-sized tasks

[Paulina Szubarczyk]

On Fri, 2016-04-01 at 15:35 +0200, Roger Pau Monné wrote:
> Please don't top post, it breaks the flow of the conversation.
> 
> I'm also adding Anthony (one of the QEMU/Xen maintainers).
> On Fri, 1 Apr 2016, Paulina Szubarczyk wrote:
> 
> > Hi Roger,
> > 
> > An another point that needs to be filled up in the application is timeline.
> > I made the proposition of it, I am not sure if it should be more complex. Could you look at it in your free time?
> > 
> > Timeline:
> > 
> > 22 April - 22 May [Before the intership]
> >  * Elaborate performance measurement methodology. That may include putting trace
> >     points using TRDCS and performing read operation with varying block sizes on 
> >     different storage devices as in [3].
> 
> Could you elaborate on TRDCS? I've tried googling for it, but nothing came 
> up.
I mistyped TRDCS, I meant putting trace points in the application to
measure the time spent in each place as shown in the talk[3]. But is I
did more research concerning the measurement I think that the good idea
would be using fio and make similar test as Adriana did during her
OWP[4] when she worked on implementing multi-queue support for
xen_backend in Linux.  

> Also, I think it would be good to write down which tool are you going to 
> use in order to perform those measurements, and how/which block sizes are 
> you going to test. Also keep into account that apart from block sizes you 
> also have to take into account the number of parallel workers and the 
> queue depth of each one.
> 
> Regarding the different storage devices, we usually did benchmarks using a 
> ramdisk, but Linux intorduced a null-block device [0] not long ago that I 
> think could be used to model different storage devices without actually 
> having to buy them.
I make myself familiar with the protocol and I debug the qdisk during
attaching one of ram devices to follow how it works. I also wondered if
maybe I could make a change to 'xl create' such as when it is run in the
qemu debug mode doesn't terminate when Device Mode isn't ready yet,
because sometimes is hard to catch the moment when the connection to
gdbserver is possible.

Whereas I saw the implementation of indirect descriptors and the
multi-queue support by creating multiple rings, I couldn't find the
information of grant copy. Do you mean by that hypervisor-driven copy of
grant pages between domains instead of mapping them? 
> 
> I don't expect that you do all this now, but it is important to think 
> about the different scenarios/benchmarks and plan ahead, so that you 
> don't end up running benchmarks that are not useful.
> 
> >  * Gain a profound knowledge about implementation of xen-blkback in Qemu and Linux kernel.
> > 
> > 23 May - 6 June [Begin of the intership - up to two weeks]
> >  * Prepare and improve performance measurement test.
> >  * Gather and document the results.
> > 
> > 6 June - 23 August
> >  * Base on the analysis and the task description implement performance improvement future 
> >    inside of QEMU's xen-blkback.
> >  * Prepare and run regression tests and continue performance measurement for 
> >    each implemented feature.
> >  * Document the results.
> > 
> > 15 August - 23 August [Close to the end]
> >  * Sum up the work:
> >    - indicate the obstacles
> >    - form the conclusion and
> >    - indicate future work.
> > 
> > References:
> > [1] http://lxr.free-electrons.com/source/drivers/block/xen-blkback/blkback.c
> > [2] http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/include/public/io/blkif.h
> > [3] https://events.linuxfoundation.org/sites/events/files/slides/20131025%20-%20Storage%20Performance%20PDF.pdf
> 
> [0] https://www.kernel.org/doc/Documentation/block/null_blk.txt
[4] https://github.com/ariava/xen-blk-mq-benchmark

Paulina


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Outreachy bite-sized tasks

[Konrad Rzeszutek Wilk]

On Wed, Apr 20, 2016 at 12:06:48PM +0200, Paulina Szubarczyk wrote:
> On Fri, 2016-04-01 at 15:35 +0200, Roger Pau Monné wrote:
> > Please don't top post, it breaks the flow of the conversation.
> > 
> > I'm also adding Anthony (one of the QEMU/Xen maintainers).
> > On Fri, 1 Apr 2016, Paulina Szubarczyk wrote:
> > 
> > > Hi Roger,
> > > 
> > > An another point that needs to be filled up in the application is timeline.
> > > I made the proposition of it, I am not sure if it should be more complex. Could you look at it in your free time?
> > > 
> > > Timeline:
> > > 
> > > 22 April - 22 May [Before the intership]
> > >  * Elaborate performance measurement methodology. That may include putting trace
> > >     points using TRDCS and performing read operation with varying block sizes on 
> > >     different storage devices as in [3].
> > 
> > Could you elaborate on TRDCS? I've tried googling for it, but nothing came 
> > up.
> I mistyped TRDCS, I meant putting trace points in the application to
> measure the time spent in each place as shown in the talk[3]. But is I
> did more research concerning the measurement I think that the good idea
> would be using fio and make similar test as Adriana did during her
> OWP[4] when she worked on implementing multi-queue support for
> xen_backend in Linux.  
> 
> > Also, I think it would be good to write down which tool are you going to 
> > use in order to perform those measurements, and how/which block sizes are 
> > you going to test. Also keep into account that apart from block sizes you 
> > also have to take into account the number of parallel workers and the 
> > queue depth of each one.
> > 
> > Regarding the different storage devices, we usually did benchmarks using a 
> > ramdisk, but Linux intorduced a null-block device [0] not long ago that I 
> > think could be used to model different storage devices without actually 
> > having to buy them.
> I make myself familiar with the protocol and I debug the qdisk during
> attaching one of ram devices to follow how it works. I also wondered if
> maybe I could make a change to 'xl create' such as when it is run in the
> qemu debug mode doesn't terminate when Device Mode isn't ready yet,
> because sometimes is hard to catch the moment when the connection to
> gdbserver is possible.
> 
> Whereas I saw the implementation of indirect descriptors and the
> multi-queue support by creating multiple rings, I couldn't find the
> information of grant copy. Do you mean by that hypervisor-driven copy of
> grant pages between domains instead of mapping them? 

Yup. The hypervisor does the memcpy. The grant mapping system has an impact
that is felt when you are done with the grant - you need to unmap it.

And when you unmap it - you need to flush out the virtual address out of
the guests vCPUS (if the guests are runninig) - otherwise the CPU could
use an stale virtual address (they are cached) and reference an memory
that now points to somewhere else. Hence the unmap means also doing an
TLB shootdown which is expensive as it flushes out the CPU cache.

There have been improvements in this - David Vrabel had posted some patches
for this I think..

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel