Ceph Community Lab: Understanding where the QA suites spend time

Ceph Community Lab: Understanding where the QA suites spend time

[Mark Nelson]

Hi Guys,

About a month ago we were going through the process of trying to figure 
out how to replace some of the hardware in the community laboratory that 
runs all of the nightly Teuthology tests.  Given a limited budget to 
replace the existing nodes, we wanted to understand how the current QA 
suites actually spend time on the hardware.  To do this, we investigated 
the composition of suites, the amount of time that is spent in each 
suite, and then a deeper dive into how the most resource intensive jobs 
spend their time.  We then wrote a new benchmark for CBT to run 
ceph-test-rados in a rather naive reproduction of what the ceph task in 
teuthology does and tested it against several different storage device 
configurations to see how much benefit SSDs in the nodes may provide.

A couple of folks at the Hackathon were interested in the paper we 
wrote.  I thought I would share it with the community as well in case 
any folks ever wondered what Teuthology actually spends it's time doing.

http://nhm.ceph.com/community/Ceph_Community_Lab_Performance_Investigation.pdf

Thanks,
Mark

Re: [Cbt] Ceph Community Lab: Understanding where the QA suites spend time

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 2329 bytes --]

Hi,

To help put things in perspective (please forgive me if it's a fact already known to everyone), > 95% of the teuthology jobs can run on virtual machines with 8GB RAM, 40GB disk, 2 processors and no attached disks. The cost of running such virtual machines in the cloud currently is 0.033 euros per hour. Running 40 teuthology jobs at all times would require an average of 80 such virtual machines (most jobs use 2 machines, some 1 or 3), that is 0.033 * 80 vms * 24 hours * 31 days == ~2000 euros per month (also note that at least one cloud provider offers 50% discount if such virtual machines are reserved full time). That's 3 to 6 times cheaper than AWS. 

I'm not sure how much it costs to run these 40 teuthology jobs on a hand made lab but I'd be surprised if it was cheaper, all included. Whatever hardware already exists in the sepia lab can easily handle the remaining 5% of jobs that require actual hardware for one reason or the other.

My 3.3cts ;-)

On 19/08/2015 21:46, Mark Nelson wrote:
> Hi Guys,
> 
> About a month ago we were going through the process of trying to figure out how to replace some of the hardware in the community laboratory that runs all of the nightly Teuthology tests.  Given a limited budget to replace the existing nodes, we wanted to understand how the current QA suites actually spend time on the hardware.  To do this, we investigated the composition of suites, the amount of time that is spent in each suite, and then a deeper dive into how the most resource intensive jobs spend their time.  We then wrote a new benchmark for CBT to run ceph-test-rados in a rather naive reproduction of what the ceph task in teuthology does and tested it against several different storage device configurations to see how much benefit SSDs in the nodes may provide.
> 
> A couple of folks at the Hackathon were interested in the paper we wrote.  I thought I would share it with the community as well in case any folks ever wondered what Teuthology actually spends it's time doing.
> 
> http://nhm.ceph.com/community/Ceph_Community_Lab_Performance_Investigation.pdf
> 
> Thanks,
> Mark
> _______________________________________________
> Cbt mailing list
> Cbt@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/cbt-ceph.com

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [Cbt] Ceph Community Lab: Understanding where the QA suites spend time

[Mark Nelson]

I have no idea about the price of cloud computing services, but if you 
can find setups that give you access to SSDs you may be able to run 
through thrasher tests much faster (cheaper?). :)

Mark

On 08/19/2015 05:05 PM, Loic Dachary wrote:
> Hi,
>
> To help put things in perspective (please forgive me if it's a fact already known to everyone), > 95% of the teuthology jobs can run on virtual machines with 8GB RAM, 40GB disk, 2 processors and no attached disks. The cost of running such virtual machines in the cloud currently is 0.033 euros per hour. Running 40 teuthology jobs at all times would require an average of 80 such virtual machines (most jobs use 2 machines, some 1 or 3), that is 0.033 * 80 vms * 24 hours * 31 days == ~2000 euros per month (also note that at least one cloud provider offers 50% discount if such virtual machines are reserved full time). That's 3 to 6 times cheaper than AWS.
>
> I'm not sure how much it costs to run these 40 teuthology jobs on a hand made lab but I'd be surprised if it was cheaper, all included. Whatever hardware already exists in the sepia lab can easily handle the remaining 5% of jobs that require actual hardware for one reason or the other.
>
> My 3.3cts ;-)
>
> On 19/08/2015 21:46, Mark Nelson wrote:
>> Hi Guys,
>>
>> About a month ago we were going through the process of trying to figure out how to replace some of the hardware in the community laboratory that runs all of the nightly Teuthology tests.  Given a limited budget to replace the existing nodes, we wanted to understand how the current QA suites actually spend time on the hardware.  To do this, we investigated the composition of suites, the amount of time that is spent in each suite, and then a deeper dive into how the most resource intensive jobs spend their time.  We then wrote a new benchmark for CBT to run ceph-test-rados in a rather naive reproduction of what the ceph task in teuthology does and tested it against several different storage device configurations to see how much benefit SSDs in the nodes may provide.
>>
>> A couple of folks at the Hackathon were interested in the paper we wrote.  I thought I would share it with the community as well in case any folks ever wondered what Teuthology actually spends it's time doing.
>>
>> http://nhm.ceph.com/community/Ceph_Community_Lab_Performance_Investigation.pdf
>>
>> Thanks,
>> Mark
>> _______________________________________________
>> Cbt mailing list
>> Cbt@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/cbt-ceph.com
>

Re: [Cbt] Ceph Community Lab: Understanding where the QA suites spend time

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 3001 bytes --]



On 20/08/2015 00:08, Mark Nelson wrote:> I have no idea about the price of cloud computing services, but if you can find setups that give you access to SSDs you may be able to run through thrasher tests much faster (cheaper?). :)

In my experience (~2 month experience, not much really ;-), VMs are twice slower than bare metal on average (sometime it's only sightly slower, sometime it's 3 times slower). The network variance also counts : I've seen providers take ~20 minutes to install the Ceph packages (it's ~2GB altogether).

My 3.3 * ~2 cts ;-)

> Mark
> 
> On 08/19/2015 05:05 PM, Loic Dachary wrote:
>> Hi,
>>
>> To help put things in perspective (please forgive me if it's a fact already known to everyone), > 95% of the teuthology jobs can run on virtual machines with 8GB RAM, 40GB disk, 2 processors and no attached disks. The cost of running such virtual machines in the cloud currently is 0.033 euros per hour. Running 40 teuthology jobs at all times would require an average of 80 such virtual machines (most jobs use 2 machines, some 1 or 3), that is 0.033 * 80 vms * 24 hours * 31 days == ~2000 euros per month (also note that at least one cloud provider offers 50% discount if such virtual machines are reserved full time). That's 3 to 6 times cheaper than AWS.
>>
>> I'm not sure how much it costs to run these 40 teuthology jobs on a hand made lab but I'd be surprised if it was cheaper, all included. Whatever hardware already exists in the sepia lab can easily handle the remaining 5% of jobs that require actual hardware for one reason or the other.
>>
>> My 3.3cts ;-)
>>
>> On 19/08/2015 21:46, Mark Nelson wrote:
>>> Hi Guys,
>>>
>>> About a month ago we were going through the process of trying to figure out how to replace some of the hardware in the community laboratory that runs all of the nightly Teuthology tests.  Given a limited budget to replace the existing nodes, we wanted to understand how the current QA suites actually spend time on the hardware.  To do this, we investigated the composition of suites, the amount of time that is spent in each suite, and then a deeper dive into how the most resource intensive jobs spend their time.  We then wrote a new benchmark for CBT to run ceph-test-rados in a rather naive reproduction of what the ceph task in teuthology does and tested it against several different storage device configurations to see how much benefit SSDs in the nodes may provide.
>>>
>>> A couple of folks at the Hackathon were interested in the paper we wrote.  I thought I would share it with the community as well in case any folks ever wondered what Teuthology actually spends it's time doing.
>>>
>>> http://nhm.ceph.com/community/Ceph_Community_Lab_Performance_Investigation.pdf
>>>
>>> Thanks,
>>> Mark
>>> _______________________________________________
>>> Cbt mailing list
>>> Cbt@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/cbt-ceph.com
>>

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [Cbt] Ceph Community Lab: Understanding where the QA suites spend time

[Shinobu Kinjo]

> The network variance also counts :

Yes, it's quite true. We have to consider of that part.

Shinobu

----- Original Message -----
From: "Loic Dachary" <loic@dachary.org>
To: "Mark Nelson" <mnelson@redhat.com>, "ceph-devel" <ceph-devel@vger.kernel.org>, cbt@lists.ceph.com
Sent: Thursday, August 20, 2015 7:22:40 AM
Subject: Re: [Cbt] Ceph Community Lab: Understanding where the QA suites spend time



On 20/08/2015 00:08, Mark Nelson wrote:> I have no idea about the price of cloud computing services, but if you can find setups that give you access to SSDs you may be able to run through thrasher tests much faster (cheaper?). :)

In my experience (~2 month experience, not much really ;-), VMs are twice slower than bare metal on average (sometime it's only sightly slower, sometime it's 3 times slower). The network variance also counts : I've seen providers take ~20 minutes to install the Ceph packages (it's ~2GB altogether).

My 3.3 * ~2 cts ;-)

> Mark
> 
> On 08/19/2015 05:05 PM, Loic Dachary wrote:
>> Hi,
>>
>> To help put things in perspective (please forgive me if it's a fact already known to everyone), > 95% of the teuthology jobs can run on virtual machines with 8GB RAM, 40GB disk, 2 processors and no attached disks. The cost of running such virtual machines in the cloud currently is 0.033 euros per hour. Running 40 teuthology jobs at all times would require an average of 80 such virtual machines (most jobs use 2 machines, some 1 or 3), that is 0.033 * 80 vms * 24 hours * 31 days == ~2000 euros per month (also note that at least one cloud provider offers 50% discount if such virtual machines are reserved full time). That's 3 to 6 times cheaper than AWS.
>>
>> I'm not sure how much it costs to run these 40 teuthology jobs on a hand made lab but I'd be surprised if it was cheaper, all included. Whatever hardware already exists in the sepia lab can easily handle the remaining 5% of jobs that require actual hardware for one reason or the other.
>>
>> My 3.3cts ;-)
>>
>> On 19/08/2015 21:46, Mark Nelson wrote:
>>> Hi Guys,
>>>
>>> About a month ago we were going through the process of trying to figure out how to replace some of the hardware in the community laboratory that runs all of the nightly Teuthology tests.  Given a limited budget to replace the existing nodes, we wanted to understand how the current QA suites actually spend time on the hardware.  To do this, we investigated the composition of suites, the amount of time that is spent in each suite, and then a deeper dive into how the most resource intensive jobs spend their time.  We then wrote a new benchmark for CBT to run ceph-test-rados in a rather naive reproduction of what the ceph task in teuthology does and tested it against several different storage device configurations to see how much benefit SSDs in the nodes may provide.
>>>
>>> A couple of folks at the Hackathon were interested in the paper we wrote.  I thought I would share it with the community as well in case any folks ever wondered what Teuthology actually spends it's time doing.
>>>
>>> http://nhm.ceph.com/community/Ceph_Community_Lab_Performance_Investigation.pdf
>>>
>>> Thanks,
>>> Mark
>>> _______________________________________________
>>> Cbt mailing list
>>> Cbt@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/cbt-ceph.com
>>

-- 
Loïc Dachary, Artisan Logiciel Libre

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html