Heavy speed difference between rbd and custom pool

Heavy speed difference between rbd and custom pool

[Stefan Priebe]

Hello list,

i'm getting these rbd bench values for pool rbd. They're high and constant.
----------------------------- RBD pool
# rados -p rbd bench 30 write -t 16
  Maintaining 16 concurrent writes of 4194304 bytes for at least 30 seconds.
    sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
      0       0         0         0         0         0         -         0
      1      16       274       258   1031.77      1032  0.043758 0.0602236
      2      16       549       533   1065.82      1100  0.072168 0.0590944
      3      16       825       809    1078.5      1104  0.040162  0.058682
      4      16      1103      1087   1086.84      1112  0.052508 0.0584277
      5      16      1385      1369   1095.04      1128  0.060233 0.0581288
      6      16      1654      1638   1091.85      1076  0.050697 0.0583385
      7      16      1939      1923   1098.71      1140  0.063716  0.057964
      8      16      2219      2203   1101.35      1120  0.055435 0.0579105
      9      16      2497      2481   1102.52      1112  0.060413 0.0578282
     10      16      2773      2757   1102.66      1104  0.051134 0.0578561
     11      16      3049      3033   1102.77      1104  0.057742 0.0578803
     12      16      3326      3310   1103.19      1108  0.053769 0.0578627
     13      16      3604      3588   1103.86      1112  0.064574 0.0578453
     14      16      3883      3867   1104.72      1116  0.056524 0.0578018
     15      16      4162      4146   1105.46      1116  0.054581 0.0577626
     16      16      4440      4424   1105.86      1112  0.079015  0.057758
     17      16      4725      4709   1107.86      1140  0.043511 0.0576647
     18      16      5007      4991   1108.97      1128  0.053005 0.0576147
     19      16      5292      5276    1110.6      1140  0.069004  0.057538
2012-06-18 23:36:19.124472min lat: 0.028568 max lat: 0.201941 avg lat: 
0.0574953
    sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
     20      16      5574      5558   1111.46      1128  0.048482 0.0574953
     21      16      5861      5845   1113.18      1148  0.051923 0.0574146
     22      16      6147      6131   1114.58      1144   0.04461 0.0573461
     23      16      6438      6422   1116.72      1164  0.050383 0.0572406
     24      16      6724      6708   1117.85      1144  0.067827 0.0571864
     25      16      7008      6992   1118.57      1136  0.049128  0.057147
     26      16      7296      7280   1119.85      1152  0.050331 0.0570879
     27      16      7573      7557    1119.4      1108  0.052711 0.0571132
     28      16      7858      7842   1120.13      1140  0.056369 0.0570764
     29      16      8143      8127   1120.81      1140  0.046558 0.0570438
     30      16      8431      8415   1121.85      1152  0.049958 0.0569942
  Total time run:         30.045481
Total writes made:      8431
Write size:             4194304
Bandwidth (MB/sec):     1122.432

Stddev Bandwidth:       26.0451
Max bandwidth (MB/sec): 1164
Min bandwidth (MB/sec): 1032
Average Latency:        0.0570069
Stddev Latency:         0.0128039
Max latency:            0.235536
Min latency:            0.028568
-----------------------------

I created then a custom pool called kvmpool.

~# ceph osd pool create kvmpool
pool 'kvmpool' created

But with this one i get slow and jumping values:
-------------------------------- kvmpool
~# rados -p kvmpool bench 30 write -t 16
  Maintaining 16 concurrent writes of 4194304 bytes for at least 30 seconds.
    sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
      0       0         0         0         0         0         -         0
      1      16       231       215   859.863       860  0.204867  0.069195
      2      16       393       377   753.899       648  0.049444 0.0811933
      3      16       535       519   691.908       568  0.232365 0.0899074
      4      16       634       618   617.913       396  0.032758 0.0963399
      5      16       806       790   631.913       688  0.075811  0.099529
      6      16       948       932   621.249       568  0.156988   0.10179
      7      16      1086      1070   611.348       552  0.036177  0.102064
      8      16      1206      1190   594.922       480  0.028491  0.105235
      9      16      1336      1320   586.589       520  0.041009  0.108735
     10      16      1512      1496    598.32       704  0.258165  0.105086
     11      16      1666      1650   599.921       616  0.040967  0.106146
     12      15      1825      1810   603.255       640  0.198851  0.105463
     13      16      1925      1909   587.309       396  0.042577  0.108449
     14      16      2135      2119   605.352       840  0.035767  0.105219
     15      16      2272      2256   601.523       548  0.246136  0.105357
     16      16      2426      2410   602.424       616   0.19881  0.105692
     17      16      2529      2513    591.22       412  0.031322  0.105463
     18      16      2696      2680    595.48       668  0.028081  0.106749
     19      16      2878      2862   602.449       728  0.044929  0.105856
2012-06-18 23:38:45.566094min lat: 0.023295 max lat: 0.763797 avg lat: 
0.105597
    sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
     20      16      3041      3025   604.921       652  0.036028  0.105597
     21      16      3182      3166   602.964       564  0.035072  0.104915
     22      16      3349      3333   605.916       668  0.030493  0.105304
     23      16      3512      3496   607.917       652  0.030523   0.10479
     24      16      3668      3652   608.584       624  0.232933   0.10475
     25      16      3821      3805   608.717       612  0.029881  0.104513
     26      16      3963      3947   607.148       568  0.050244   0.10531
     27      16      4112      4096   606.733       596  0.259069  0.105008
     28      16      4261      4245   606.347       596  0.211877  0.105215
     29      16      4437      4421   609.712       704   0.02802  0.104613
     30      16      4566      4550   606.586       516  0.047076  0.105111
  Total time run:         30.062141
Total writes made:      4566
Write size:             4194304
Bandwidth (MB/sec):     607.542

Stddev Bandwidth:       109.112
Max bandwidth (MB/sec): 860
Min bandwidth (MB/sec): 396
Average Latency:        0.10532
Stddev Latency:         0.108369
Max latency:            0.763797
Min latency:            0.023295
--------------------------------

Why do these pools differ? Where is the difference?

Stefan

Re: Heavy speed difference between rbd and custom pool

[Mark Nelson]

On 06/18/2012 04:39 PM, Stefan Priebe wrote:
> Hello list,
>
> i'm getting these rbd bench values for pool rbd. They're high and constant.
> ----------------------------- RBD pool
> # rados -p rbd bench 30 write -t 16
> Maintaining 16 concurrent writes of 4194304 bytes for at least 30 seconds.
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> 0 0 0 0 0 0 - 0
> 1 16 274 258 1031.77 1032 0.043758 0.0602236
> 2 16 549 533 1065.82 1100 0.072168 0.0590944
> 3 16 825 809 1078.5 1104 0.040162 0.058682
> 4 16 1103 1087 1086.84 1112 0.052508 0.0584277
> 5 16 1385 1369 1095.04 1128 0.060233 0.0581288
> 6 16 1654 1638 1091.85 1076 0.050697 0.0583385
> 7 16 1939 1923 1098.71 1140 0.063716 0.057964
> 8 16 2219 2203 1101.35 1120 0.055435 0.0579105
> 9 16 2497 2481 1102.52 1112 0.060413 0.0578282
> 10 16 2773 2757 1102.66 1104 0.051134 0.0578561
> 11 16 3049 3033 1102.77 1104 0.057742 0.0578803
> 12 16 3326 3310 1103.19 1108 0.053769 0.0578627
> 13 16 3604 3588 1103.86 1112 0.064574 0.0578453
> 14 16 3883 3867 1104.72 1116 0.056524 0.0578018
> 15 16 4162 4146 1105.46 1116 0.054581 0.0577626
> 16 16 4440 4424 1105.86 1112 0.079015 0.057758
> 17 16 4725 4709 1107.86 1140 0.043511 0.0576647
> 18 16 5007 4991 1108.97 1128 0.053005 0.0576147
> 19 16 5292 5276 1110.6 1140 0.069004 0.057538
> 2012-06-18 23:36:19.124472min lat: 0.028568 max lat: 0.201941 avg lat:
> 0.0574953
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> 20 16 5574 5558 1111.46 1128 0.048482 0.0574953
> 21 16 5861 5845 1113.18 1148 0.051923 0.0574146
> 22 16 6147 6131 1114.58 1144 0.04461 0.0573461
> 23 16 6438 6422 1116.72 1164 0.050383 0.0572406
> 24 16 6724 6708 1117.85 1144 0.067827 0.0571864
> 25 16 7008 6992 1118.57 1136 0.049128 0.057147
> 26 16 7296 7280 1119.85 1152 0.050331 0.0570879
> 27 16 7573 7557 1119.4 1108 0.052711 0.0571132
> 28 16 7858 7842 1120.13 1140 0.056369 0.0570764
> 29 16 8143 8127 1120.81 1140 0.046558 0.0570438
> 30 16 8431 8415 1121.85 1152 0.049958 0.0569942
> Total time run: 30.045481
> Total writes made: 8431
> Write size: 4194304
> Bandwidth (MB/sec): 1122.432
>
> Stddev Bandwidth: 26.0451
> Max bandwidth (MB/sec): 1164
> Min bandwidth (MB/sec): 1032
> Average Latency: 0.0570069
> Stddev Latency: 0.0128039
> Max latency: 0.235536
> Min latency: 0.028568
> -----------------------------
>
> I created then a custom pool called kvmpool.
>
> ~# ceph osd pool create kvmpool
> pool 'kvmpool' created
>
> But with this one i get slow and jumping values:
> -------------------------------- kvmpool
> ~# rados -p kvmpool bench 30 write -t 16
> Maintaining 16 concurrent writes of 4194304 bytes for at least 30 seconds.
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> 0 0 0 0 0 0 - 0
> 1 16 231 215 859.863 860 0.204867 0.069195
> 2 16 393 377 753.899 648 0.049444 0.0811933
> 3 16 535 519 691.908 568 0.232365 0.0899074
> 4 16 634 618 617.913 396 0.032758 0.0963399
> 5 16 806 790 631.913 688 0.075811 0.099529
> 6 16 948 932 621.249 568 0.156988 0.10179
> 7 16 1086 1070 611.348 552 0.036177 0.102064
> 8 16 1206 1190 594.922 480 0.028491 0.105235
> 9 16 1336 1320 586.589 520 0.041009 0.108735
> 10 16 1512 1496 598.32 704 0.258165 0.105086
> 11 16 1666 1650 599.921 616 0.040967 0.106146
> 12 15 1825 1810 603.255 640 0.198851 0.105463
> 13 16 1925 1909 587.309 396 0.042577 0.108449
> 14 16 2135 2119 605.352 840 0.035767 0.105219
> 15 16 2272 2256 601.523 548 0.246136 0.105357
> 16 16 2426 2410 602.424 616 0.19881 0.105692
> 17 16 2529 2513 591.22 412 0.031322 0.105463
> 18 16 2696 2680 595.48 668 0.028081 0.106749
> 19 16 2878 2862 602.449 728 0.044929 0.105856
> 2012-06-18 23:38:45.566094min lat: 0.023295 max lat: 0.763797 avg lat:
> 0.105597
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> 20 16 3041 3025 604.921 652 0.036028 0.105597
> 21 16 3182 3166 602.964 564 0.035072 0.104915
> 22 16 3349 3333 605.916 668 0.030493 0.105304
> 23 16 3512 3496 607.917 652 0.030523 0.10479
> 24 16 3668 3652 608.584 624 0.232933 0.10475
> 25 16 3821 3805 608.717 612 0.029881 0.104513
> 26 16 3963 3947 607.148 568 0.050244 0.10531
> 27 16 4112 4096 606.733 596 0.259069 0.105008
> 28 16 4261 4245 606.347 596 0.211877 0.105215
> 29 16 4437 4421 609.712 704 0.02802 0.104613
> 30 16 4566 4550 606.586 516 0.047076 0.105111
> Total time run: 30.062141
> Total writes made: 4566
> Write size: 4194304
> Bandwidth (MB/sec): 607.542
>
> Stddev Bandwidth: 109.112
> Max bandwidth (MB/sec): 860
> Min bandwidth (MB/sec): 396
> Average Latency: 0.10532
> Stddev Latency: 0.108369
> Max latency: 0.763797
> Min latency: 0.023295
> --------------------------------
>
> Why do these pools differ? Where is the difference?
>
> Stefan

Are the number of placement groups the same for each pool?

try running "ceph osd dump -o - | grep <pool>" and looking for the 
pg_num value.

Mark

Re: Heavy speed difference between rbd and custom pool

[Dan Mick]

Yes, this is almost certainly the problem.  When you create the pool, 
you can specify a pg count; the default is 8, which is quite low.
The count can't currently be adjusted after pool-creation time (we're 
working on an enhancement for that).

http://ceph.com/docs/master/control/  shows

ceph osd pool create POOL [pg_num [pgp_num]]

You'll want to set pg_num the same for similar pools in order to get for 
similar pool performance.

I note also that you can get that field directlty:
$ ceph osd pool get rbd pg_num
PG_NUM: 448

I have a 'nova' pool that was created with "pool create":

$ ceph osd pool get nova pg_num
PG_NUM: 8



On 06/18/2012 03:23 PM, Mark Nelson wrote:
> On 06/18/2012 04:39 PM, Stefan Priebe wrote:
>> Hello list,
>>
>> i'm getting these rbd bench values for pool rbd. They're high and
>> constant.
>> ----------------------------- RBD pool
>> # rados -p rbd bench 30 write -t 16
>> Maintaining 16 concurrent writes of 4194304 bytes for at least 30
>> seconds.
>> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
>> 0 0 0 0 0 0 - 0
>> 1 16 274 258 1031.77 1032 0.043758 0.0602236
>> 2 16 549 533 1065.82 1100 0.072168 0.0590944
>> 3 16 825 809 1078.5 1104 0.040162 0.058682
>> 4 16 1103 1087 1086.84 1112 0.052508 0.0584277
>> 5 16 1385 1369 1095.04 1128 0.060233 0.0581288
>> 6 16 1654 1638 1091.85 1076 0.050697 0.0583385
>> 7 16 1939 1923 1098.71 1140 0.063716 0.057964
>> 8 16 2219 2203 1101.35 1120 0.055435 0.0579105
>> 9 16 2497 2481 1102.52 1112 0.060413 0.0578282
>> 10 16 2773 2757 1102.66 1104 0.051134 0.0578561
>> 11 16 3049 3033 1102.77 1104 0.057742 0.0578803
>> 12 16 3326 3310 1103.19 1108 0.053769 0.0578627
>> 13 16 3604 3588 1103.86 1112 0.064574 0.0578453
>> 14 16 3883 3867 1104.72 1116 0.056524 0.0578018
>> 15 16 4162 4146 1105.46 1116 0.054581 0.0577626
>> 16 16 4440 4424 1105.86 1112 0.079015 0.057758
>> 17 16 4725 4709 1107.86 1140 0.043511 0.0576647
>> 18 16 5007 4991 1108.97 1128 0.053005 0.0576147
>> 19 16 5292 5276 1110.6 1140 0.069004 0.057538
>> 2012-06-18 23:36:19.124472min lat: 0.028568 max lat: 0.201941 avg lat:
>> 0.0574953
>> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
>> 20 16 5574 5558 1111.46 1128 0.048482 0.0574953
>> 21 16 5861 5845 1113.18 1148 0.051923 0.0574146
>> 22 16 6147 6131 1114.58 1144 0.04461 0.0573461
>> 23 16 6438 6422 1116.72 1164 0.050383 0.0572406
>> 24 16 6724 6708 1117.85 1144 0.067827 0.0571864
>> 25 16 7008 6992 1118.57 1136 0.049128 0.057147
>> 26 16 7296 7280 1119.85 1152 0.050331 0.0570879
>> 27 16 7573 7557 1119.4 1108 0.052711 0.0571132
>> 28 16 7858 7842 1120.13 1140 0.056369 0.0570764
>> 29 16 8143 8127 1120.81 1140 0.046558 0.0570438
>> 30 16 8431 8415 1121.85 1152 0.049958 0.0569942
>> Total time run: 30.045481
>> Total writes made: 8431
>> Write size: 4194304
>> Bandwidth (MB/sec): 1122.432
>>
>> Stddev Bandwidth: 26.0451
>> Max bandwidth (MB/sec): 1164
>> Min bandwidth (MB/sec): 1032
>> Average Latency: 0.0570069
>> Stddev Latency: 0.0128039
>> Max latency: 0.235536
>> Min latency: 0.028568
>> -----------------------------
>>
>> I created then a custom pool called kvmpool.
>>
>> ~# ceph osd pool create kvmpool
>> pool 'kvmpool' created
>>
>> But with this one i get slow and jumping values:
>> -------------------------------- kvmpool
>> ~# rados -p kvmpool bench 30 write -t 16
>> Maintaining 16 concurrent writes of 4194304 bytes for at least 30
>> seconds.
>> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
>> 0 0 0 0 0 0 - 0
>> 1 16 231 215 859.863 860 0.204867 0.069195
>> 2 16 393 377 753.899 648 0.049444 0.0811933
>> 3 16 535 519 691.908 568 0.232365 0.0899074
>> 4 16 634 618 617.913 396 0.032758 0.0963399
>> 5 16 806 790 631.913 688 0.075811 0.099529
>> 6 16 948 932 621.249 568 0.156988 0.10179
>> 7 16 1086 1070 611.348 552 0.036177 0.102064
>> 8 16 1206 1190 594.922 480 0.028491 0.105235
>> 9 16 1336 1320 586.589 520 0.041009 0.108735
>> 10 16 1512 1496 598.32 704 0.258165 0.105086
>> 11 16 1666 1650 599.921 616 0.040967 0.106146
>> 12 15 1825 1810 603.255 640 0.198851 0.105463
>> 13 16 1925 1909 587.309 396 0.042577 0.108449
>> 14 16 2135 2119 605.352 840 0.035767 0.105219
>> 15 16 2272 2256 601.523 548 0.246136 0.105357
>> 16 16 2426 2410 602.424 616 0.19881 0.105692
>> 17 16 2529 2513 591.22 412 0.031322 0.105463
>> 18 16 2696 2680 595.48 668 0.028081 0.106749
>> 19 16 2878 2862 602.449 728 0.044929 0.105856
>> 2012-06-18 23:38:45.566094min lat: 0.023295 max lat: 0.763797 avg lat:
>> 0.105597
>> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
>> 20 16 3041 3025 604.921 652 0.036028 0.105597
>> 21 16 3182 3166 602.964 564 0.035072 0.104915
>> 22 16 3349 3333 605.916 668 0.030493 0.105304
>> 23 16 3512 3496 607.917 652 0.030523 0.10479
>> 24 16 3668 3652 608.584 624 0.232933 0.10475
>> 25 16 3821 3805 608.717 612 0.029881 0.104513
>> 26 16 3963 3947 607.148 568 0.050244 0.10531
>> 27 16 4112 4096 606.733 596 0.259069 0.105008
>> 28 16 4261 4245 606.347 596 0.211877 0.105215
>> 29 16 4437 4421 609.712 704 0.02802 0.104613
>> 30 16 4566 4550 606.586 516 0.047076 0.105111
>> Total time run: 30.062141
>> Total writes made: 4566
>> Write size: 4194304
>> Bandwidth (MB/sec): 607.542
>>
>> Stddev Bandwidth: 109.112
>> Max bandwidth (MB/sec): 860
>> Min bandwidth (MB/sec): 396
>> Average Latency: 0.10532
>> Stddev Latency: 0.108369
>> Max latency: 0.763797
>> Min latency: 0.023295
>> --------------------------------
>>
>> Why do these pools differ? Where is the difference?
>>
>> Stefan
>
> Are the number of placement groups the same for each pool?
>
> try running "ceph osd dump -o - | grep <pool>" and looking for the
> pg_num value.
>
> Mark
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Heavy speed difference between rbd and custom pool

[Alexandre DERUMIER]

Hi Stephann
recommandations are 30-50 PGS by osd if I remember.

----- Mail original ----- 

De: "Mark Nelson" <mark.nelson@inktank.com> 
À: "Stefan Priebe" <s.priebe@profihost.ag> 
Cc: ceph-devel@vger.kernel.org 
Envoyé: Mardi 19 Juin 2012 00:23:49 
Objet: Re: Heavy speed difference between rbd and custom pool 

On 06/18/2012 04:39 PM, Stefan Priebe wrote: 
> Hello list, 
> 
> i'm getting these rbd bench values for pool rbd. They're high and constant. 
> ----------------------------- RBD pool 
> # rados -p rbd bench 30 write -t 16 
> Maintaining 16 concurrent writes of 4194304 bytes for at least 30 seconds. 
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 
> 0 0 0 0 0 0 - 0 
> 1 16 274 258 1031.77 1032 0.043758 0.0602236 
> 2 16 549 533 1065.82 1100 0.072168 0.0590944 
> 3 16 825 809 1078.5 1104 0.040162 0.058682 
> 4 16 1103 1087 1086.84 1112 0.052508 0.0584277 
> 5 16 1385 1369 1095.04 1128 0.060233 0.0581288 
> 6 16 1654 1638 1091.85 1076 0.050697 0.0583385 
> 7 16 1939 1923 1098.71 1140 0.063716 0.057964 
> 8 16 2219 2203 1101.35 1120 0.055435 0.0579105 
> 9 16 2497 2481 1102.52 1112 0.060413 0.0578282 
> 10 16 2773 2757 1102.66 1104 0.051134 0.0578561 
> 11 16 3049 3033 1102.77 1104 0.057742 0.0578803 
> 12 16 3326 3310 1103.19 1108 0.053769 0.0578627 
> 13 16 3604 3588 1103.86 1112 0.064574 0.0578453 
> 14 16 3883 3867 1104.72 1116 0.056524 0.0578018 
> 15 16 4162 4146 1105.46 1116 0.054581 0.0577626 
> 16 16 4440 4424 1105.86 1112 0.079015 0.057758 
> 17 16 4725 4709 1107.86 1140 0.043511 0.0576647 
> 18 16 5007 4991 1108.97 1128 0.053005 0.0576147 
> 19 16 5292 5276 1110.6 1140 0.069004 0.057538 
> 2012-06-18 23:36:19.124472min lat: 0.028568 max lat: 0.201941 avg lat: 
> 0.0574953 
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 
> 20 16 5574 5558 1111.46 1128 0.048482 0.0574953 
> 21 16 5861 5845 1113.18 1148 0.051923 0.0574146 
> 22 16 6147 6131 1114.58 1144 0.04461 0.0573461 
> 23 16 6438 6422 1116.72 1164 0.050383 0.0572406 
> 24 16 6724 6708 1117.85 1144 0.067827 0.0571864 
> 25 16 7008 6992 1118.57 1136 0.049128 0.057147 
> 26 16 7296 7280 1119.85 1152 0.050331 0.0570879 
> 27 16 7573 7557 1119.4 1108 0.052711 0.0571132 
> 28 16 7858 7842 1120.13 1140 0.056369 0.0570764 
> 29 16 8143 8127 1120.81 1140 0.046558 0.0570438 
> 30 16 8431 8415 1121.85 1152 0.049958 0.0569942 
> Total time run: 30.045481 
> Total writes made: 8431 
> Write size: 4194304 
> Bandwidth (MB/sec): 1122.432 
> 
> Stddev Bandwidth: 26.0451 
> Max bandwidth (MB/sec): 1164 
> Min bandwidth (MB/sec): 1032 
> Average Latency: 0.0570069 
> Stddev Latency: 0.0128039 
> Max latency: 0.235536 
> Min latency: 0.028568 
> ----------------------------- 
> 
> I created then a custom pool called kvmpool. 
> 
> ~# ceph osd pool create kvmpool 
> pool 'kvmpool' created 
> 
> But with this one i get slow and jumping values: 
> -------------------------------- kvmpool 
> ~# rados -p kvmpool bench 30 write -t 16 
> Maintaining 16 concurrent writes of 4194304 bytes for at least 30 seconds. 
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 
> 0 0 0 0 0 0 - 0 
> 1 16 231 215 859.863 860 0.204867 0.069195 
> 2 16 393 377 753.899 648 0.049444 0.0811933 
> 3 16 535 519 691.908 568 0.232365 0.0899074 
> 4 16 634 618 617.913 396 0.032758 0.0963399 
> 5 16 806 790 631.913 688 0.075811 0.099529 
> 6 16 948 932 621.249 568 0.156988 0.10179 
> 7 16 1086 1070 611.348 552 0.036177 0.102064 
> 8 16 1206 1190 594.922 480 0.028491 0.105235 
> 9 16 1336 1320 586.589 520 0.041009 0.108735 
> 10 16 1512 1496 598.32 704 0.258165 0.105086 
> 11 16 1666 1650 599.921 616 0.040967 0.106146 
> 12 15 1825 1810 603.255 640 0.198851 0.105463 
> 13 16 1925 1909 587.309 396 0.042577 0.108449 
> 14 16 2135 2119 605.352 840 0.035767 0.105219 
> 15 16 2272 2256 601.523 548 0.246136 0.105357 
> 16 16 2426 2410 602.424 616 0.19881 0.105692 
> 17 16 2529 2513 591.22 412 0.031322 0.105463 
> 18 16 2696 2680 595.48 668 0.028081 0.106749 
> 19 16 2878 2862 602.449 728 0.044929 0.105856 
> 2012-06-18 23:38:45.566094min lat: 0.023295 max lat: 0.763797 avg lat: 
> 0.105597 
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 
> 20 16 3041 3025 604.921 652 0.036028 0.105597 
> 21 16 3182 3166 602.964 564 0.035072 0.104915 
> 22 16 3349 3333 605.916 668 0.030493 0.105304 
> 23 16 3512 3496 607.917 652 0.030523 0.10479 
> 24 16 3668 3652 608.584 624 0.232933 0.10475 
> 25 16 3821 3805 608.717 612 0.029881 0.104513 
> 26 16 3963 3947 607.148 568 0.050244 0.10531 
> 27 16 4112 4096 606.733 596 0.259069 0.105008 
> 28 16 4261 4245 606.347 596 0.211877 0.105215 
> 29 16 4437 4421 609.712 704 0.02802 0.104613 
> 30 16 4566 4550 606.586 516 0.047076 0.105111 
> Total time run: 30.062141 
> Total writes made: 4566 
> Write size: 4194304 
> Bandwidth (MB/sec): 607.542 
> 
> Stddev Bandwidth: 109.112 
> Max bandwidth (MB/sec): 860 
> Min bandwidth (MB/sec): 396 
> Average Latency: 0.10532 
> Stddev Latency: 0.108369 
> Max latency: 0.763797 
> Min latency: 0.023295 
> -------------------------------- 
> 
> Why do these pools differ? Where is the difference? 
> 
> Stefan 

Are the number of placement groups the same for each pool? 

try running "ceph osd dump -o - | grep <pool>" and looking for the 
pg_num value. 

Mark 

-- 
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
the body of a message to majordomo@vger.kernel.org 
More majordomo info at http://vger.kernel.org/majordomo-info.html 



-- 

-- 




	Alexandre D erumier 
Ingénieur Système 
Fixe : 03 20 68 88 90 
Fax : 03 20 68 90 81 
45 Bvd du Général Leclerc 59100 Roubaix - France 
12 rue Marivaux 75002 Paris - France 
	
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Heavy speed difference between rbd and custom pool

[Stefan Priebe]

Ok thanks but wouldn't it make sense to  set the default to the same as rbd has? How is the value for rbd calculated? I've also seen that rbd has a different crushmap. What's the difference between crushmap 0 and 2?

Stefan

Am 19.06.2012 um 00:41 schrieb Dan Mick <dan.mick@inktank.com>:

> Yes, this is almost certainly the problem.  When you create the pool, you can specify a pg count; the default is 8, which is quite low.
> The count can't currently be adjusted after pool-creation time (we're working on an enhancement for that).
> 
> http://ceph.com/docs/master/control/  shows
> 
> ceph osd pool create POOL [pg_num [pgp_num]]
> 
> You'll want to set pg_num the same for similar pools in order to get for similar pool performance.
> 
> I note also that you can get that field directlty:
> $ ceph osd pool get rbd pg_num
> PG_NUM: 448
> 
> I have a 'nova' pool that was created with "pool create":
> 
> $ ceph osd pool get nova pg_num
> PG_NUM: 8
> 
> 
> 
> On 06/18/2012 03:23 PM, Mark Nelson wrote:
>> On 06/18/2012 04:39 PM, Stefan Priebe wrote:
>>> Hello list,
>>> 
>>> i'm getting these rbd bench values for pool rbd. They're high and
>>> constant.
>>> ----------------------------- RBD pool
>>> # rados -p rbd bench 30 write -t 16
>>> Maintaining 16 concurrent writes of 4194304 bytes for at least 30
>>> seconds.
>>> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
>>> 0 0 0 0 0 0 - 0
>>> 1 16 274 258 1031.77 1032 0.043758 0.0602236
>>> 2 16 549 533 1065.82 1100 0.072168 0.0590944
>>> 3 16 825 809 1078.5 1104 0.040162 0.058682
>>> 4 16 1103 1087 1086.84 1112 0.052508 0.0584277
>>> 5 16 1385 1369 1095.04 1128 0.060233 0.0581288
>>> 6 16 1654 1638 1091.85 1076 0.050697 0.0583385
>>> 7 16 1939 1923 1098.71 1140 0.063716 0.057964
>>> 8 16 2219 2203 1101.35 1120 0.055435 0.0579105
>>> 9 16 2497 2481 1102.52 1112 0.060413 0.0578282
>>> 10 16 2773 2757 1102.66 1104 0.051134 0.0578561
>>> 11 16 3049 3033 1102.77 1104 0.057742 0.0578803
>>> 12 16 3326 3310 1103.19 1108 0.053769 0.0578627
>>> 13 16 3604 3588 1103.86 1112 0.064574 0.0578453
>>> 14 16 3883 3867 1104.72 1116 0.056524 0.0578018
>>> 15 16 4162 4146 1105.46 1116 0.054581 0.0577626
>>> 16 16 4440 4424 1105.86 1112 0.079015 0.057758
>>> 17 16 4725 4709 1107.86 1140 0.043511 0.0576647
>>> 18 16 5007 4991 1108.97 1128 0.053005 0.0576147
>>> 19 16 5292 5276 1110.6 1140 0.069004 0.057538
>>> 2012-06-18 23:36:19.124472min lat: 0.028568 max lat: 0.201941 avg lat:
>>> 0.0574953
>>> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
>>> 20 16 5574 5558 1111.46 1128 0.048482 0.0574953
>>> 21 16 5861 5845 1113.18 1148 0.051923 0.0574146
>>> 22 16 6147 6131 1114.58 1144 0.04461 0.0573461
>>> 23 16 6438 6422 1116.72 1164 0.050383 0.0572406
>>> 24 16 6724 6708 1117.85 1144 0.067827 0.0571864
>>> 25 16 7008 6992 1118.57 1136 0.049128 0.057147
>>> 26 16 7296 7280 1119.85 1152 0.050331 0.0570879
>>> 27 16 7573 7557 1119.4 1108 0.052711 0.0571132
>>> 28 16 7858 7842 1120.13 1140 0.056369 0.0570764
>>> 29 16 8143 8127 1120.81 1140 0.046558 0.0570438
>>> 30 16 8431 8415 1121.85 1152 0.049958 0.0569942
>>> Total time run: 30.045481
>>> Total writes made: 8431
>>> Write size: 4194304
>>> Bandwidth (MB/sec): 1122.432
>>> 
>>> Stddev Bandwidth: 26.0451
>>> Max bandwidth (MB/sec): 1164
>>> Min bandwidth (MB/sec): 1032
>>> Average Latency: 0.0570069
>>> Stddev Latency: 0.0128039
>>> Max latency: 0.235536
>>> Min latency: 0.028568
>>> -----------------------------
>>> 
>>> I created then a custom pool called kvmpool.
>>> 
>>> ~# ceph osd pool create kvmpool
>>> pool 'kvmpool' created
>>> 
>>> But with this one i get slow and jumping values:
>>> -------------------------------- kvmpool
>>> ~# rados -p kvmpool bench 30 write -t 16
>>> Maintaining 16 concurrent writes of 4194304 bytes for at least 30
>>> seconds.
>>> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
>>> 0 0 0 0 0 0 - 0
>>> 1 16 231 215 859.863 860 0.204867 0.069195
>>> 2 16 393 377 753.899 648 0.049444 0.0811933
>>> 3 16 535 519 691.908 568 0.232365 0.0899074
>>> 4 16 634 618 617.913 396 0.032758 0.0963399
>>> 5 16 806 790 631.913 688 0.075811 0.099529
>>> 6 16 948 932 621.249 568 0.156988 0.10179
>>> 7 16 1086 1070 611.348 552 0.036177 0.102064
>>> 8 16 1206 1190 594.922 480 0.028491 0.105235
>>> 9 16 1336 1320 586.589 520 0.041009 0.108735
>>> 10 16 1512 1496 598.32 704 0.258165 0.105086
>>> 11 16 1666 1650 599.921 616 0.040967 0.106146
>>> 12 15 1825 1810 603.255 640 0.198851 0.105463
>>> 13 16 1925 1909 587.309 396 0.042577 0.108449
>>> 14 16 2135 2119 605.352 840 0.035767 0.105219
>>> 15 16 2272 2256 601.523 548 0.246136 0.105357
>>> 16 16 2426 2410 602.424 616 0.19881 0.105692
>>> 17 16 2529 2513 591.22 412 0.031322 0.105463
>>> 18 16 2696 2680 595.48 668 0.028081 0.106749
>>> 19 16 2878 2862 602.449 728 0.044929 0.105856
>>> 2012-06-18 23:38:45.566094min lat: 0.023295 max lat: 0.763797 avg lat:
>>> 0.105597
>>> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
>>> 20 16 3041 3025 604.921 652 0.036028 0.105597
>>> 21 16 3182 3166 602.964 564 0.035072 0.104915
>>> 22 16 3349 3333 605.916 668 0.030493 0.105304
>>> 23 16 3512 3496 607.917 652 0.030523 0.10479
>>> 24 16 3668 3652 608.584 624 0.232933 0.10475
>>> 25 16 3821 3805 608.717 612 0.029881 0.104513
>>> 26 16 3963 3947 607.148 568 0.050244 0.10531
>>> 27 16 4112 4096 606.733 596 0.259069 0.105008
>>> 28 16 4261 4245 606.347 596 0.211877 0.105215
>>> 29 16 4437 4421 609.712 704 0.02802 0.104613
>>> 30 16 4566 4550 606.586 516 0.047076 0.105111
>>> Total time run: 30.062141
>>> Total writes made: 4566
>>> Write size: 4194304
>>> Bandwidth (MB/sec): 607.542
>>> 
>>> Stddev Bandwidth: 109.112
>>> Max bandwidth (MB/sec): 860
>>> Min bandwidth (MB/sec): 396
>>> Average Latency: 0.10532
>>> Stddev Latency: 0.108369
>>> Max latency: 0.763797
>>> Min latency: 0.023295
>>> --------------------------------
>>> 
>>> Why do these pools differ? Where is the difference?
>>> 
>>> Stefan
>> 
>> Are the number of placement groups the same for each pool?
>> 
>> try running "ceph osd dump -o - | grep <pool>" and looking for the
>> pg_num value.
>> 
>> Mark
>> 
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Heavy speed difference between rbd and custom pool

[Stefan Priebe - Profihost AG]

Sorry i meant crush_ruleset

Am 19.06.2012 08:06, schrieb Stefan Priebe:
> Ok thanks but wouldn't it make sense to  set the default to the same as rbd has? How is the value for rbd calculated? I've also seen that rbd has a different crushmap. What's the difference between crushmap 0 and 2?
>
> Stefan
>
> Am 19.06.2012 um 00:41 schrieb Dan Mick<dan.mick@inktank.com>:
>
>> Yes, this is almost certainly the problem.  When you create the pool, you can specify a pg count; the default is 8, which is quite low.
>> The count can't currently be adjusted after pool-creation time (we're working on an enhancement for that).
>>
>> http://ceph.com/docs/master/control/  shows
>>
>> ceph osd pool create POOL [pg_num [pgp_num]]
>>
>> You'll want to set pg_num the same for similar pools in order to get for similar pool performance.
>>
>> I note also that you can get that field directlty:
>> $ ceph osd pool get rbd pg_num
>> PG_NUM: 448
>>
>> I have a 'nova' pool that was created with "pool create":
>>
>> $ ceph osd pool get nova pg_num
>> PG_NUM: 8
>>
>>
>>
>> On 06/18/2012 03:23 PM, Mark Nelson wrote:
>>> On 06/18/2012 04:39 PM, Stefan Priebe wrote:
>>>> Hello list,
>>>>
>>>> i'm getting these rbd bench values for pool rbd. They're high and
>>>> constant.
>>>> ----------------------------- RBD pool
>>>> # rados -p rbd bench 30 write -t 16
>>>> Maintaining 16 concurrent writes of 4194304 bytes for at least 30
>>>> seconds.
>>>> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
>>>> 0 0 0 0 0 0 - 0
>>>> 1 16 274 258 1031.77 1032 0.043758 0.0602236
>>>> 2 16 549 533 1065.82 1100 0.072168 0.0590944
>>>> 3 16 825 809 1078.5 1104 0.040162 0.058682
>>>> 4 16 1103 1087 1086.84 1112 0.052508 0.0584277
>>>> 5 16 1385 1369 1095.04 1128 0.060233 0.0581288
>>>> 6 16 1654 1638 1091.85 1076 0.050697 0.0583385
>>>> 7 16 1939 1923 1098.71 1140 0.063716 0.057964
>>>> 8 16 2219 2203 1101.35 1120 0.055435 0.0579105
>>>> 9 16 2497 2481 1102.52 1112 0.060413 0.0578282
>>>> 10 16 2773 2757 1102.66 1104 0.051134 0.0578561
>>>> 11 16 3049 3033 1102.77 1104 0.057742 0.0578803
>>>> 12 16 3326 3310 1103.19 1108 0.053769 0.0578627
>>>> 13 16 3604 3588 1103.86 1112 0.064574 0.0578453
>>>> 14 16 3883 3867 1104.72 1116 0.056524 0.0578018
>>>> 15 16 4162 4146 1105.46 1116 0.054581 0.0577626
>>>> 16 16 4440 4424 1105.86 1112 0.079015 0.057758
>>>> 17 16 4725 4709 1107.86 1140 0.043511 0.0576647
>>>> 18 16 5007 4991 1108.97 1128 0.053005 0.0576147
>>>> 19 16 5292 5276 1110.6 1140 0.069004 0.057538
>>>> 2012-06-18 23:36:19.124472min lat: 0.028568 max lat: 0.201941 avg lat:
>>>> 0.0574953
>>>> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
>>>> 20 16 5574 5558 1111.46 1128 0.048482 0.0574953
>>>> 21 16 5861 5845 1113.18 1148 0.051923 0.0574146
>>>> 22 16 6147 6131 1114.58 1144 0.04461 0.0573461
>>>> 23 16 6438 6422 1116.72 1164 0.050383 0.0572406
>>>> 24 16 6724 6708 1117.85 1144 0.067827 0.0571864
>>>> 25 16 7008 6992 1118.57 1136 0.049128 0.057147
>>>> 26 16 7296 7280 1119.85 1152 0.050331 0.0570879
>>>> 27 16 7573 7557 1119.4 1108 0.052711 0.0571132
>>>> 28 16 7858 7842 1120.13 1140 0.056369 0.0570764
>>>> 29 16 8143 8127 1120.81 1140 0.046558 0.0570438
>>>> 30 16 8431 8415 1121.85 1152 0.049958 0.0569942
>>>> Total time run: 30.045481
>>>> Total writes made: 8431
>>>> Write size: 4194304
>>>> Bandwidth (MB/sec): 1122.432
>>>>
>>>> Stddev Bandwidth: 26.0451
>>>> Max bandwidth (MB/sec): 1164
>>>> Min bandwidth (MB/sec): 1032
>>>> Average Latency: 0.0570069
>>>> Stddev Latency: 0.0128039
>>>> Max latency: 0.235536
>>>> Min latency: 0.028568
>>>> -----------------------------
>>>>
>>>> I created then a custom pool called kvmpool.
>>>>
>>>> ~# ceph osd pool create kvmpool
>>>> pool 'kvmpool' created
>>>>
>>>> But with this one i get slow and jumping values:
>>>> -------------------------------- kvmpool
>>>> ~# rados -p kvmpool bench 30 write -t 16
>>>> Maintaining 16 concurrent writes of 4194304 bytes for at least 30
>>>> seconds.
>>>> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
>>>> 0 0 0 0 0 0 - 0
>>>> 1 16 231 215 859.863 860 0.204867 0.069195
>>>> 2 16 393 377 753.899 648 0.049444 0.0811933
>>>> 3 16 535 519 691.908 568 0.232365 0.0899074
>>>> 4 16 634 618 617.913 396 0.032758 0.0963399
>>>> 5 16 806 790 631.913 688 0.075811 0.099529
>>>> 6 16 948 932 621.249 568 0.156988 0.10179
>>>> 7 16 1086 1070 611.348 552 0.036177 0.102064
>>>> 8 16 1206 1190 594.922 480 0.028491 0.105235
>>>> 9 16 1336 1320 586.589 520 0.041009 0.108735
>>>> 10 16 1512 1496 598.32 704 0.258165 0.105086
>>>> 11 16 1666 1650 599.921 616 0.040967 0.106146
>>>> 12 15 1825 1810 603.255 640 0.198851 0.105463
>>>> 13 16 1925 1909 587.309 396 0.042577 0.108449
>>>> 14 16 2135 2119 605.352 840 0.035767 0.105219
>>>> 15 16 2272 2256 601.523 548 0.246136 0.105357
>>>> 16 16 2426 2410 602.424 616 0.19881 0.105692
>>>> 17 16 2529 2513 591.22 412 0.031322 0.105463
>>>> 18 16 2696 2680 595.48 668 0.028081 0.106749
>>>> 19 16 2878 2862 602.449 728 0.044929 0.105856
>>>> 2012-06-18 23:38:45.566094min lat: 0.023295 max lat: 0.763797 avg lat:
>>>> 0.105597
>>>> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
>>>> 20 16 3041 3025 604.921 652 0.036028 0.105597
>>>> 21 16 3182 3166 602.964 564 0.035072 0.104915
>>>> 22 16 3349 3333 605.916 668 0.030493 0.105304
>>>> 23 16 3512 3496 607.917 652 0.030523 0.10479
>>>> 24 16 3668 3652 608.584 624 0.232933 0.10475
>>>> 25 16 3821 3805 608.717 612 0.029881 0.104513
>>>> 26 16 3963 3947 607.148 568 0.050244 0.10531
>>>> 27 16 4112 4096 606.733 596 0.259069 0.105008
>>>> 28 16 4261 4245 606.347 596 0.211877 0.105215
>>>> 29 16 4437 4421 609.712 704 0.02802 0.104613
>>>> 30 16 4566 4550 606.586 516 0.047076 0.105111
>>>> Total time run: 30.062141
>>>> Total writes made: 4566
>>>> Write size: 4194304
>>>> Bandwidth (MB/sec): 607.542
>>>>
>>>> Stddev Bandwidth: 109.112
>>>> Max bandwidth (MB/sec): 860
>>>> Min bandwidth (MB/sec): 396
>>>> Average Latency: 0.10532
>>>> Stddev Latency: 0.108369
>>>> Max latency: 0.763797
>>>> Min latency: 0.023295
>>>> --------------------------------
>>>>
>>>> Why do these pools differ? Where is the difference?
>>>>
>>>> Stefan
>>>
>>> Are the number of placement groups the same for each pool?
>>>
>>> try running "ceph osd dump -o - | grep<pool>" and looking for the
>>> pg_num value.
>>>
>>> Mark
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Heavy speed difference between rbd and custom pool

[Stefan Priebe - Profihost AG]

Am 19.06.2012 06:41, schrieb Alexandre DERUMIER:
> Hi Stephann
> recommandations are 30-50 PGS by osd if I remember.
>
rbd, data and metadata have 2176 PGs with 12 OSD. This is 181,333333333 
per OSD?!

Stefan

Re: Heavy speed difference between rbd and custom pool

[Mark Nelson]

On 06/19/2012 01:32 AM, Stefan Priebe - Profihost AG wrote:
> Am 19.06.2012 06:41, schrieb Alexandre DERUMIER:
>> Hi Stephann
>> recommandations are 30-50 PGS by osd if I remember.
>>
> rbd, data and metadata have 2176 PGs with 12 OSD. This is 181,333333333
> per OSD?!
>
> Stefan

That's probably fine, it just means that you will have a better 
pseudo-random distribution of OSD combinations (It does have higher 
cpu/memory overhead though).  Figuring out how many PGs you should have 
per OSD depends on a lot of factors including how many OSDs you have, 
how many nodes, CPU, memory, etc.  I'm guessing ~180 per OSD won't cause 
problems.  On the other hand, with low OSD counts you could probably 
have fewer and be fine too.

Mark

Re: Heavy speed difference between rbd and custom pool

[Stefan Priebe - Profihost AG]

Am 19.06.2012 15:01, schrieb Mark Nelson:
> On 06/19/2012 01:32 AM, Stefan Priebe - Profihost AG wrote:
>> Am 19.06.2012 06:41, schrieb Alexandre DERUMIER:
>>> Hi Stephann
>>> recommandations are 30-50 PGS by osd if I remember.
>>>
>> rbd, data and metadata have 2176 PGs with 12 OSD. This is 181,333333333
>> per OSD?!
>>
>> Stefan
>
> That's probably fine, it just means that you will have a better
> pseudo-random distribution of OSD combinations (It does have higher
> cpu/memory overhead though). Figuring out how many PGs you should have
> per OSD depends on a lot of factors including how many OSDs you have,
> how many nodes, CPU, memory, etc. I'm guessing ~180 per OSD won't cause
> problems. On the other hand, with low OSD counts you could probably have
> fewer and be fine too.

But this number 2176 of PGs were set while doing mkcephfs - how is it 
calculated?

Stefan

bad performance fio random write - rados bench random write to compare?

[Alexandre DERUMIER]

Hi,
Is it possible to do random write bench with rados bench command ?

I have very base random write performance with 4K block size inside qemu-kvm,
1000 iops/s max with 3 nodes with 3x 5 disk 15k
(Maybe it's related to my constant disk writes, like datas are not flushed sequentially to disk)

seekwatcher movie of 1 osd here :
http://odisoweb1.odiso.net/random-write-4k.mpg

I would like to do tests on kvm host with rados command to compare.
(I don't have rbd module in my kvm host kernel,so I can't use fio on the host)

Regards,

Alexandre

Re: Heavy speed difference between rbd and custom pool

[Sage Weil]

On Tue, 19 Jun 2012, Stefan Priebe - Profihost AG wrote:
> Am 19.06.2012 15:01, schrieb Mark Nelson:
> > On 06/19/2012 01:32 AM, Stefan Priebe - Profihost AG wrote:
> > > Am 19.06.2012 06:41, schrieb Alexandre DERUMIER:
> > > > Hi Stephann
> > > > recommandations are 30-50 PGS by osd if I remember.
> > > > 
> > > rbd, data and metadata have 2176 PGs with 12 OSD. This is 181,333333333
> > > per OSD?!
> > > 
> > > Stefan
> > 
> > That's probably fine, it just means that you will have a better
> > pseudo-random distribution of OSD combinations (It does have higher
> > cpu/memory overhead though). Figuring out how many PGs you should have
> > per OSD depends on a lot of factors including how many OSDs you have,
> > how many nodes, CPU, memory, etc. I'm guessing ~180 per OSD won't cause
> > problems. On the other hand, with low OSD counts you could probably have
> > fewer and be fine too.
> 
> But this number 2176 of PGs were set while doing mkcephfs - how is it
> calculated?

	num_pgs = num_osds << osd_pg_bits

which is configurable via --osd-pg-bits N or ceph.conf (at mkcephfs time).  
The default is 6.

sage

Re: Heavy speed difference between rbd and custom pool

[Stefan Priebe]

Am 19.06.2012 um 17:42 schrieb Sage Weil <sage@inktank.com>:
>> 
>> But this number 2176 of PGs were set while doing mkcephfs - how is it
>> calculated?
> 
>    num_pgs = num_osds << osd_pg_bits
> 
> which is configurable via --osd-pg-bits N or ceph.conf (at mkcephfs time).  
> The default is 6.
What happens if I add more osds later?

Re: Heavy speed difference between rbd and custom pool

[Sage Weil]

On Tue, 19 Jun 2012, Stefan Priebe wrote:
> Am 19.06.2012 um 17:42 schrieb Sage Weil <sage@inktank.com>:
> >> 
> >> But this number 2176 of PGs were set while doing mkcephfs - how is it
> >> calculated?
> > 
> >    num_pgs = num_osds << osd_pg_bits
> > 
> > which is configurable via --osd-pg-bits N or ceph.conf (at mkcephfs time).  
> > The default is 6.
>
> What happens if I add more osds later?

Currently, nothing.  The existing PGs are spread out among a larger number 
of OSDs.  This is partly why the default shoots a bit high.

One of the upcoming items on the todo list is to finish PG 
splitting/merging, which will allow a pool to be resharded into more or 
less PGs so that the data distribution can be adjusted as the cluster 
grows or shrinks.

sage

Re: Heavy speed difference between rbd and custom pool

[Dan Mick]

The number doesn't change currently (and can't currently be set manually). 

On Jun 19, 2012, at 9:24 AM, Stefan Priebe <s.priebe@profihost.ag> wrote:

> Am 19.06.2012 um 17:42 schrieb Sage Weil <sage@inktank.com>:
>>> 
>>> But this number 2176 of PGs were set while doing mkcephfs - how is it
>>> calculated?
>> 
>>   num_pgs = num_osds << osd_pg_bits
>> 
>> which is configurable via --osd-pg-bits N or ceph.conf (at mkcephfs time).  
>> The default is 6.
> What happens if I add more osds later?--
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Heavy speed difference between rbd and custom pool

[Stefan Priebe - Profihost AG]

Am 19.06.2012 18:29, schrieb Dan Mick:
> The number doesn't change currently (and can't currently be set manually).

OK thanks, so for extending the storage this will be pretty important. 
Do you have any plans for which version this feature will be complete?

Stefan

Re: Heavy speed difference between rbd and custom pool

[Dan Mick]

On 06/19/2012 11:46 PM, Stefan Priebe - Profihost AG wrote:
> Am 19.06.2012 18:29, schrieb Dan Mick:
>> The number doesn't change currently (and can't currently be set
>> manually).
>
> OK thanks, so for extending the storage this will be pretty important.
> Do you have any plans for which version this feature will be complete?
>
> Stefan

It is one of the next things on our list, and depends on the fixup of 
'pg splitting' (which once worked, but other features have made it
impractical without rework).  Until then we have tracker issues about 
the subject:

http://tracker.newdream.net/issues/1515
osd: pg split

http://tracker.newdream.net/issues/84
mon: auto adjust pg_num as pool grows

http://tracker.newdream.net/issues/85
osd: pg_num shrink

Re: bad performance fio random write - rados bench random write to compare?

[Gregory Farnum]

On Tue, Jun 19, 2012 at 7:05 AM, Alexandre DERUMIER <aderumier@odiso.com> wrote:
> Hi,
> Is it possible to do random write bench with rados bench command ?
>
> I have very base random write performance with 4K block size inside qemu-kvm,
> 1000 iops/s max with 3 nodes with 3x 5 disk 15k
> (Maybe it's related to my constant disk writes, like datas are not flushed sequentially to disk)
>
> seekwatcher movie of 1 osd here :
> http://odisoweb1.odiso.net/random-write-4k.mpg
>
> I would like to do tests on kvm host with rados command to compare.
> (I don't have rbd module in my kvm host kernel,so I can't use fio on the host)

Unfortunately, nobody's implemented a random write test in rados bench
yet (this might be a good bug for somebody who's interested in getting
in to the project — rados bench is pretty simple). I'd offer more
suggestions but I think we've already discussed this elsewhere. :)
-Greg
PS: Sorry this message got lost.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bad performance fio random write - rados bench random write to compare?

[Stefan Priebe - Profihost AG]

Am 02.07.2012 21:52, schrieb Gregory Farnum:
> On Tue, Jun 19, 2012 at 7:05 AM, Alexandre DERUMIER <aderumier@odiso.com> wrote:
>> Hi,
>> Is it possible to do random write bench with rados bench command ?
>>
>> I have very base random write performance with 4K block size inside qemu-kvm,
>> 1000 iops/s max with 3 nodes with 3x 5 disk 15k
>> (Maybe it's related to my constant disk writes, like datas are not flushed sequentially to disk)
>>
>> seekwatcher movie of 1 osd here :
>> http://odisoweb1.odiso.net/random-write-4k.mpg
>>
>> I would like to do tests on kvm host with rados command to compare.
>> (I don't have rbd module in my kvm host kernel,so I can't use fio on the host)
>
> Unfortunately, nobody's implemented a random write test in rados bench
> yet (this might be a good bug for somebody who's interested in getting
> in to the project — rados bench is pretty simple). I'd offer more
> suggestions but I think we've already discussed this elsewhere. :)

I think random I/O is very important for a lot of people at least for 
people using KVM on top of ceph. Sadly i can't implement it.

Stefan
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html