* luminous OSD memory usage @ 2017-08-30 10:00 Aleksei Gutikov 2017-08-30 15:17 ` Sage Weil 0 siblings, 1 reply; 7+ messages in thread From: Aleksei Gutikov @ 2017-08-30 10:00 UTC (permalink / raw) To: Ceph Development Hi. I'm trying to synchronize osd daemons memory limits and bluestore cache settings. For 12.1.4 we have hdd osds usage about 4G with default settings. For ssds we have limit 6G and they are been oom killed periodically. While osd_op_num_threads_per_shard_hdd=1 osd_op_num_threads_per_shard_ssd=2 and bluestore_cache_size_hdd=1G bluestore_cache_size_ssd=3G and osd_op_num_shards_hdd=5 osd_op_num_shards_ssd=8 Does it mean that ssd osds will use 4G*2*3*8/5, or 3G*2*8/5, or other? Does anybody have an idea about the equation for upper bound of memory consumption? Can bluestore_cache_size be decreased safely for example to 2G, or to 1G? I want to calculate the maximum expected size of bluestore metadata (that must be definitely fit into cache) using size of raw space, average size of objects, rocksdb space amplification. I thought it should be something simple like raw_space/avg_obj_size*obj_overhead*rocksdb_space_amp. For example if obj_overhead=1k, hdd size=1T, rocksdb space amplification is 2 and avg obj size=4M then 1T/4M*1k*2=500M so I need at least 512M for cache. But wise guys said that I have to take into account number of extents also. But bluestore_extent_map_shard_max_size=1200, I hope this number is not a multiplicator... What would be correct approach for calculation of this minimum cache size? What can be expected size of key-values stored in rocksdb per rados object? Default bluestore_cache_kv_ratio*bluestore_cache_size_ssd=0.99*3G while default bluestore_cache_kv_max=512M Looks like BlueStore::_set_cache_sizes() will set cache_kv_ratio to 1/6 in default case. Is 512M enough for bluestore metadata? Thanks! Aleksei ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: luminous OSD memory usage 2017-08-30 10:00 luminous OSD memory usage Aleksei Gutikov @ 2017-08-30 15:17 ` Sage Weil 2017-09-01 2:00 ` xiaoyan li 2017-10-20 8:52 ` Aleksei Gutikov 0 siblings, 2 replies; 7+ messages in thread From: Sage Weil @ 2017-08-30 15:17 UTC (permalink / raw) To: Aleksei Gutikov; +Cc: Ceph Development Hi Aleksei, On Wed, 30 Aug 2017, Aleksei Gutikov wrote: > Hi. > > I'm trying to synchronize osd daemons memory limits and bluestore cache > settings. > For 12.1.4 we have hdd osds usage about 4G with default settings. > For ssds we have limit 6G and they are been oom killed periodically. So, > While > osd_op_num_threads_per_shard_hdd=1 > osd_op_num_threads_per_shard_ssd=2 > and > osd_op_num_shards_hdd=5 > osd_op_num_shards_ssd=8 aren't relevant to memory usage. The _per_shard is about how many bytes are stored in each rocksdb key, and the num_shards is about how many threads we use. This is the one that matters: > bluestore_cache_size_hdd=1G > bluestore_cache_size_ssd=3G It governs how much memory bluestore limits itself to. He bad news is that bluestore counts what it allocates, not how much memory the allocator uses, so there is some overhead. From what I've anecdotally seen it's something like 1.5x, which kind of sucks; there is more to be done here. On top of that is usage by the OSD outside of bluestore, which is somewhere in the 500m to 1.5g range. We're very interested in hearing what observed RSS users see relative to the configured bluestore size and pg count, along with a dump of the mempool metadata (ceph daemon osd.NNN dump_mempools). > Does anybody have an idea about the equation for upper bound of memory > consumption? Very roughly, something like: osd_overhead + bluestore_cache_size * 1.5 ? > Can bluestore_cache_size be decreased safely for example to 2G, or to 1G? Yes, you can/should change this to whatever you like (big or small). > I want to calculate the maximum expected size of bluestore metadata (that must > be definitely fit into cache) using size of raw space, average size of > objects, rocksdb space amplification. > I thought it should be something simple like > raw_space/avg_obj_size*obj_overhead*rocksdb_space_amp. > For example if obj_overhead=1k, hdd size=1T, rocksdb space amplification is 2 > and avg obj size=4M then 1T/4M*1k*2=500M so I need at least 512M for cache. > But wise guys said that I have to take into account number of extents also. > But bluestore_extent_map_shard_max_size=1200, I hope this number is not a > multiplicator... Nope, just a shard size... > What would be correct approach for calculation of this minimum cache size? > What can be expected size of key-values stored in rocksdb per rados object? This depends, unfortunately, on what the write pattern is for the objects. If they're written by RGW in big chunks, the overhead will be smaller. If it comes out of a 4k random write pattern it will be bigger. Again, very interestd in hearing user reports of what is observed in real world situations. I would trust that more than a calculation from first principles like the above. > Default bluestore_cache_kv_ratio*bluestore_cache_size_ssd=0.99*3G > while default bluestore_cache_kv_max=512M > Looks like BlueStore::_set_cache_sizes() will set cache_kv_ratio to 1/6 in > default case. Is 512M enough for bluestore metadata? In Mark's testing he found that we got more performance benefit when small caches were devoted to rocksdb and large caches were devoted mostly to the bluestore metadata cache (parsed onodes vs caching the encoded on-disk content). You can always adjust the 512m value upwards (and that may make sense for large cache sizes). Again, very interested in hearing whther that works better or worse for your workload! Thanks- sage ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: luminous OSD memory usage 2017-08-30 15:17 ` Sage Weil @ 2017-09-01 2:00 ` xiaoyan li 2017-09-01 12:51 ` Mark Nelson 2017-09-01 13:10 ` Sage Weil 2017-10-20 8:52 ` Aleksei Gutikov 1 sibling, 2 replies; 7+ messages in thread From: xiaoyan li @ 2017-09-01 2:00 UTC (permalink / raw) To: Sage Weil; +Cc: Aleksei Gutikov, Ceph Development On Wed, Aug 30, 2017 at 11:17 PM, Sage Weil <sage@newdream.net> wrote: > Hi Aleksei, > > On Wed, 30 Aug 2017, Aleksei Gutikov wrote: >> Hi. >> >> I'm trying to synchronize osd daemons memory limits and bluestore cache >> settings. >> For 12.1.4 we have hdd osds usage about 4G with default settings. >> For ssds we have limit 6G and they are been oom killed periodically. > > So, > >> While >> osd_op_num_threads_per_shard_hdd=1 >> osd_op_num_threads_per_shard_ssd=2 >> and >> osd_op_num_shards_hdd=5 >> osd_op_num_shards_ssd=8 > > aren't relevant to memory usage. The _per_shard is about how many bytes > are stored in each rocksdb key, and the num_shards is about how many > threads we use. I can't understand the point about _per_shard. I notice that osd_op_num_threads_per_shard is used to set cache shards in BlueStore. > > This is the one that matters: > >> bluestore_cache_size_hdd=1G >> bluestore_cache_size_ssd=3G > > It governs how much memory bluestore limits itself to. He bad news is > that bluestore counts what it allocates, not how much memory the allocator > uses, so there is some overhead. From what I've anecdotally seen it's > something like 1.5x, which kind of sucks; there is more to be done here. > > On top of that is usage by the OSD outside of bluestore, > which is somewhere in the 500m to 1.5g range. > > We're very interested in hearing what observed RSS users see relative to > the configured bluestore size and pg count, along with a dump of the > mempool metadata (ceph daemon osd.NNN dump_mempools). > >> Does anybody have an idea about the equation for upper bound of memory >> consumption? > > Very roughly, something like: osd_overhead + bluestore_cache_size * 1.5 ? > >> Can bluestore_cache_size be decreased safely for example to 2G, or to 1G? > > Yes, you can/should change this to whatever you like (big or small). > >> I want to calculate the maximum expected size of bluestore metadata (that must >> be definitely fit into cache) using size of raw space, average size of >> objects, rocksdb space amplification. >> I thought it should be something simple like >> raw_space/avg_obj_size*obj_overhead*rocksdb_space_amp. >> For example if obj_overhead=1k, hdd size=1T, rocksdb space amplification is 2 >> and avg obj size=4M then 1T/4M*1k*2=500M so I need at least 512M for cache. >> But wise guys said that I have to take into account number of extents also. >> But bluestore_extent_map_shard_max_size=1200, I hope this number is not a >> multiplicator... > > Nope, just a shard size... > >> What would be correct approach for calculation of this minimum cache size? >> What can be expected size of key-values stored in rocksdb per rados object? > > This depends, unfortunately, on what the write pattern is for the objects. > If they're written by RGW in big chunks, the overhead will be smaller. > If it comes out of a 4k random write pattern it will be bigger. Again, > very interestd in hearing user reports of what is observed in real world > situations. I would trust that more than a calculation from first > principles like the above. > >> Default bluestore_cache_kv_ratio*bluestore_cache_size_ssd=0.99*3G >> while default bluestore_cache_kv_max=512M >> Looks like BlueStore::_set_cache_sizes() will set cache_kv_ratio to 1/6 in >> default case. Is 512M enough for bluestore metadata? > > In Mark's testing he found that we got more performance benefit when > small caches were devoted to rocksdb and large caches were devoted mostly > to the bluestore metadata cache (parsed onodes vs caching the encoded > on-disk content). You can always adjust the 512m value upwards (and that > may make sense for large cache sizes). Again, very interested in hearing > whther that works better or worse for your workload! > I am wondering whether Bluestore should have Rocksdb block cache. Rocksdb uses block cache to cache data blocks in memory for reads. But the data is already cached by bluestore metadata cache. And bluestore metadata cache has two advantages: 1. no cache for pg_log and deferred io logs. 2. Rocksdb cache data blocks. it doesn't have fine-granularity. Thanks- > sage > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Best wishes Lisa ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: luminous OSD memory usage 2017-09-01 2:00 ` xiaoyan li @ 2017-09-01 12:51 ` Mark Nelson 2017-09-01 13:10 ` Sage Weil 1 sibling, 0 replies; 7+ messages in thread From: Mark Nelson @ 2017-09-01 12:51 UTC (permalink / raw) To: xiaoyan li, Sage Weil; +Cc: Aleksei Gutikov, Ceph Development On 08/31/2017 09:00 PM, xiaoyan li wrote: > On Wed, Aug 30, 2017 at 11:17 PM, Sage Weil <sage@newdream.net> wrote: >> Hi Aleksei, >> >> On Wed, 30 Aug 2017, Aleksei Gutikov wrote: >>> Hi. >>> >>> I'm trying to synchronize osd daemons memory limits and bluestore cache >>> settings. >>> For 12.1.4 we have hdd osds usage about 4G with default settings. >>> For ssds we have limit 6G and they are been oom killed periodically. >> >> So, >> >>> While >>> osd_op_num_threads_per_shard_hdd=1 >>> osd_op_num_threads_per_shard_ssd=2 >>> and >>> osd_op_num_shards_hdd=5 >>> osd_op_num_shards_ssd=8 >> >> aren't relevant to memory usage. The _per_shard is about how many bytes >> are stored in each rocksdb key, and the num_shards is about how many >> threads we use. > > I can't understand the point about _per_shard. I notice that > osd_op_num_threads_per_shard is used to set cache shards in BlueStore. >> >> This is the one that matters: >> >>> bluestore_cache_size_hdd=1G >>> bluestore_cache_size_ssd=3G >> >> It governs how much memory bluestore limits itself to. He bad news is >> that bluestore counts what it allocates, not how much memory the allocator >> uses, so there is some overhead. From what I've anecdotally seen it's >> something like 1.5x, which kind of sucks; there is more to be done here. >> >> On top of that is usage by the OSD outside of bluestore, >> which is somewhere in the 500m to 1.5g range. >> >> We're very interested in hearing what observed RSS users see relative to >> the configured bluestore size and pg count, along with a dump of the >> mempool metadata (ceph daemon osd.NNN dump_mempools). >> >>> Does anybody have an idea about the equation for upper bound of memory >>> consumption? >> >> Very roughly, something like: osd_overhead + bluestore_cache_size * 1.5 ? >> >>> Can bluestore_cache_size be decreased safely for example to 2G, or to 1G? >> >> Yes, you can/should change this to whatever you like (big or small). >> >>> I want to calculate the maximum expected size of bluestore metadata (that must >>> be definitely fit into cache) using size of raw space, average size of >>> objects, rocksdb space amplification. >>> I thought it should be something simple like >>> raw_space/avg_obj_size*obj_overhead*rocksdb_space_amp. >>> For example if obj_overhead=1k, hdd size=1T, rocksdb space amplification is 2 >>> and avg obj size=4M then 1T/4M*1k*2=500M so I need at least 512M for cache. >>> But wise guys said that I have to take into account number of extents also. >>> But bluestore_extent_map_shard_max_size=1200, I hope this number is not a >>> multiplicator... >> >> Nope, just a shard size... >> >>> What would be correct approach for calculation of this minimum cache size? >>> What can be expected size of key-values stored in rocksdb per rados object? >> >> This depends, unfortunately, on what the write pattern is for the objects. >> If they're written by RGW in big chunks, the overhead will be smaller. >> If it comes out of a 4k random write pattern it will be bigger. Again, >> very interestd in hearing user reports of what is observed in real world >> situations. I would trust that more than a calculation from first >> principles like the above. >> >>> Default bluestore_cache_kv_ratio*bluestore_cache_size_ssd=0.99*3G >>> while default bluestore_cache_kv_max=512M >>> Looks like BlueStore::_set_cache_sizes() will set cache_kv_ratio to 1/6 in >>> default case. Is 512M enough for bluestore metadata? >> >> In Mark's testing he found that we got more performance benefit when >> small caches were devoted to rocksdb and large caches were devoted mostly >> to the bluestore metadata cache (parsed onodes vs caching the encoded >> on-disk content). You can always adjust the 512m value upwards (and that >> may make sense for large cache sizes). Again, very interested in hearing >> whther that works better or worse for your workload! >> > I am wondering whether Bluestore should have Rocksdb block cache. > Rocksdb uses block cache to cache data blocks in memory for reads. But > the data is already cached by bluestore metadata cache. And bluestore > metadata cache has two advantages: 1. no cache for pg_log and deferred > io logs. 2. Rocksdb cache data blocks. it doesn't have > fine-granularity. We use the block cache in rocksdb to store the bloom filters as well so we'll want to at least give it enough to cover those. We also found that performance tanked at small cache values when favoring onode cache vs rocksdb block cache. There could be a couple of reasons for that. rocksdb is storing the data already encoded, and since we use varint, I suspect that we can store more in the rocksdb cache. It might be that at low cache sizes it's worth taking the CPU hit to fit more data in cache. We can do that in the bluestore cache as well, but it might be that the two-level apporach is better, ie we want the cache closest to bluestore to not be encoded while cache at the rocksdb level is encoded (and perhaps we even want to use compression at some point). Mark > > Thanks- >> sage >> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: luminous OSD memory usage 2017-09-01 2:00 ` xiaoyan li 2017-09-01 12:51 ` Mark Nelson @ 2017-09-01 13:10 ` Sage Weil 1 sibling, 0 replies; 7+ messages in thread From: Sage Weil @ 2017-09-01 13:10 UTC (permalink / raw) To: xiaoyan li; +Cc: Aleksei Gutikov, Ceph Development On Fri, 1 Sep 2017, xiaoyan li wrote: > On Wed, Aug 30, 2017 at 11:17 PM, Sage Weil <sage@newdream.net> wrote: > > Hi Aleksei, > > > > On Wed, 30 Aug 2017, Aleksei Gutikov wrote: > >> Hi. > >> > >> I'm trying to synchronize osd daemons memory limits and bluestore cache > >> settings. > >> For 12.1.4 we have hdd osds usage about 4G with default settings. > >> For ssds we have limit 6G and they are been oom killed periodically. > > > > So, > > > >> While > >> osd_op_num_threads_per_shard_hdd=1 > >> osd_op_num_threads_per_shard_ssd=2 > >> and > >> osd_op_num_shards_hdd=5 > >> osd_op_num_shards_ssd=8 > > > > aren't relevant to memory usage. The _per_shard is about how many bytes > > are stored in each rocksdb key, and the num_shards is about how many > > threads we use. > > I can't understand the point about _per_shard. I notice that > osd_op_num_threads_per_shard is used to set cache shards in BlueStore. store->set_cache_shards(get_num_op_shards()); and the osd op queue thread count is then shards * threads_per_shard. sage ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: luminous OSD memory usage 2017-08-30 15:17 ` Sage Weil 2017-09-01 2:00 ` xiaoyan li @ 2017-10-20 8:52 ` Aleksei Gutikov 2017-10-20 12:12 ` Sage Weil 1 sibling, 1 reply; 7+ messages in thread From: Aleksei Gutikov @ 2017-10-20 8:52 UTC (permalink / raw) To: Sage Weil; +Cc: Ceph Development [-- Attachment #1: Type: text/plain, Size: 4411 bytes --] Hi, Here is some stats about OSD memory usage on our lab cluster (110hdd*1T + 36ssd*200G) https://drive.google.com/drive/folders/0B1s9jTJ0z59JcmtEX283WUd5bzg?usp=sharing osd_mem_stats.txt contains stats for hdd and ssd OSDs: - stats for process memory usage from /proc/pid/status, - mempools stats - and heap stats We use luminous 12.2.1. We set 9G memory limit in /lib/systemd/system/ceph-osd@.service.d/override.conf But that is not enough - OSDs are still been killed because they use more (with bluestore_cache_size_ssd=3G) OSDs with hdd (with bluestore_cache_size_hdd=1G) use up to 4.6G. And, btw, seems that systemd's oom killer looks at VmData, not on VmRss. Thanks. On 08/30/2017 06:17 PM, Sage Weil wrote: > Hi Aleksei, > > On Wed, 30 Aug 2017, Aleksei Gutikov wrote: >> Hi. >> >> I'm trying to synchronize osd daemons memory limits and bluestore cache >> settings. >> For 12.1.4 we have hdd osds usage about 4G with default settings. >> For ssds we have limit 6G and they are been oom killed periodically. > > So, > >> While >> osd_op_num_threads_per_shard_hdd=1 >> osd_op_num_threads_per_shard_ssd=2 >> and >> osd_op_num_shards_hdd=5 >> osd_op_num_shards_ssd=8 > > aren't relevant to memory usage. The _per_shard is about how many bytes > are stored in each rocksdb key, and the num_shards is about how many > threads we use. > > This is the one that matters: > >> bluestore_cache_size_hdd=1G >> bluestore_cache_size_ssd=3G > > It governs how much memory bluestore limits itself to. He bad news is > that bluestore counts what it allocates, not how much memory the allocator > uses, so there is some overhead. From what I've anecdotally seen it's > something like 1.5x, which kind of sucks; there is more to be done here. > > On top of that is usage by the OSD outside of bluestore, > which is somewhere in the 500m to 1.5g range. > > We're very interested in hearing what observed RSS users see relative to > the configured bluestore size and pg count, along with a dump of the > mempool metadata (ceph daemon osd.NNN dump_mempools). > >> Does anybody have an idea about the equation for upper bound of memory >> consumption? > > Very roughly, something like: osd_overhead + bluestore_cache_size * 1.5 ? > >> Can bluestore_cache_size be decreased safely for example to 2G, or to 1G? > > Yes, you can/should change this to whatever you like (big or small). > >> I want to calculate the maximum expected size of bluestore metadata (that must >> be definitely fit into cache) using size of raw space, average size of >> objects, rocksdb space amplification. >> I thought it should be something simple like >> raw_space/avg_obj_size*obj_overhead*rocksdb_space_amp. >> For example if obj_overhead=1k, hdd size=1T, rocksdb space amplification is 2 >> and avg obj size=4M then 1T/4M*1k*2=500M so I need at least 512M for cache. >> But wise guys said that I have to take into account number of extents also. >> But bluestore_extent_map_shard_max_size=1200, I hope this number is not a >> multiplicator... > > Nope, just a shard size... > >> What would be correct approach for calculation of this minimum cache size? >> What can be expected size of key-values stored in rocksdb per rados object? > > This depends, unfortunately, on what the write pattern is for the objects. > If they're written by RGW in big chunks, the overhead will be smaller. > If it comes out of a 4k random write pattern it will be bigger. Again, > very interestd in hearing user reports of what is observed in real world > situations. I would trust that more than a calculation from first > principles like the above. > >> Default bluestore_cache_kv_ratio*bluestore_cache_size_ssd=0.99*3G >> while default bluestore_cache_kv_max=512M >> Looks like BlueStore::_set_cache_sizes() will set cache_kv_ratio to 1/6 in >> default case. Is 512M enough for bluestore metadata? > > In Mark's testing he found that we got more performance benefit when > small caches were devoted to rocksdb and large caches were devoted mostly > to the bluestore metadata cache (parsed onodes vs caching the encoded > on-disk content). You can always adjust the 512m value upwards (and that > may make sense for large cache sizes). Again, very interested in hearing > whther that works better or worse for your workload! > > Thanks- > sage > -- Best regards, Aleksei Gutikov Software Engineer | synesis.ru | Minsk. BY [-- Attachment #2: osd_mem_stats.txt --] [-- Type: text/plain, Size: 17077 bytes --] OSD device class: ssd ===================== heap_dump --------- +-----------------------------------+----------+-----------+-----------+ | metric | min | mean | max | +-----------------------------------+----------+-----------+-----------+ | Actual_memory_used_physical_swap | 1.67 Gi | 6.58 Gi | 8.70 Gi | | Bytes_in_central_cache_freelist | 4.95 Mi | 118.15 Mi | 294.81 Mi | | Bytes_in_malloc_metadata | 5.01 Mi | 26.56 Mi | 39.82 Mi | | Bytes_in_page_heap_freelist | 0.0 | 0.0 | 0.0 | | Bytes_in_thread_cache_freelists | 24.26 Mi | 94.57 Mi | 109.40 Mi | | Bytes_in_transfer_cache_freelist | 7.04 Mi | 10.93 Mi | 18.05 Mi | | Bytes_in_use_by_application | 1.63 Gi | 6.34 Gi | 8.40 Gi | | Bytes_released_to_OS_aka_unmapped | 4.42 Mi | 417.02 Mi | 933.85 Mi | | Spans_in_use | 44.98 Ki | 383.28 Ki | 611.09 Ki | | Tcmalloc_page_size | 8.00 Ki | 8.00 Ki | 8.00 Ki | | Thread_heaps_in_use | 41.0 | 45.8 | 47.0 | | Virtual_address_space_used | 1.67 Gi | 6.99 Gi | 9.61 Gi | +-----------------------------------+----------+-----------+-----------+ mempool_bytes ------------- +----------------------------+----------+-----------+-----------+ | metric | min | mean | max | +----------------------------+----------+-----------+-----------+ | bloom_filter | 0.0 | 0.0 | 0.0 | | bluefs | 21.97 Ki | 26.27 Ki | 33.38 Ki | | bluestore_alloc | 1.07 Mi | 1.38 Mi | 1.72 Mi | | bluestore_cache_data | 53.20 Mi | 965.17 Mi | 1.73 Gi | | bluestore_cache_onode | 1.44 Mi | 682.79 Mi | 1.25 Gi | | bluestore_cache_other | 11.58 Mi | 718.71 Mi | 1.16 Gi | | bluestore_fsck | 0.0 | 0.0 | 0.0 | | bluestore_txc | 1.41 Ki | 6.84 Ki | 12.66 Ki | | bluestore_writing | 87.17 Ki | 693.05 Ki | 905.40 Ki | | bluestore_writing_deferred | 44.00 Ki | 457.67 Ki | 4.09 Mi | | buffer_anon | 10.09 Mi | 995.24 Mi | 1.55 Gi | | buffer_meta | 98.14 Ki | 1.51 Mi | 3.07 Mi | | mds_co | 0.0 | 0.0 | 0.0 | | osd | 1.27 Mi | 1.49 Mi | 1.77 Mi | | osd_mapbl | 0.0 | 338.79 Ki | 1.80 Mi | | osd_pglog | 68.36 Mi | 89.42 Mi | 117.34 Mi | | osdmap | 1.11 Mi | 1.26 Mi | 1.38 Mi | | osdmap_mapping | 0.0 | 0.0 | 0.0 | | pgmap | 0.0 | 0.0 | 0.0 | | total | 1.31 Gi | 3.38 Gi | 4.09 Gi | | unittest_1 | 0.0 | 0.0 | 0.0 | | unittest_2 | 0.0 | 0.0 | 0.0 | +----------------------------+----------+-----------+-----------+ mempool_item_size ----------------- +----------------------------+----------+----------+-----------+ | metric | min | mean | max | +----------------------------+----------+----------+-----------+ | bloom_filter | 0.0 | 0.0 | 0.0 | | bluefs | 34.4 | 45.1 | 63.3 | | bluestore_alloc | 1.0 | 1.0 | 1.0 | | bluestore_cache_data | 34.73 Ki | 55.22 Ki | 58.67 Ki | | bluestore_cache_onode | 680.0 | 680.0 | 680.0 | | bluestore_cache_other | 2.9 | 4.0 | 15.5 | | bluestore_fsck | 0.0 | 0.0 | 0.0 | | bluestore_txc | 720.0 | 720.0 | 720.0 | | bluestore_writing | 43.59 Ki | 65.47 Ki | 69.65 Ki | | bluestore_writing_deferred | 4.40 Ki | 25.40 Ki | 209.60 Ki | | buffer_anon | 92.4 | 112.3 | 276.7 | | buffer_meta | 88.0 | 88.0 | 88.0 | | mds_co | 0.0 | 0.0 | 0.0 | | osd | 11.78 Ki | 11.78 Ki | 11.78 Ki | | osd_mapbl | 0.0 | 14.44 Ki | 47.27 Ki | | osd_pglog | 217.2 | 261.3 | 279.0 | | osdmap | 29.2 | 32.4 | 33.1 | | osdmap_mapping | 0.0 | 0.0 | 0.0 | | pgmap | 0.0 | 0.0 | 0.0 | | total | 10.1 | 60.4 | 599.6 | | unittest_1 | 0.0 | 0.0 | 0.0 | | unittest_2 | 0.0 | 0.0 | 0.0 | +----------------------------+----------+----------+-----------+ mempool_items ------------- +----------------------------+-----------+-----------+-----------+ | metric | min | mean | max | +----------------------------+-----------+-----------+-----------+ | bloom_filter | 0.0 | 0.0 | 0.0 | | bluefs | 387.0 | 612.1 | 917.0 | | bluestore_alloc | 1.07 Mi | 1.38 Mi | 1.72 Mi | | bluestore_cache_data | 987.0 | 17.46 Ki | 35.55 Ki | | bluestore_cache_onode | 2.17 Ki | 1.00 Mi | 1.88 Mi | | bluestore_cache_other | 769.63 Ki | 240.06 Mi | 392.61 Mi | | bluestore_fsck | 0.0 | 0.0 | 0.0 | | bluestore_txc | 2.0 | 9.7 | 18.0 | | bluestore_writing | 2.0 | 10.4 | 13.0 | | bluestore_writing_deferred | 10.0 | 17.6 | 27.0 | | buffer_anon | 48.34 Ki | 9.69 Mi | 15.35 Mi | | buffer_meta | 1.12 Ki | 17.59 Ki | 35.68 Ki | | mds_co | 0.0 | 0.0 | 0.0 | | osd | 110.0 | 129.8 | 154.0 | | osd_mapbl | 0.0 | 7.2 | 39.0 | | osd_pglog | 267.34 Ki | 353.21 Ki | 501.38 Ki | | osdmap | 38.36 Ki | 39.74 Ki | 47.61 Ki | | osdmap_mapping | 0.0 | 0.0 | 0.0 | | pgmap | 0.0 | 0.0 | 0.0 | | total | 2.50 Mi | 252.55 Mi | 411.59 Mi | | unittest_1 | 0.0 | 0.0 | 0.0 | | unittest_2 | 0.0 | 0.0 | 0.0 | +----------------------------+-----------+-----------+-----------+ proc_status ----------- +----------------------------+-----------+-----------+-----------+ | metric | min | mean | max | +----------------------------+-----------+-----------+-----------+ | RssAnon | 1.65 Gi | 6.56 Gi | 8.66 Gi | | RssFile | 27.91 Mi | 28.70 Mi | 29.05 Mi | | RssShmem | 0.0 | 0.0 | 0.0 | | Threads | 65.0 | 66.9 | 67.0 | | VmData | 2.30 Gi | 7.63 Gi | 10.26 Gi | | VmExe | 18.79 Mi | 18.79 Mi | 18.79 Mi | | VmHWM | 1.68 Gi | 6.77 Gi | 8.92 Gi | | VmLck | 0.0 | 0.0 | 0.0 | | VmLib | 15.21 Mi | 15.21 Mi | 15.21 Mi | | VmPMD | 20.00 Ki | 43.44 Ki | 56.00 Ki | | VmPTE | 3.87 Mi | 14.52 Mi | 19.77 Mi | | VmPeak | 2.43 Gi | 8.08 Gi | 10.77 Gi | | VmPin | 0.0 | 0.0 | 0.0 | | VmRSS | 1.67 Gi | 6.58 Gi | 8.69 Gi | | VmSize | 2.43 Gi | 7.77 Gi | 10.40 Gi | | VmStk | 804.00 Ki | 808.11 Ki | 812.00 Ki | | VmSwap | 0.0 | 0.0 | 0.0 | | nonvoluntary_ctxt_switches | 53.0 | 299.6 | 978.0 | | voluntary_ctxt_switches | 19.93 Ki | 26.59 Ki | 39.16 Ki | +----------------------------+-----------+-----------+-----------+ OSD device class: hdd ===================== heap_dump --------- +-----------------------------------+-----------+-----------+-----------+ | metric | min | mean | max | +-----------------------------------+-----------+-----------+-----------+ | Actual_memory_used_physical_swap | 2.93 Gi | 3.19 Gi | 3.42 Gi | | Bytes_in_central_cache_freelist | 173.28 Mi | 231.17 Mi | 285.54 Mi | | Bytes_in_malloc_metadata | 12.61 Mi | 13.46 Mi | 14.00 Mi | | Bytes_in_page_heap_freelist | 0.0 | 0.0 | 0.0 | | Bytes_in_thread_cache_freelists | 74.56 Mi | 93.96 Mi | 105.09 Mi | | Bytes_in_transfer_cache_freelist | 10.89 Mi | 15.77 Mi | 19.35 Mi | | Bytes_in_use_by_application | 2.62 Gi | 2.85 Gi | 3.05 Gi | | Bytes_released_to_OS_aka_unmapped | 95.78 Mi | 285.71 Mi | 480.20 Mi | | Spans_in_use | 171.07 Ki | 185.28 Ki | 195.04 Ki | | Tcmalloc_page_size | 8.00 Ki | 8.00 Ki | 8.00 Ki | | Thread_heaps_in_use | 33.0 | 33.2 | 36.0 | | Virtual_address_space_used | 3.14 Gi | 3.47 Gi | 3.69 Gi | +-----------------------------------+-----------+-----------+-----------+ mempool_bytes ------------- +----------------------------+-----------+-----------+-----------+ | metric | min | mean | max | +----------------------------+-----------+-----------+-----------+ | bloom_filter | 0.0 | 0.0 | 0.0 | | bluefs | 9.63 Ki | 11.94 Ki | 13.67 Ki | | bluestore_alloc | 26.64 Ki | 379.62 Ki | 1.10 Mi | | bluestore_cache_data | 0.0 | 552.91 Ki | 19.61 Mi | | bluestore_cache_onode | 187.35 Mi | 192.72 Mi | 197.83 Mi | | bluestore_cache_other | 304.98 Mi | 318.62 Mi | 322.48 Mi | | bluestore_fsck | 0.0 | 0.0 | 0.0 | | bluestore_txc | 1.41 Ki | 53.80 Ki | 113.91 Ki | | bluestore_writing | 808.52 Ki | 1.25 Mi | 7.77 Mi | | bluestore_writing_deferred | 6.92 Ki | 730.02 Ki | 4.60 Mi | | buffer_anon | 835.34 Mi | 967.42 Mi | 1.08 Gi | | buffer_meta | 7.65 Ki | 10.18 Ki | 20.54 Ki | | mds_co | 0.0 | 0.0 | 0.0 | | osd | 1.39 Mi | 1.71 Mi | 1.97 Mi | | osd_mapbl | 0.0 | 0.0 | 0.0 | | osd_pglog | 103.92 Mi | 129.63 Mi | 149.63 Mi | | osdmap | 1.24 Mi | 1.25 Mi | 1.26 Mi | | osdmap_mapping | 0.0 | 0.0 | 0.0 | | pgmap | 0.0 | 0.0 | 0.0 | | total | 1.42 Gi | 1.58 Gi | 1.73 Gi | | unittest_1 | 0.0 | 0.0 | 0.0 | | unittest_2 | 0.0 | 0.0 | 0.0 | +----------------------------+-----------+-----------+-----------+ mempool_item_size ----------------- +----------------------------+----------+----------+-----------+ | metric | min | mean | max | +----------------------------+----------+----------+-----------+ | bloom_filter | 0.0 | 0.0 | 0.0 | | bluefs | 40.0 | 43.1 | 61.8 | | bluestore_alloc | 1.0 | 1.0 | 1.0 | | bluestore_cache_data | 0.0 | 15.18 Ki | 248.00 Ki | | bluestore_cache_onode | 680.0 | 680.0 | 680.0 | | bluestore_cache_other | 7.1 | 7.3 | 7.4 | | bluestore_fsck | 0.0 | 0.0 | 0.0 | | bluestore_txc | 720.0 | 720.0 | 720.0 | | bluestore_writing | 9.11 Ki | 37.39 Ki | 357.00 Ki | | bluestore_writing_deferred | 5.16 Ki | 26.33 Ki | 503.09 Ki | | buffer_anon | 381.1 | 421.3 | 475.5 | | buffer_meta | 88.0 | 88.0 | 88.0 | | mds_co | 0.0 | 0.0 | 0.0 | | osd | 11.78 Ki | 11.78 Ki | 11.78 Ki | | osd_mapbl | 0.0 | 0.0 | 0.0 | | osd_pglog | 264.0 | 272.6 | 281.4 | | osdmap | 32.6 | 32.7 | 33.1 | | osdmap_mapping | 0.0 | 0.0 | 0.0 | | pgmap | 0.0 | 0.0 | 0.0 | | total | 30.9 | 34.3 | 37.6 | | unittest_1 | 0.0 | 0.0 | 0.0 | | unittest_2 | 0.0 | 0.0 | 0.0 | +----------------------------+----------+----------+-----------+ mempool_items ------------- +----------------------------+-----------+-----------+-----------+ | metric | min | mean | max | +----------------------------+-----------+-----------+-----------+ | bloom_filter | 0.0 | 0.0 | 0.0 | | bluefs | 172.0 | 286.0 | 315.0 | | bluestore_alloc | 26.64 Ki | 379.62 Ki | 1.10 Mi | | bluestore_cache_data | 0.0 | 3.7 | 147.0 | | bluestore_cache_onode | 282.12 Ki | 290.21 Ki | 297.91 Ki | | bluestore_cache_other | 42.30 Mi | 43.65 Mi | 44.03 Mi | | bluestore_fsck | 0.0 | 0.0 | 0.0 | | bluestore_txc | 2.0 | 76.5 | 162.0 | | bluestore_writing | 12.0 | 85.5 | 133.0 | | bluestore_writing_deferred | 1.0 | 42.3 | 80.0 | | buffer_anon | 2.18 Mi | 2.30 Mi | 2.35 Mi | | buffer_meta | 89.0 | 118.4 | 239.0 | | mds_co | 0.0 | 0.0 | 0.0 | | osd | 121.0 | 148.9 | 171.0 | | osd_mapbl | 0.0 | 0.0 | 0.0 | | osd_pglog | 378.11 Ki | 487.10 Ki | 569.86 Ki | | osdmap | 38.36 Ki | 39.18 Ki | 39.65 Ki | | osdmap_mapping | 0.0 | 0.0 | 0.0 | | pgmap | 0.0 | 0.0 | 0.0 | | total | 45.39 Mi | 47.11 Mi | 48.00 Mi | | unittest_1 | 0.0 | 0.0 | 0.0 | | unittest_2 | 0.0 | 0.0 | 0.0 | +----------------------------+-----------+-----------+-----------+ proc_status ----------- +----------------------------+-----------+-----------+-----------+ | metric | min | mean | max | +----------------------------+-----------+-----------+-----------+ | RssAnon | 2.93 Gi | 3.17 Gi | 3.41 Gi | | RssFile | 28.10 Mi | 28.65 Mi | 29.01 Mi | | RssShmem | 0.0 | 0.0 | 0.0 | | Threads | 56.0 | 56.0 | 56.0 | | VmData | 3.71 Gi | 4.03 Gi | 4.25 Gi | | VmExe | 18.79 Mi | 18.79 Mi | 18.79 Mi | | VmHWM | 3.06 Gi | 3.34 Gi | 3.54 Gi | | VmLck | 0.0 | 0.0 | 0.0 | | VmLib | 15.21 Mi | 15.21 Mi | 15.21 Mi | | VmPMD | 24.00 Ki | 28.62 Ki | 32.00 Ki | | VmPTE | 6.79 Mi | 7.45 Mi | 7.89 Mi | | VmPeak | 3.99 Gi | 4.35 Gi | 4.60 Gi | | VmPin | 0.0 | 0.0 | 0.0 | | VmRSS | 2.96 Gi | 3.20 Gi | 3.44 Gi | | VmSize | 3.84 Gi | 4.16 Gi | 4.39 Gi | | VmStk | 804.00 Ki | 808.40 Ki | 812.00 Ki | | VmSwap | 0.0 | 0.0 | 0.0 | | nonvoluntary_ctxt_switches | 462.0 | 24.04 Ki | 40.61 Ki | | voluntary_ctxt_switches | 43.45 Ki | 61.96 Ki | 89.04 Ki | +----------------------------+-----------+-----------+-----------+ ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: luminous OSD memory usage 2017-10-20 8:52 ` Aleksei Gutikov @ 2017-10-20 12:12 ` Sage Weil 0 siblings, 0 replies; 7+ messages in thread From: Sage Weil @ 2017-10-20 12:12 UTC (permalink / raw) To: Aleksei Gutikov; +Cc: Ceph Development On Fri, 20 Oct 2017, Aleksei Gutikov wrote: > Hi, > > Here is some stats about OSD memory usage on our lab cluster > (110hdd*1T + 36ssd*200G) > > https://drive.google.com/drive/folders/0B1s9jTJ0z59JcmtEX283WUd5bzg?usp=sharing > > osd_mem_stats.txt contains stats for hdd and ssd OSDs: > - stats for process memory usage from /proc/pid/status, > - mempools stats > - and heap stats > > We use luminous 12.2.1. > We set 9G memory limit in > /lib/systemd/system/ceph-osd@.service.d/override.conf > But that is not enough - OSDs are still been killed because they use more > (with bluestore_cache_size_ssd=3G) > OSDs with hdd (with bluestore_cache_size_hdd=1G) use up to 4.6G. > > And, btw, seems that systemd's oom killer looks at VmData, not on VmRss. There was a fix for the bluestore memory usage calculation that didn't make it into 12.1.1 (e.g., f60a942023088cbba53a816e6ef846994921cab3). You repeat your test with the latest luminous branch or wait a week or so for 12.2.2. Thanks! sage > > Thanks. > > On 08/30/2017 06:17 PM, Sage Weil wrote: > > Hi Aleksei, > > > > On Wed, 30 Aug 2017, Aleksei Gutikov wrote: > > > Hi. > > > > > > I'm trying to synchronize osd daemons memory limits and bluestore cache > > > settings. > > > For 12.1.4 we have hdd osds usage about 4G with default settings. > > > For ssds we have limit 6G and they are been oom killed periodically. > > > > So, > > > > > While > > > osd_op_num_threads_per_shard_hdd=1 > > > osd_op_num_threads_per_shard_ssd=2 > > > and > > > osd_op_num_shards_hdd=5 > > > osd_op_num_shards_ssd=8 > > > > aren't relevant to memory usage. The _per_shard is about how many bytes > > are stored in each rocksdb key, and the num_shards is about how many > > threads we use. > > > > This is the one that matters: > > > > > bluestore_cache_size_hdd=1G > > > bluestore_cache_size_ssd=3G > > > > It governs how much memory bluestore limits itself to. He bad news is > > that bluestore counts what it allocates, not how much memory the allocator > > uses, so there is some overhead. From what I've anecdotally seen it's > > something like 1.5x, which kind of sucks; there is more to be done here. > > > > On top of that is usage by the OSD outside of bluestore, > > which is somewhere in the 500m to 1.5g range. > > > > We're very interested in hearing what observed RSS users see relative to > > the configured bluestore size and pg count, along with a dump of the > > mempool metadata (ceph daemon osd.NNN dump_mempools). > > > > > Does anybody have an idea about the equation for upper bound of memory > > > consumption? > > > > Very roughly, something like: osd_overhead + bluestore_cache_size * 1.5 ? > > > > > Can bluestore_cache_size be decreased safely for example to 2G, or to 1G? > > > > Yes, you can/should change this to whatever you like (big or small). > > > > > I want to calculate the maximum expected size of bluestore metadata (that > > > must > > > be definitely fit into cache) using size of raw space, average size of > > > objects, rocksdb space amplification. > > > I thought it should be something simple like > > > raw_space/avg_obj_size*obj_overhead*rocksdb_space_amp. > > > For example if obj_overhead=1k, hdd size=1T, rocksdb space amplification > > > is 2 > > > and avg obj size=4M then 1T/4M*1k*2=500M so I need at least 512M for > > > cache. > > > But wise guys said that I have to take into account number of extents > > > also. > > > But bluestore_extent_map_shard_max_size=1200, I hope this number is not a > > > multiplicator... > > > > Nope, just a shard size... > > > > > What would be correct approach for calculation of this minimum cache size? > > > What can be expected size of key-values stored in rocksdb per rados > > > object? > > > > This depends, unfortunately, on what the write pattern is for the objects. > > If they're written by RGW in big chunks, the overhead will be smaller. > > If it comes out of a 4k random write pattern it will be bigger. Again, > > very interestd in hearing user reports of what is observed in real world > > situations. I would trust that more than a calculation from first > > principles like the above. > > > > > Default bluestore_cache_kv_ratio*bluestore_cache_size_ssd=0.99*3G > > > while default bluestore_cache_kv_max=512M > > > Looks like BlueStore::_set_cache_sizes() will set cache_kv_ratio to 1/6 in > > > default case. Is 512M enough for bluestore metadata? > > > > In Mark's testing he found that we got more performance benefit when > > small caches were devoted to rocksdb and large caches were devoted mostly > > to the bluestore metadata cache (parsed onodes vs caching the encoded > > on-disk content). You can always adjust the 512m value upwards (and that > > may make sense for large cache sizes). Again, very interested in hearing > > whther that works better or worse for your workload! > > > > Thanks- > > sage > > > > -- > > Best regards, > Aleksei Gutikov > Software Engineer | synesis.ru | Minsk. BY > ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2017-10-20 12:12 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-08-30 10:00 luminous OSD memory usage Aleksei Gutikov 2017-08-30 15:17 ` Sage Weil 2017-09-01 2:00 ` xiaoyan li 2017-09-01 12:51 ` Mark Nelson 2017-09-01 13:10 ` Sage Weil 2017-10-20 8:52 ` Aleksei Gutikov 2017-10-20 12:12 ` Sage Weil
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.