From: Aleksei Gutikov <aleksey.gutikov@synesis.ru>
To: Ceph Development <ceph-devel@vger.kernel.org>
Subject: Tuning radosgw for constant uniform high load.
Date: Fri, 26 May 2017 14:18:45 +0300 [thread overview]
Message-ID: <59054d80-97de-328d-8e94-5048b7cf9776@synesis.ru> (raw)
Hi,
We are designing ceph+rgw setup for constant uniform high load.
We prefer higher throughput than lower latency so seems that we do not
need asynchronous features, especially garbage collection.
Currently we observing issue that after amount of time rgw's gc becoming
very very slow (removing 1 rados object per second for example)
on one rgw, than on another and so forth,
but while s3 delete operations proceeding completes fast clients
continue creating and removing objects,
then pool quotas starting overflowing.
Partially this issue can be eliminated by running 10-th of additional
'radosgw-admin gc process',
but we assume that is workaround but not common approach.
Profiling on our load model that RGWGC::process() take about 30%-40% of CPU,
and while seems that CPU consuming level is ok, in real time it took
much more.
For example deleting buckets with radosgw-admin with --bypass-gc option
and without --inconsistent-index option
takes 3 times less real time than deleting all objects via s3 and wait
while gc finish his work
(with rgw_gc_processor_period value much less than total time of deletion).
I've used code from rgw_remove_bucket_bypass_gc() in RGWDeleteObj::execute()
instead of calling RGWRados::Object::Delete::delete_obj()
and measured time consumed by RGWDeleteObj::execute() + RGWGC::process()
on the same tasks and without gc consumed time is 3 times lower.
At first look seems that bucket index stay consistent without gc.
So, while I'm not yet completely understand operations required to
perform to delete object my questions:
1) rgw_remove_bucket_bypass_gc() is called from radosgw-admin.
Is it safe to call it from rgw itself?
2) rgw_remove_bucket_bypass_gc() uses librados::IoCtx::aio_operate(),
while RGWGC::process() uses librados::IoCtx::operate().
So maybe making gc multithread can speed-up it, is there any
constraints that prevents make gc multithreaded?
Thanks in advance,
Aleksei Gutikov
reply other threads:[~2017-05-26 11:18 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=59054d80-97de-328d-8e94-5048b7cf9776@synesis.ru \
--to=aleksey.gutikov@synesis.ru \
--cc=ceph-devel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.