From mboxrd@z Thu Jan 1 00:00:00 1970 From: "J. Eric Ivancich" Subject: Re: How best to integrate dmClock QoS library into ceph codebase Date: Wed, 5 Jul 2017 18:06:01 -0400 Message-ID: <942EFDE5-CB43-4074-B79A-52E036DDF19E@redhat.com> References: <6D8EA95A-572E-4C71-A6AF-6BB8A6E8B26C@redhat.com> <9c1d41e7-a218-d52b-f17f-0490c3a94d50@redhat.com> Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8BIT Return-path: Received: from mail-qt0-f178.google.com ([209.85.216.178]:35528 "EHLO mail-qt0-f178.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751784AbdGEWGF (ORCPT ); Wed, 5 Jul 2017 18:06:05 -0400 Received: by mail-qt0-f178.google.com with SMTP id b40so1945561qtb.2 for ; Wed, 05 Jul 2017 15:06:04 -0700 (PDT) In-Reply-To: Sender: ceph-devel-owner@vger.kernel.org List-ID: To: Jin Cai Cc: sheng qiu , Ceph Development Hello, Thank you for reporting that. I need to verify whether this error was generated by the code on the master branch, or whether the code has been modified in any way? The dequeue function with the failed assert (in mClockPriorityQueue.h, line 315) only has two asserts. The first assert checks that the queue is not empty, and it should not be since OSD::ShardedOpWQ::_process verifies that it’s not empty before calling the dequeue function. The second assert verifies that when an op was requested from the dmclock queue that it actually got an op, and not an empty code, or a future code (indicating that limits were strictly enforced, which they should not be since allow_limit_break should be true). So please let me know if this is modified code. If need be I’ll try to duplicate your scenario, so I can debug it. Please provide all the necessary details, so I can reproduce it. Thanks, Eric > On Jul 4, 2017, at 8:35 AM, Jin Cai wrote: > > Hi, Eric > We are now testing the mclock priority queue that you contributed > several days ago. > Our test environment consists of four machines: one machine for > the monitor and mgr daemon, two OSDs in each of the left three ones. > mclock-related configurations are as the following: > > osd_op_queue = mclock_opclass > osd_op_queue_mclock_client_op_res = 20000.0 > osd_op_queue_mclock_client_op_wgt = 0.0 > osd_op_queue_mclock_client_op_lim = 30000.0 > osd_op_queue_mclock_recov_res = 0.0 > osd_op_queue_mclock_recov_wgt = 0.0 > osd_op_queue_mclock_recov_lim = 2000.0 > > > When we kiledl one OSD daemon to test the effects of recovery on > the client op, other OSDs crashed as well because of assert semantics: > > ceph version 12.0.3-2318-g32ab095 > (32ab09536207b4b261874c0063b3275b97537045) luminous (dev) > 1: (()+0x9e86b1) [0x7f4287e846b1] > 2: (()+0xf100) [0x7f4284d18100] > 3: (gsignal()+0x37) [0x7f4283d415f7] > 4: (abort()+0x148) [0x7f4283d42ce8] > 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x284) [0x7f4287ec2364] > 6: (ceph::mClockQueue, > ceph::mClockOpClassQueue::osd_op_type_t>::dequeue()+0x45f) > [0x7f4287baac3f] > 7: (ceph::mClockOpClassQueue::dequeue()+0xd) [0x7f4287baacfd] > 8: (OSD::ShardedOpWQ::_process(unsigned int, > ceph::heartbeat_handle_d*)+0x314) [0x7f428798a174] > 9: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x8e9) > [0x7f4287ec7d39] > 10: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x7f4287ec9ec0] > 11: (()+0x7dc5) [0x7f4284d10dc5] > 12: (clone()+0x6d) [0x7f4283e02ced] > NOTE: a copy of the executable, or `objdump -rdS ` is > needed to interpret this. > > > However, if the value of osd_op_queue was set to wpq or prio, it worked well. > So do you know why the assert semantics was triggered. > > Thanks very much. > > > > > 2017-06-30 2:03 GMT+08:00 J. Eric Ivancich : >> On 06/28/2017 02:55 PM, sheng qiu wrote: >>> May I ask do you have any plan to implement client side logic for a >>> true "D"mclock, seems to me know it's mclock on each individual OSD. >>> And each client also has a common iops config. We are planing to >>> working on that part and integrate with your current work. >> >> That is not a high priority in the short-term. Our main goal with >> integrating dmclock/mclock was to better manage priorities among >> operation classes. >> >> Developers at SK Telecom have done some work towards this, though. For >> example, see here: >> >> https://github.com/ivancich/ceph/pull/1 >> >> Eric >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html