All of lore.kernel.org
 help / color / mirror / Atom feed
From: Ming Lin <mlin@kernel.org>
To: Brian Foster <bfoster@redhat.com>
Cc: Dave Chinner <david@fromorbit.com>,
	Christoph Hellwig <hch@lst.de>,
	linux-xfs@vger.kernel.org,
	Ceph Development <ceph-devel@vger.kernel.org>,
	"LIU, Fei" <james.liu@alibaba-inc.com>,
	xiongwei.jiang@alibaba-inc.com, boqian.zy@alibaba-inc.com,
	sheng.qiu@alibaba-inc.com, Sage Weil <sweil@redhat.com>
Subject: Re: xlog_write: reservation ran out
Date: Mon, 1 May 2017 13:18:58 -0700	[thread overview]
Message-ID: <CAF1ivSZgh-v1=LqeFwKZWuzcARwy9w37Qg_J3r_Hm-jnbmaH6g@mail.gmail.com> (raw)
In-Reply-To: <20170501184822.GB3775@bfoster.bfoster>

On Mon, May 1, 2017 at 11:48 AM, Brian Foster <bfoster@redhat.com> wrote:
>>
>> It takes about 10 hours to reproduce the problem.
>>
>
> Out of curiosity, is that 10 hours of removing files or 10 hours of
> repopulating and removing until the problem happens to occur? If the
> latter, roughly how many fill/remove cycles does that entail (tens,
> hundreds, thousands)?

10 hours of repopulating the cluster. Then remove all rbd images with
"rbd rm xxx"
Just 1 cycle: fill then remove.

> You could try to populate the fs using Ceph as with your current
> reproducer, particularly since it may use patterns or features that
> could affect this problem (xattrs?) that fio may not induce, and then
> try to directly reproduce the overrun via manual file removals. This
> would be sufficient for debugging because if you can share a metadump
> image of the original fs and appropriate steps to reproduce, we don't
> particularly need to care about how the fs was constructed in the first
> place.
>
> For example, if you have a test that currently populates and depopulates
> the fs through Ceph, something I might try is to update the test to
> generate a metadump image of the fs every time your test cycles from
> populating to depopulating. Once the problem reproduces, you now have a
> metadump image of the original fs that you can restore and use to try to
> reproduce the overrun manually (repeatedly, if nec.).

That's a nice idea to debug it.
I'll try.

Thanks.

  reply	other threads:[~2017-05-01 20:19 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-04-28 20:15 xlog_write: reservation ran out Ming Lin
2017-04-28 20:24 ` Ming Lin
2017-04-28 20:56   ` Ming Lin
2017-05-01  6:10     ` Ming Lin
2017-05-01 13:12       ` Brian Foster
2017-05-01 17:41         ` Ming Lin
2017-05-01 18:48           ` Brian Foster
2017-05-01 20:18             ` Ming Lin [this message]
2017-05-02 23:51           ` Dave Chinner
2017-05-03 14:10             ` Brian Foster
2017-05-03 15:53               ` Darrick J. Wong
2017-05-03 16:15                 ` Brian Foster
2017-05-04  0:37                   ` Dave Chinner
2017-05-04  0:34               ` Dave Chinner
2017-05-04 11:51                 ` Brian Foster
2017-05-05  3:16                   ` Dave Chinner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAF1ivSZgh-v1=LqeFwKZWuzcARwy9w37Qg_J3r_Hm-jnbmaH6g@mail.gmail.com' \
    --to=mlin@kernel.org \
    --cc=bfoster@redhat.com \
    --cc=boqian.zy@alibaba-inc.com \
    --cc=ceph-devel@vger.kernel.org \
    --cc=david@fromorbit.com \
    --cc=hch@lst.de \
    --cc=james.liu@alibaba-inc.com \
    --cc=linux-xfs@vger.kernel.org \
    --cc=sheng.qiu@alibaba-inc.com \
    --cc=sweil@redhat.com \
    --cc=xiongwei.jiang@alibaba-inc.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.