From: Martin <m_btrfs@ml1.co.uk>
To: linux-btrfs@vger.kernel.org
Subject: Re: Btrfs filesystem freezing during snapshots
Date: Mon, 26 May 2014 16:20:55 +0100 [thread overview]
Message-ID: <llvm4o$9uv$1@ger.gmane.org> (raw)
In-Reply-To: <CA+3u+RcGa2Xr+mzwGL-V89A7DEa05B_NS+cgS-Es1b3d8b5xKg@mail.gmail.com>
On 26/05/14 13:28, David Bloquel wrote:
> Hi,
>
> I have a problem with my btrfs filesystem which is freezing when I am
> doing snapshots.
>
> I have a cron that is snapshoting around 70 sub volume every ten
> minutes. The sub volumes that btrfs is snapshoting are containers
> folders that are running through my virtual environment.
> Sub directories that btrfs is snapshoting are not that big (from 500MB
> to 10GB max and usually around 3GB) but there is a lot of IO on the
> filesystem because of the intensive use of the CTs and VMs.
>
> At some point the snapshot process becomes really slow, at first it
> snapshot around one folder per seconds but then after a while it can
> take 30seconds or even few minutes to snapshot one single sub volumes.
> Subvolumes are really similar to each other in size and number of
> files so there is no reason that it takes 1second for one sub volume
> and then 3minutes for another one.
>
> Moreover when my snapshot cron is running all my vms and containers
> are slowing down until the whole filesystem freezes which leads to
> frozen CT and VMs (which is a real problem for me).
>
> Moreover I can see that my CPU load is really high during the process.
>
> when I'm am looking to dmesg there is a lot of messages of this kind:
>
> [96537.686467] BTRFS debug (device drbd0): unlinked 290 orphans
[...]
That looks to be running on top of drbd which will add a network write
overhead (unless you are dangerously running asynchronously!). Hence you
will see IO speed related limits a little sooner...
However, I will guess that your primary problem is likely due to
accumulating fragmentation due to adding ever more snapshots every 10
mins for the VMs/containers.
There are other people far more practised here than I, but some guesses
to try are:
Use "nocow" for the VM images (and container images);
Try using the btrfs auto defrag (beware your IO speed limit vs file size
to be defragged);
Avoid accumulating too many versions of any one snapshot.
Note also the "experimental" status for btrfs... I'm sure you will have
noticed the previous race problems for deleting snapshots.
Aside: I've held off from using kernel 3.12 and 3.13 due to curious
happenings on my test system. kernel 3.14.4 is behaving well so far.
Hope that gives a few clues.
Good luck,
Martin
next prev parent reply other threads:[~2014-05-26 15:21 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-05-26 12:28 Btrfs filesystem freezing during snapshots David Bloquel
2014-05-26 15:20 ` Martin [this message]
2014-05-26 16:19 ` Russell Coker
2014-05-26 15:39 ` Duncan
2014-05-26 16:39 ` Roman Mamedov
2014-05-26 17:02 ` Roman Mamedov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='llvm4o$9uv$1@ger.gmane.org' \
--to=m_btrfs@ml1.co.uk \
--cc=linux-btrfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).