* Defragmentation vw. Deduplication
@ 2021-03-02 21:31 Christian Völker
2021-03-04 4:54 ` Zygo Blaxell
0 siblings, 1 reply; 2+ messages in thread
From: Christian Völker @ 2021-03-02 21:31 UTC (permalink / raw)
To: linux-btrfs
Hi all,
might be a simple question but I did not find a trustable source for this.
BTRFS uses COW which might lead to fragmentation.
So using "btrfs fi defrag -r /mnt" will bring most file extend in a row
and copy previously deduplicated extends.
Obviously this uses more disk space. This is not what I want, but I need
to run "defrag" because I initially skipped the "compress=zstd" option
when mounting. So many files are stored without compression. Therefor I
neede to do the "defrag".
I am now unsure about the deduplication itself. How does it work?
I create a file in a directory (ie on Monday). Some days later I create
a file which has some extents with equal data. Does btrfs recon the
equal extents and does it doe deduplication then? Or does it only do
deduplication when ie "cp --reflink" is used?
However as I needed the compression and not the defragmentation is there
any way to add compression and recreate deduplication later?
Sorry if this is a dumb question.
/KNEBB
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: Defragmentation vw. Deduplication
2021-03-02 21:31 Defragmentation vw. Deduplication Christian Völker
@ 2021-03-04 4:54 ` Zygo Blaxell
0 siblings, 0 replies; 2+ messages in thread
From: Zygo Blaxell @ 2021-03-04 4:54 UTC (permalink / raw)
To: Christian Völker; +Cc: linux-btrfs
On Tue, Mar 02, 2021 at 10:31:08PM +0100, Christian Völker wrote:
> Hi all,
>
> might be a simple question but I did not find a trustable source for this.
>
> BTRFS uses COW which might lead to fragmentation.
> So using "btrfs fi defrag -r /mnt" will bring most file extend in a row and
> copy previously deduplicated extends.
> Obviously this uses more disk space. This is not what I want, but I need to
> run "defrag" because I initially skipped the "compress=zstd" option when
> mounting. So many files are stored without compression. Therefor I neede to
> do the "defrag".
>
> I am now unsure about the deduplication itself. How does it work?
> I create a file in a directory (ie on Monday). Some days later I create a
> file which has some extents with equal data. Does btrfs recon the equal
> extents and does it doe deduplication then?
You have to run a deduper like duperemove or bees to get that behavior. See
https://btrfs.wiki.kernel.org/index.php/Deduplication
There are a number of different tools that are also available for other
filesystems as well, e.g. jdupes, which dedupe entire identical files
as opposed to blocks. If most of your duplicates are entire files then
this can significantly increase dedupe speed and reduce dedupe memory
compared to the block-oriented dedupers.
> Or does it only do deduplication when ie "cp --reflink" is used?
With cp --reflink there is no duplication in the first place.
You can choose a deduper depending on how much RAM and IO you want to
commit to the task, whether you need block- or file-oriented dedupe, etc.
If you don't run a deduper then duplicate data stays duplicated.
If you don't run defrag then reflinked data stays reflinked.
> However as I needed the compression and not the defragmentation is there any
> way to add compression and recreate deduplication later?
You can simply run a deduper after compression is done.
> Sorry if this is a dumb question.
>
> /KNEBB
>
>
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2021-03-04 4:56 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-02 21:31 Defragmentation vw. Deduplication Christian Völker
2021-03-04 4:54 ` Zygo Blaxell
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.