All of lore.kernel.org
 help / color / mirror / Atom feed
From: "Xin Zhou" <xin.zhou@gmx.com>
To: "Peter Becker" <floyd.net@gmail.com>
Cc: linux-btrfs <linux-btrfs@vger.kernel.org>
Subject: Re: [markfasheh/duperemove] Why blocksize is limit to 1MB?
Date: Sun, 1 Jan 2017 05:38:24 +0100	[thread overview]
Message-ID: <trinity-99778bf7-78b4-4455-8c1a-32e20c3e71e8-1483245504786@3capp-mailcom-bs08> (raw)
In-Reply-To: <CAEtw4r0-wgdQj4nO9HxCFvzXpRVhU=nqcKxRmDUsFVBuVrfong@mail.gmail.com>

Hi,

In general, the larger the block / chunk size is, the less dedup can be achieved.
1M is already a little bit too big in size.

Thanks,
Xin

 
 

Sent: Friday, December 30, 2016 at 12:28 PM
From: "Peter Becker" <floyd.net@gmail.com>
To: linux-btrfs <linux-btrfs@vger.kernel.org>
Subject: [markfasheh/duperemove] Why blocksize is limit to 1MB?
Hello, i have a 8 TB volume with multiple files with hundreds of GB each.
I try to dedupe this because the first hundred GB of many files are identical.
With 128KB blocksize with nofiemap and lookup-extends=no option, will
take more then a week (only dedupe, previously hashed). So i tryed -b
100M but this returned me an error: "Blocksize is bounded ...".

The reason is that the blocksize is limit to

#define MAX_BLOCKSIZE (1024U*1024)

But i can't found any description why.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

  reply	other threads:[~2017-01-01  4:38 UTC|newest]

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-12-30 20:28 [markfasheh/duperemove] Why blocksize is limit to 1MB? Peter Becker
2017-01-01  4:38 ` Xin Zhou [this message]
2017-01-02 12:32   ` Peter Becker
2017-01-02 19:36     ` Xin Zhou
2017-01-03 12:40 ` Austin S. Hemmelgarn
2017-01-03 19:24   ` Peter Becker
2017-01-03 23:07     ` Hans van Kranenburg
2017-01-03 23:12       ` Peter Becker
2017-01-03 23:43         ` Hans van Kranenburg
2017-01-04  0:08           ` Martin Raiber
     [not found]   ` <CAEtw4r3mUA_4vcS-dbxagQn3NPRh8Cxcz0iF0L7jHwv5c9Ui+g@mail.gmail.com>
     [not found]     ` <7b0c897f-844c-e7f4-0ce7-c9f888b95983@gmail.com>
2017-01-03 20:20       ` Peter Becker
2017-01-03 20:40         ` Austin S. Hemmelgarn
2017-01-03 21:35           ` Peter Becker
2017-01-04 12:58             ` Austin S. Hemmelgarn
2017-01-04 14:42               ` Peter Becker
2017-01-09  1:09               ` Zygo Blaxell
2017-01-09  9:29                 ` Peter Becker
2017-01-10  4:12                   ` Zygo Blaxell
2017-01-03 20:21       ` Fwd: " Peter Becker

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=trinity-99778bf7-78b4-4455-8c1a-32e20c3e71e8-1483245504786@3capp-mailcom-bs08 \
    --to=xin.zhou@gmx.com \
    --cc=floyd.net@gmail.com \
    --cc=linux-btrfs@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.