From: Jan Kara <jack@suse.cz>
To: "Theodore Y. Ts'o" <tytso@mit.edu>
Cc: Jan Kara <jack@suse.cz>, linux-ext4@vger.kernel.org
Subject: Re: [PATCH 18/22] jbd2: Reserve space for revoke descriptor blocks
Date: Wed, 23 Oct 2019 15:27:19 +0200 [thread overview]
Message-ID: <20191023132719.GB31271@quack2.suse.cz> (raw)
In-Reply-To: <20191021214754.GC24015@mit.edu>
On Mon 21-10-19 17:47:54, Theodore Y. Ts'o wrote:
> On Fri, Oct 04, 2019 at 12:06:04AM +0200, Jan Kara wrote:
> > Extend functions for starting, extending, and restarting transaction
> > handles to take number of revoke records handle must be able to
> > accommodate. These functions then make sure transaction has enough
> > credits to be able to store resulting revoke descriptor blocks. Also
> > revoke code tracks number of revoke records created by a handle to catch
> > situation where some place didn't reserve enough space for revoke
> > records. Similarly to standard transaction credits, space for unused
> > reserved revoke records is released when the handle is stopped.
> >
> > On the ext4 side we currently take a simplistic approach of reserving
> > space for 1024 revoke records for any transaction. This grows amount of
> > credits reserved for each handle only by a few and is enough for any
> > normal workload so that we don't hit warnings in jbd2. We will refine
> > the logic in following commits.
> >
> > Signed-off-by: Jan Kara <jack@suse.cz>
>
> So let me summarize the way I think this commit is handling things.
>
> 1) When a handle is created, the caller specifies how many revokes it
> plans to do. If during the life of the handle, more than this number
> of revokes are done, a warning will be emited.
Correct.
> 2) For the purposes of reserving transaction credits, when we start
> the handle we assume the worst case number of number of revoke
> descriptors necessary, and we reserve that much space, and we add it
> to t_oustanding_credits.
Again correct.
> 3) When we stop the handle, we decrement t_outstanding_credits by the
> number of blocks that were originally reserved for this handle --- but
> *not* the number of worst case revoke descriptor blocks needed. Which
> means that after the handle is started and then closed,
> t_outstanding_credits will be increased by ROUND_UP((max # of revoked
> blocks) / # of revoke blocks per block group descriptor).
>
> If we delete a large number of files which are but a single 4k block
> in data=journal mode, each deleted file will increase
> t_outstanding_credits by one block, even though we won't be using
> anywhere *near* that number of blocks for revoke blocks. So we will
> end up closing the transactions *much* earlier than we would have.
Right. Any handle that revokes at least one block will reserve at least one
block for revoke descriptor. I agree that will overestimate number of
necessary revoke blocks heavily in some cases. If you think that's
problematic, I can refine the logic so that rounding errors don't
accumulate that much (probably by tracking exact number of revokes in the
transaction).
> It also means that t_outstanding_credits will be a much higher number
> that we would ever need, so it's not clear to me why it's worth it to
> decrement t_outstanding_credits in jbd2_journal_get_descriptor_buffer()
> and warn if it is less than zero.
Well, that tracking is a sanity check that we did reserve enough descriptor
blocks for each transaction.
> And it goes back to the question I had asked earler: "so what is the
> formal definition of t_outstanding_credits after this patch series,
> anyway"?
That should be answered in my previous answer.
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
next prev parent reply other threads:[~2019-10-23 13:27 UTC|newest]
Thread overview: 101+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-10-03 22:05 [PATCH 0/19 v3] ext4: Fix transaction overflow due to revoke descriptors Jan Kara
2019-10-03 22:05 ` [PATCH 01/22] jbd2: Fix possible overflow in jbd2_log_space_left() Jan Kara
2019-10-21 1:08 ` Theodore Y. Ts'o
2019-10-03 22:05 ` [PATCH 02/22] jbd2: Fixup stale comment in commit code Jan Kara
2019-10-21 1:08 ` Theodore Y. Ts'o
2019-10-03 22:05 ` [PATCH 03/22] ext4: Do not iput inode under running transaction in ext4_mkdir() Jan Kara
2019-10-21 1:21 ` Theodore Y. Ts'o
2019-10-24 10:19 ` Jan Kara
2019-10-24 12:09 ` Theodore Y. Ts'o
2019-10-24 13:37 ` Jan Kara
2019-11-04 12:35 ` Theodore Y. Ts'o
2019-10-03 22:05 ` [PATCH 04/22] ext4: Fix credit estimate for final inode freeing Jan Kara
2019-10-21 1:07 ` Theodore Y. Ts'o
2019-10-24 10:30 ` Jan Kara
2019-10-03 22:05 ` [PATCH 05/22] ext4: Fix ext4_should_journal_data() for EA inodes Jan Kara
2019-10-21 1:38 ` Theodore Y. Ts'o
2019-10-23 16:55 ` Jan Kara
2019-10-03 22:05 ` [PATCH 06/22] ext4: Use ext4_journal_extend() instead of jbd2_journal_extend() Jan Kara
2019-10-21 1:39 ` Theodore Y. Ts'o
2019-10-03 22:05 ` [PATCH 07/22] ext4: Avoid unnecessary revokes in ext4_alloc_branch() Jan Kara
2019-10-21 13:39 ` Theodore Y. Ts'o
2019-10-03 22:05 ` [PATCH 08/22] ext4: Provide function to handle transaction restarts Jan Kara
2019-10-21 16:20 ` Theodore Y. Ts'o
2019-10-23 16:25 ` Jan Kara
2019-10-03 22:05 ` [PATCH 09/22] ext4, jbd2: Provide accessor function for handle credits Jan Kara
2019-10-21 16:21 ` Theodore Y. Ts'o
2019-10-03 22:05 ` [PATCH 10/22] ocfs2: Use accessor function for h_buffer_credits Jan Kara
2019-10-21 16:21 ` Theodore Y. Ts'o
2019-10-03 22:05 ` [PATCH 11/22] jbd2: Fix statistics for the number of logged blocks Jan Kara
2019-10-21 16:24 ` Theodore Y. Ts'o
2019-10-03 22:05 ` [PATCH 12/22] jbd2: Reorganize jbd2_journal_stop() Jan Kara
2019-10-21 17:29 ` Theodore Y. Ts'o
2019-10-03 22:05 ` [PATCH 13/22] jbd2: Drop pointless check from jbd2_journal_stop() Jan Kara
2019-10-21 17:30 ` Theodore Y. Ts'o
2019-10-03 22:06 ` [PATCH 14/22] jbd2: Drop pointless wakeup " Jan Kara
2019-10-21 17:34 ` Theodore Y. Ts'o
2019-10-03 22:06 ` [PATCH 15/22] jbd2: Factor out common parts of stopping and restarting a handle Jan Kara
2019-10-21 17:49 ` Theodore Y. Ts'o
2019-10-23 16:17 ` Jan Kara
2019-11-04 12:36 ` Theodore Y. Ts'o
2019-11-04 12:59 ` Jan Kara
2019-10-03 22:06 ` [PATCH 16/22] jbd2: Account descriptor blocks into t_outstanding_credits Jan Kara
2019-10-21 21:04 ` Theodore Y. Ts'o
2019-10-23 13:09 ` Jan Kara
2019-10-03 22:06 ` [PATCH 17/22] jbd2: Drop jbd2_space_needed() Jan Kara
2019-10-21 21:05 ` Theodore Y. Ts'o
2019-10-03 22:06 ` [PATCH 18/22] jbd2: Reserve space for revoke descriptor blocks Jan Kara
2019-10-21 21:47 ` Theodore Y. Ts'o
2019-10-23 13:27 ` Jan Kara [this message]
2019-10-03 22:06 ` [PATCH 19/22] jbd2: Rename h_buffer_credits to h_total_credits Jan Kara
2019-10-21 21:48 ` Theodore Y. Ts'o
2019-10-03 22:06 ` [PATCH 20/22] jbd2: Make credit checking more strict Jan Kara
2019-10-21 22:29 ` Theodore Y. Ts'o
2019-10-23 13:30 ` Jan Kara
2019-10-03 22:06 ` [PATCH 21/22] ext4: Reserve revoke credits for freed blocks Jan Kara
2019-10-21 23:18 ` Theodore Y. Ts'o
2019-10-23 16:13 ` Jan Kara
2019-11-04 13:08 ` Theodore Y. Ts'o
2019-11-05 8:31 ` Jan Kara
2019-10-03 22:06 ` [PATCH 22/22] jbd2: Provide trace event for handle restarts Jan Kara
2019-10-21 23:18 ` Theodore Y. Ts'o
2019-10-19 19:19 ` [PATCH 0/19 v3] ext4: Fix transaction overflow due to revoke descriptors Theodore Y. Ts'o
2019-10-24 13:09 ` Jan Kara
2019-10-24 15:12 ` Jan Kara
2019-11-04 3:32 ` Theodore Y. Ts'o
2019-11-04 11:22 ` Jan Kara
2019-11-04 13:09 ` Theodore Y. Ts'o
2019-11-05 16:44 ` [PATCH 0/25 " Jan Kara
2019-11-05 16:44 ` [PATCH 01/25] jbd2: Fix possible overflow in jbd2_log_space_left() Jan Kara
2019-11-05 16:44 ` [PATCH 02/25] jbd2: Fixup stale comment in commit code Jan Kara
2019-11-05 16:44 ` [PATCH 03/25] jbd2: Completely fill journal descriptor blocks Jan Kara
2019-11-05 16:44 ` [PATCH 04/25] ext4: Move marking of handle as sync to ext4_add_nondir() Jan Kara
2019-11-05 16:44 ` [PATCH 05/25] ext4: Do not iput inode under running transaction Jan Kara
2019-11-05 16:44 ` [PATCH 06/25] ext4: Fix credit estimate for final inode freeing Jan Kara
2019-11-05 21:00 ` Theodore Y. Ts'o
2019-11-05 16:44 ` [PATCH 07/25] ext4: Fix ext4_should_journal_data() for EA inodes Jan Kara
2019-11-05 16:44 ` [PATCH 08/25] ext4: Use ext4_journal_extend() instead of jbd2_journal_extend() Jan Kara
2019-11-05 16:44 ` [PATCH 09/25] ext4: Avoid unnecessary revokes in ext4_alloc_branch() Jan Kara
2019-11-05 16:44 ` [PATCH 10/25] ext4: Provide function to handle transaction restarts Jan Kara
2019-11-05 16:44 ` [PATCH 11/25] ext4, jbd2: Provide accessor function for handle credits Jan Kara
2019-11-05 16:44 ` [PATCH 12/25] ocfs2: Use accessor function for h_buffer_credits Jan Kara
2019-11-05 16:44 ` [PATCH 13/25] jbd2: Fix statistics for the number of logged blocks Jan Kara
2019-11-05 16:44 ` [PATCH 14/25] jbd2: Reorganize jbd2_journal_stop() Jan Kara
2019-11-05 16:44 ` [PATCH 15/25] jbd2: Drop pointless check from jbd2_journal_stop() Jan Kara
2019-11-05 16:44 ` [PATCH 16/25] jbd2: Drop pointless wakeup " Jan Kara
2019-11-05 16:44 ` [PATCH 17/25] jbd2: Factor out common parts of stopping and restarting a handle Jan Kara
2019-11-05 16:44 ` [PATCH 18/25] jbd2: Account descriptor blocks into t_outstanding_credits Jan Kara
2019-11-05 16:44 ` [PATCH 19/25] jbd2: Drop jbd2_space_needed() Jan Kara
2019-11-05 16:44 ` [PATCH 20/25] jbd2: Reserve space for revoke descriptor blocks Jan Kara
2019-11-15 7:52 ` Eric Biggers
2019-11-15 10:02 ` Jan Kara
2019-11-15 14:20 ` Theodore Y. Ts'o
2019-11-15 17:10 ` Eric Biggers
2019-11-05 16:44 ` [PATCH 21/25] jbd2: Rename h_buffer_credits to h_total_credits Jan Kara
2019-11-05 16:44 ` [PATCH 22/25] jbd2: Make credit checking more strict Jan Kara
2019-11-05 16:44 ` [PATCH 23/25] ext4: Reserve revoke credits for freed blocks Jan Kara
2019-11-05 16:44 ` [PATCH 24/25] jbd2: Provide trace event for handle restarts Jan Kara
2019-11-05 16:44 ` [PATCH 25/25] jbd2: Fine tune estimate of necessary descriptor blocks Jan Kara
2019-11-05 21:04 ` [PATCH 0/25 v3] ext4: Fix transaction overflow due to revoke descriptors Theodore Y. Ts'o
[not found] ` <20191112220614.GA11089@mit.edu>
[not found] ` <20191113094545.GC6367@quack2.suse.cz>
2019-11-14 5:26 ` [PATCH 0/19 " Theodore Y. Ts'o
2019-11-14 8:49 ` Jan Kara
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20191023132719.GB31271@quack2.suse.cz \
--to=jack@suse.cz \
--cc=linux-ext4@vger.kernel.org \
--cc=tytso@mit.edu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).