All of lore.kernel.org
 help / color / mirror / Atom feed
From: Artem Bityutskiy <dedekind1@gmail.com>
To: Jan Kara <jack@suse.cz>
Cc: Fengguang Wu <fengguang.wu@intel.com>,
	Andrew Morton <akpm@linux-foundation.org>,
	Greg Thelen <gthelen@google.com>, Ying Han <yinghan@google.com>,
	"hannes@cmpxchg.org" <hannes@cmpxchg.org>,
	KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>,
	Rik van Riel <riel@redhat.com>, Mel Gorman <mgorman@suse.de>,
	Minchan Kim <minchan.kim@gmail.com>,
	Linux Memory Management List <linux-mm@kvack.org>,
	LKML <linux-kernel@vger.kernel.org>,
	Adrian Hunter <adrian.hunter@intel.com>
Subject: Re: [PATCH 5/9] writeback: introduce the pageout work
Date: Mon, 12 Mar 2012 16:21:49 +0200	[thread overview]
Message-ID: <1331562109.12037.54.camel@sauron.fi.intel.com> (raw)
In-Reply-To: <20120312140229.GG5998@quack.suse.cz>

On Mon, 2012-03-12 at 15:02 +0100, Jan Kara wrote:
> > The second part of the overall deletion job will be when we commit - the
> > updated version of the FS index will be written to the flash media.
>   Oh, I see. This is what I was missing. And I presume you always make sure
> to have enough space for new FS index so it cannot deadlock when trying to
> push out dirty pages.

Yes, this is one of the hardest part and this is what the budgeting
subsystem does. Every VFS call (even unlink()) first invokes something
like 'ubifs_budget_space()' with arguments describing the space needs,
and the budgeting subsystem will account for the space, including the
possibility of the index growth. And the budgeting subsystem actually
forces write-back when it sees that there is not enough free space for
the operation. Because all the calculations are pessimistic, write-back
helps: the data nodes are compressed, and so on. The budgeting subsystem
may also force commit, which will clarify many unclarities and make the
calculations more precise. If nothing helps - ENOSPC is reported. For
deletions we also have a bit of reserve space to prevent -ENOSPC when
you actually want to delete a file on full file-system.

But the shorted answer: yes, we reserve 2 times the current index size
of the space for the index growths.

Long time ago I tried to describe this and the surrounding issues here:
http://www.linux-mtd.infradead.org/doc/ubifs.html#L_spaceacc

-- 
Best Regards,
Artem Bityutskiy


WARNING: multiple messages have this Message-ID (diff)
From: Artem Bityutskiy <dedekind1@gmail.com>
To: Jan Kara <jack@suse.cz>
Cc: Fengguang Wu <fengguang.wu@intel.com>,
	Andrew Morton <akpm@linux-foundation.org>,
	Greg Thelen <gthelen@google.com>, Ying Han <yinghan@google.com>,
	"hannes@cmpxchg.org" <hannes@cmpxchg.org>,
	KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>,
	Rik van Riel <riel@redhat.com>, Mel Gorman <mgorman@suse.de>,
	Minchan Kim <minchan.kim@gmail.com>,
	Linux Memory Management List <linux-mm@kvack.org>,
	LKML <linux-kernel@vger.kernel.org>,
	Adrian Hunter <adrian.hunter@intel.com>
Subject: Re: [PATCH 5/9] writeback: introduce the pageout work
Date: Mon, 12 Mar 2012 16:21:49 +0200	[thread overview]
Message-ID: <1331562109.12037.54.camel@sauron.fi.intel.com> (raw)
In-Reply-To: <20120312140229.GG5998@quack.suse.cz>

On Mon, 2012-03-12 at 15:02 +0100, Jan Kara wrote:
> > The second part of the overall deletion job will be when we commit - the
> > updated version of the FS index will be written to the flash media.
>   Oh, I see. This is what I was missing. And I presume you always make sure
> to have enough space for new FS index so it cannot deadlock when trying to
> push out dirty pages.

Yes, this is one of the hardest part and this is what the budgeting
subsystem does. Every VFS call (even unlink()) first invokes something
like 'ubifs_budget_space()' with arguments describing the space needs,
and the budgeting subsystem will account for the space, including the
possibility of the index growth. And the budgeting subsystem actually
forces write-back when it sees that there is not enough free space for
the operation. Because all the calculations are pessimistic, write-back
helps: the data nodes are compressed, and so on. The budgeting subsystem
may also force commit, which will clarify many unclarities and make the
calculations more precise. If nothing helps - ENOSPC is reported. For
deletions we also have a bit of reserve space to prevent -ENOSPC when
you actually want to delete a file on full file-system.

But the shorted answer: yes, we reserve 2 times the current index size
of the space for the index growths.

Long time ago I tried to describe this and the surrounding issues here:
http://www.linux-mtd.infradead.org/doc/ubifs.html#L_spaceacc

-- 
Best Regards,
Artem Bityutskiy

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

  reply	other threads:[~2012-03-12 14:19 UTC|newest]

Thread overview: 116+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2012-02-28 14:00 [PATCH 0/9] [RFC] pageout work and dirty reclaim throttling Fengguang Wu
2012-02-28 14:00 ` Fengguang Wu
2012-02-28 14:00 ` [PATCH 1/9] memcg: add page_cgroup flags for dirty page tracking Fengguang Wu
2012-02-28 14:00   ` Fengguang Wu
2012-02-29  0:50   ` KAMEZAWA Hiroyuki
2012-02-29  0:50     ` KAMEZAWA Hiroyuki
2012-03-04  1:29     ` Fengguang Wu
2012-03-04  1:29       ` Fengguang Wu
2012-02-28 14:00 ` [PATCH 2/9] memcg: add dirty page accounting infrastructure Fengguang Wu
2012-02-28 14:00   ` Fengguang Wu
2012-02-28 22:37   ` Andrew Morton
2012-02-28 22:37     ` Andrew Morton
2012-02-29  0:27     ` Fengguang Wu
2012-02-29  0:27       ` Fengguang Wu
2012-02-28 14:00 ` [PATCH 3/9] memcg: add kernel calls for memcg dirty page stats Fengguang Wu
2012-02-28 14:00   ` Fengguang Wu
2012-02-29  1:10   ` KAMEZAWA Hiroyuki
2012-02-29  1:10     ` KAMEZAWA Hiroyuki
2012-02-28 14:00 ` [PATCH 4/9] memcg: dirty page accounting support routines Fengguang Wu
2012-02-28 14:00   ` Fengguang Wu
2012-02-28 15:15   ` Fengguang Wu
2012-02-28 15:15     ` Fengguang Wu
2012-02-28 22:45   ` Andrew Morton
2012-02-28 22:45     ` Andrew Morton
2012-02-29  1:15     ` KAMEZAWA Hiroyuki
2012-02-29  1:15       ` KAMEZAWA Hiroyuki
2012-02-28 14:00 ` [PATCH 5/9] writeback: introduce the pageout work Fengguang Wu
2012-02-28 14:00   ` Fengguang Wu
2012-02-29  0:04   ` Andrew Morton
2012-02-29  0:04     ` Andrew Morton
2012-02-29  2:31     ` Fengguang Wu
2012-02-29  2:31       ` Fengguang Wu
2012-02-29 13:28     ` Fengguang Wu
2012-02-29 13:28       ` Fengguang Wu
2012-03-01 11:04     ` Jan Kara
2012-03-01 11:04       ` Jan Kara
2012-03-01 11:41       ` Fengguang Wu
2012-03-01 11:41         ` Fengguang Wu
2012-03-01 16:50         ` Jan Kara
2012-03-01 16:50           ` Jan Kara
2012-03-01 19:46         ` Andrew Morton
2012-03-01 19:46           ` Andrew Morton
2012-03-03 13:25           ` Fengguang Wu
2012-03-03 13:25             ` Fengguang Wu
2012-03-07  0:37             ` Andrew Morton
2012-03-07  0:37               ` Andrew Morton
2012-03-07  5:40               ` Fengguang Wu
2012-03-07  5:40                 ` Fengguang Wu
2012-03-01 19:42       ` Andrew Morton
2012-03-01 19:42         ` Andrew Morton
2012-03-01 21:15         ` Jan Kara
2012-03-01 21:15           ` Jan Kara
2012-03-01 21:22           ` Andrew Morton
2012-03-01 21:22             ` Andrew Morton
2012-03-01 12:36     ` Fengguang Wu
2012-03-01 12:36       ` Fengguang Wu
2012-03-01 16:38       ` Jan Kara
2012-03-01 16:38         ` Jan Kara
2012-03-02  4:48         ` Fengguang Wu
2012-03-02  4:48           ` Fengguang Wu
2012-03-02  9:59           ` Jan Kara
2012-03-02  9:59             ` Jan Kara
2012-03-02 10:39             ` Fengguang Wu
2012-03-02 10:39               ` Fengguang Wu
2012-03-02 19:57               ` Andrew Morton
2012-03-02 19:57                 ` Andrew Morton
2012-03-03 13:55                 ` Fengguang Wu
2012-03-03 13:55                   ` Fengguang Wu
2012-03-03 14:27                   ` Fengguang Wu
2012-03-03 14:27                     ` Fengguang Wu
2012-03-04 11:13                     ` Fengguang Wu
2012-03-04 11:13                       ` Fengguang Wu
2012-03-07 15:48                   ` Artem Bityutskiy
2012-03-07 15:48                     ` Artem Bityutskiy
2012-03-09  7:31                     ` Fengguang Wu
2012-03-09  7:31                       ` Fengguang Wu
2012-03-09  9:51                       ` Jan Kara
2012-03-09  9:51                         ` Jan Kara
2012-03-09 10:24                         ` Artem Bityutskiy
2012-03-09 10:24                           ` Artem Bityutskiy
2012-03-09 16:10                         ` Artem Bityutskiy
2012-03-09 16:10                           ` Artem Bityutskiy
2012-03-09 21:11                           ` Jan Kara
2012-03-09 21:11                             ` Jan Kara
2012-03-12 12:36                             ` Artem Bityutskiy
2012-03-12 12:36                               ` Artem Bityutskiy
2012-03-12 14:02                               ` Jan Kara
2012-03-12 14:02                                 ` Jan Kara
2012-03-12 14:21                                 ` Artem Bityutskiy [this message]
2012-03-12 14:21                                   ` Artem Bityutskiy
2012-03-09 10:15                   ` Jan Kara
2012-03-09 10:15                     ` Jan Kara
2012-03-09 15:10                     ` Fengguang Wu
2012-03-09 15:10                       ` Fengguang Wu
2012-02-29 13:51   ` [PATCH v2 " Fengguang Wu
2012-02-29 13:51     ` Fengguang Wu
2012-03-01 13:35     ` Fengguang Wu
2012-03-01 13:35       ` Fengguang Wu
2012-03-02  6:22       ` [PATCH v3 " Fengguang Wu
2012-03-02  6:22         ` Fengguang Wu
2012-02-28 14:00 ` [PATCH 6/9] vmscan: dirty reclaim throttling Fengguang Wu
2012-02-28 14:00   ` Fengguang Wu
2012-02-28 14:00 ` [PATCH 7/9] mm: pass __GFP_WRITE to memcg charge and reclaim routines Fengguang Wu
2012-02-28 14:00   ` Fengguang Wu
2012-02-28 14:00 ` [PATCH 8/9] mm: dont set __GFP_WRITE on ramfs/sysfs writes Fengguang Wu
2012-02-28 14:00   ` Fengguang Wu
2012-03-01 10:13   ` Johannes Weiner
2012-03-01 10:13     ` Johannes Weiner
2012-03-01 10:30     ` Fengguang Wu
2012-03-01 10:30       ` Fengguang Wu
2012-02-28 14:00 ` [PATCH 9/9] mm: debug vmscan waits Fengguang Wu
2012-02-28 14:00   ` Fengguang Wu
2012-03-02  6:59   ` [RFC PATCH] mm: don't treat anonymous pages as dirtyable pages Fengguang Wu
2012-03-02  6:59     ` Fengguang Wu
2012-03-02  7:18     ` Fengguang Wu
2012-03-02  7:18       ` Fengguang Wu

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1331562109.12037.54.camel@sauron.fi.intel.com \
    --to=dedekind1@gmail.com \
    --cc=adrian.hunter@intel.com \
    --cc=akpm@linux-foundation.org \
    --cc=fengguang.wu@intel.com \
    --cc=gthelen@google.com \
    --cc=hannes@cmpxchg.org \
    --cc=jack@suse.cz \
    --cc=kamezawa.hiroyu@jp.fujitsu.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=mgorman@suse.de \
    --cc=minchan.kim@gmail.com \
    --cc=riel@redhat.com \
    --cc=yinghan@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.