* [rfc] Ignore Fsync Calls in Laptop_Mode @ 2011-05-19 13:34 Dennis Jansen 2011-05-19 13:43 ` Alan Cox ` (2 more replies) 0 siblings, 3 replies; 53+ messages in thread From: Dennis Jansen @ 2011-05-19 13:34 UTC (permalink / raw) To: linux-kernel; +Cc: akpm, tytso This is my first proper kernel code proposal so please bear with me! =Summary for busy kernel hackers= Problem: laptop_mode wants to keep applications from waking the hard disks but fsync calls can "sneak through". (IMHO this is a bug.) Proposed solution: Pretend the fsync was executed and successful. Insert two lines into the fsync and fdatasync calls in fs/sync.c: if (unlikely(laptop_mode)) return 0; =More background information and flamewar suggestions below= What is Laptop_Mode? Laptop_Mode is a feature that saves power by forced buffering of hard disk writes in memory. It flushes the data out to disk only e.g. every 15 minutes, means it can be off most of the time. Check this for Andrew's intro: http://lwn.net/Articles/1652/. It is disabled by default in most distributions afaik, e.g. http://samwel.tk/laptop_mode/packages/ubuntu. The Problem: Any time an application uses fsync, the hard disk wakes up despite laptop_mode being active. This not only causes higher energy consumption than necessary (either because laptop_mode can't be used or because of constat hd spin ups), it also damages hard disks, as they are created to last only a certain amount of head load cycles. See https://ata.wiki.kernel.org/index.php/Known_issues#Drives_which_perform_frequent_head_unloads_under_Linux for details. Applications that use fsync include LibreOffice, Firefox 3 (used to a lot), Syslog (https://www-304.ibm.com/support/docview.wss?uid=swg21255028) and e.g. anything which uses sqlite in synchronious mode (http://www.sqlite.org/speed.html). The Fix was already suggested by tytso here, but not followed up: http://tytso.livejournal.com/2009/03/15/ Existing Workarounds: - Libraries: Severeal libraries exist which you can use with LD_PRELOAD to disable fsync for certain applications. - Applications: You could disable fsync in the applications. Why are they not enough? - Lib raries: Libraries don't allow a switch to comfortably enable and disable fsync. You always miss a program. It's a hassle. - Applications: Disabling the fsync call may not make sense in the libraries. One good example is Office Software. Normally, I do want it to fsync the file after saving. But on battery I prefer the data to be left in memory. So fixing LibreOffice would not make a lot of sense. Not saving there would also not be ideal: If it crashes, I would end up with nothing. If I keep autosave active, I will at least be save from the software crashing. The big advantige of this proposal is not only that laptop_mode will work as advertized and prevent all automatic spin ups to write data. ("sync" on the command line still works.), but also that this behavior is already implemented in many distributions via laptop-mode to switch depending on battery/AC status. Side effects: This would make laptop_mode a great way to cheat benchmarks that rely on fsync for their timing, (to ensure data is actually written to disk before the timer ends). People who don't care so much about data safety can gain some disk speed with one switch. Crashing applications are _not_ affected. Their data is written to memory, just not flushed to disk. "But this means work will be lost" This is a complaint against this solution made e.g. here in the second part. http://lwn.net/Articles/323780/ Yes, this is true. But Laptop_Mode makes that very, very clear. The configuration file actually calls the setting "LM_BATT_MAX_LOST_WORK_SECONDS". Also laptop_mode decreases and then disables itself when the battery runs down. And finally, I don't know distributions that enable it out of the box. If a user does enable laptop_mode without reading anything at all about what it means, I don't think he should be protected. If a distribution uses it by default, it should warn users about possible consequences. In short: That it makes laptop_mode work as advertised IHO is no valid point against this solution. And if, we could consider a sending a printk the first time an fsync is skipped. "This is not necessary with SSDs" I'm not sure, but I think that SSDs will also keep an active SATA connection if you write to them all the time. They also have standby modes which also save power. But yes, this make not or hardly make sense with them. But as long as you don't send me one, I still like a fix. Also, this might extend SSD lifetime as this reduces the total times a sector is written. Testing: I've been using this workaround on my netbook for over six months now. It works as expected for me with all software in a Ubuntu 9.10 environment and saves me at least 0.5 Watt or roughly 10 % battery time - without destroying my hard disk. I have seen no negative side effects. What I didn't test: I didn't test this on different platforms and environments. But as it's generic code I expect no different behavior. I didn't benchmark how this might affect performance due to the additional if() check for every fsync call. But as the fsync is rather expensive and not used *that* much, I think it should have not noticeable impact. Implementation Ideas: If this patch does turn out to affect some scenarios negatively, a switch would be necessary. Laptop_Mode is in proc, but we don't want more files in there. In any case, a configuration on a by device level would make sense (e.g. no fsync on sda when laptop_mode active). That would fit nicely into sys and allow to disable fsync only for the hard disk, but not for the SSD or usb stick if both are present. But let's try not to blow this up too much. Where's the Trojan Horse? Laptop_Mode was the trojan horse, this is a soldier. There's no need. This already is "_obviously_ good at first sight". ;-) Great Flame Topics (please add [flame] to the subject for better overview!): - User Space vs. Kernel Space Fixes - Is this obsoleted by SSDs - Is this a bug or a feature Thanks for your feedback! Thanks for the feedback at LinuxTag Kernel Kwestioning. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-19 13:34 [rfc] Ignore Fsync Calls in Laptop_Mode Dennis Jansen @ 2011-05-19 13:43 ` Alan Cox [not found] ` <BANLkTikFRwPY_qOQpPmCmNUJbBUsGcuGUw@mail.gmail.com> 2011-05-19 15:02 ` D. Jansen 2011-05-20 3:39 ` Dave Chinner 2011-05-20 15:28 ` Valdis.Kletnieks 2 siblings, 2 replies; 53+ messages in thread From: Alan Cox @ 2011-05-19 13:43 UTC (permalink / raw) To: d.g.jansen; +Cc: Dennis.Jansen, linux-kernel, akpm, tytso > Problem: laptop_mode wants to keep applications from waking the hard > disks but fsync calls can "sneak through". (IMHO this is a bug.) Its a standards requirement and thing many apps rely on. Also you can do it perfectly well in user space for all apps. Thats why you can preconfigure preloads and it's why you have glic library source code! > Proposed solution: Pretend the fsync was executed and successful. > Insert two lines into the fsync and fdatasync calls in fs/sync.c: > if (unlikely(laptop_mode)) > return 0; Whoops you've just risked corrupting any app which relies on fsync for ordering if there is a crash. > "This is not necessary with SSDs" > I'm not sure, but I think that SSDs will also keep an active SATA > connection if you write to them all the time. They also have standby On SSD the impact is miniscule as far as I can tell. A graph would tell you more. The "right" way to do this is more complicated by far. An fsync is an ordering guarantee so you need to implement that ordering guarantee even if you don't force writes to physical media. That at least cuts down most failures (but not all - eg commits with a network component such as email receives) Anyway you can do it in userspace trivially if you want. ^ permalink raw reply [flat|nested] 53+ messages in thread
[parent not found: <BANLkTikFRwPY_qOQpPmCmNUJbBUsGcuGUw@mail.gmail.com>]
[parent not found: <20110519153928.40521b93@lxorguk.ukuu.org.uk>]
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode [not found] ` <20110519153928.40521b93@lxorguk.ukuu.org.uk> @ 2011-05-19 15:01 ` D. Jansen 0 siblings, 0 replies; 53+ messages in thread From: D. Jansen @ 2011-05-19 15:01 UTC (permalink / raw) To: Alan Cox; +Cc: linux-kernel, akpm, tytso On Thu, May 19, 2011 at 4:39 PM, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote: >> > That at least cuts down most failures (but not all - eg commits with a network >> > component such as email receives) >> >> I don't understand your email example. > > Think about sendmail and SMTP > > Mail arrives > Commit to disk > fsync > [X] > Confirm receipt to other system > Other system removes it from the queue it holds > > So if it crashes at X even though you've kept ordering and you've > got an internally consistent view (as if it crashed earlier), that > has been observed and acted upon by another system - so your email > just went into the cosmic trashcan. > > Ok, I think I understand your example. But isn't that the user's risk and more a problem of the general idea of laptop mode? If I enable laptop_mode and set MAX_LOST_WORK_SECONDS, I risk to lose data. I would say that includes email and whatever else may happen. (And if I don't switch it on, I risk running out of battery and being unable to create data...). btw. I don't think it would be a good idea for people to use sendmail or other crucial data storage applications with laptop_mode active. I would see this as a switch for e.g. notebooks at conferences, in class, lectures, etc. (sorry, lost the CCs) ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-19 13:43 ` Alan Cox [not found] ` <BANLkTikFRwPY_qOQpPmCmNUJbBUsGcuGUw@mail.gmail.com> @ 2011-05-19 15:02 ` D. Jansen 2011-05-20 15:34 ` Valdis.Kletnieks 1 sibling, 1 reply; 53+ messages in thread From: D. Jansen @ 2011-05-19 15:02 UTC (permalink / raw) To: Alan Cox; +Cc: linux-kernel, akpm, tytso On Thu, May 19, 2011 at 3:43 PM, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote: >> Problem: laptop_mode wants to keep applications from waking the hard >> disks but fsync calls can "sneak through". (IMHO this is a bug.) > > Its a standards requirement and thing many apps rely on. Also you can do > it perfectly well in user space for all apps. Thats why you can > preconfigure preloads and it's why you have glic library source code! Yes, it's another possibility I've considered. So then the patch should go into glibc and glibc check for active laptop_mode? But if implement it inside a library won't I get the same ordering guarantee failure? > >> Proposed solution: Pretend the fsync was executed and successful. >> Insert two lines into the fsync and fdatasync calls in fs/sync.c: >> if (unlikely(laptop_mode)) >> return 0; > > Whoops you've just risked corrupting any app which relies on fsync for > ordering if there is a crash. (...) > The "right" way to do this is more complicated by far. An fsync is an > ordering guarantee so you need to implement that ordering guarantee even > if you don't force writes to physical media. Ok. No problem, so the exit point would be a different one. > That at least cuts down most failures (but not all - eg commits with a network > component such as email receives) I don't understand your email example. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-19 15:02 ` D. Jansen @ 2011-05-20 15:34 ` Valdis.Kletnieks 0 siblings, 0 replies; 53+ messages in thread From: Valdis.Kletnieks @ 2011-05-20 15:34 UTC (permalink / raw) To: D. Jansen; +Cc: Alan Cox, linux-kernel, akpm, tytso [-- Attachment #1: Type: text/plain, Size: 1693 bytes --] On Thu, 19 May 2011 17:02:21 +0200, "D. Jansen" said: > On Thu, May 19, 2011 at 3:43 PM, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote: > > That at least cuts down most failures (but not all - eg commits with a network > > component such as email receives) > > I don't understand your email example. If you don't understand Alan's example, maybe you shouldn't be messing with code that's used for correctness guarantees. But I'll spell it out for you: You turn on laptop mode. You check your mail. You download 5 new messages, which your laptop then *assures* the mail server "I've got them, we're cool". Now, this is a pretty heavy responsibility (for example, RFC2821 says this: 6.1 Reliable Delivery and Replies by Email When the receiver-SMTP accepts a piece of mail (by sending a "250 OK" message in response to DATA), it is accepting responsibility for delivering or relaying the message. It must take this responsibility seriously. It MUST NOT lose the message for frivolous reasons, such as because the host later crashes or because of a predictable resource shortage. So - the mail server then deletes the mail because the laptop has told it "Yes, I've gotten it, it's stable, and even if I crash I won't lose it". But since the mail program's fsync() calls got suppressed, it really *isn't* stable, so when you crash, the mails are gone. Poof. Second order effects crop up after that - after you recover from the crash, your end has no memory of downloading the 5 messages, so it tries again - only to have the mail server say "No such message". This is the sort of inconsistency that gives guys at the support desk indigestion... Understand better now? [-- Attachment #2: Type: application/pgp-signature, Size: 227 bytes --] ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-19 13:34 [rfc] Ignore Fsync Calls in Laptop_Mode Dennis Jansen 2011-05-19 13:43 ` Alan Cox @ 2011-05-20 3:39 ` Dave Chinner 2011-05-20 6:01 ` D. Jansen 2011-05-20 15:28 ` Valdis.Kletnieks 2 siblings, 1 reply; 53+ messages in thread From: Dave Chinner @ 2011-05-20 3:39 UTC (permalink / raw) To: d.g.jansen; +Cc: linux-kernel, akpm, tytso On Thu, May 19, 2011 at 03:34:46PM +0200, Dennis Jansen wrote: > This is my first proper kernel code proposal so please bear with me! > > =Summary for busy kernel hackers= > Problem: laptop_mode wants to keep applications from waking the hard > disks but fsync calls can "sneak through". (IMHO this is a bug.) > > Proposed solution: Pretend the fsync was executed and successful. > Insert two lines into the fsync and fdatasync calls in fs/sync.c: > if (unlikely(laptop_mode)) > return 0; No, no, no, no, no, no, no, no, no, no. There is _absolutely no justification_ for putting people's data at risk like this. If you want to do make fsync/fdatasync calls no-ops, then go install libeatmydata on your systems. It's your data, and you make the decision to risk it, not us. And BTW, I just added your name to my "ignore fileystem/data corruption bug reports from these people" list, because that's exactly where you'll end up with if you follow this path. Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-20 3:39 ` Dave Chinner @ 2011-05-20 6:01 ` D. Jansen 2011-05-22 0:48 ` Dave Chinner 0 siblings, 1 reply; 53+ messages in thread From: D. Jansen @ 2011-05-20 6:01 UTC (permalink / raw) To: Dave Chinner; +Cc: linux-kernel, akpm, tytso On Fri, May 20, 2011 at 5:39 AM, Dave Chinner <david@fromorbit.com> wrote: > On Thu, May 19, 2011 at 03:34:46PM +0200, Dennis Jansen wrote: >> This is my first proper kernel code proposal so please bear with me! >> >> =Summary for busy kernel hackers= >> Problem: laptop_mode wants to keep applications from waking the hard >> disks but fsync calls can "sneak through". (IMHO this is a bug.) >> >> Proposed solution: Pretend the fsync was executed and successful. >> Insert two lines into the fsync and fdatasync calls in fs/sync.c: >> if (unlikely(laptop_mode)) >> return 0; > > No, no, no, no, no, no, no, no, no, no. > > There is _absolutely no justification_ for putting people's data at > risk like this. If you want to do make fsync/fdatasync calls > no-ops, then go install libeatmydata on your systems. It's your > data, and you make the decision to risk it, not us. 1. I thought I (may) make that decision by using laptop mode. 2. libeatmydata would _always_ be active. 3. A lib doesn't fix the ordering guarantee problem. 4. It's clear that it's not the right code. (And it is a rfc and my first one, too...) Any suggestions? Thanks! Dennis ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-20 6:01 ` D. Jansen @ 2011-05-22 0:48 ` Dave Chinner 2011-05-23 8:12 ` Oliver Neukum 0 siblings, 1 reply; 53+ messages in thread From: Dave Chinner @ 2011-05-22 0:48 UTC (permalink / raw) To: D. Jansen; +Cc: linux-kernel, akpm, tytso On Fri, May 20, 2011 at 08:01:17AM +0200, D. Jansen wrote: > On Fri, May 20, 2011 at 5:39 AM, Dave Chinner <david@fromorbit.com> wrote: > > On Thu, May 19, 2011 at 03:34:46PM +0200, Dennis Jansen wrote: > >> This is my first proper kernel code proposal so please bear with me! > >> > >> =Summary for busy kernel hackers= > >> Problem: laptop_mode wants to keep applications from waking the hard > >> disks but fsync calls can "sneak through". (IMHO this is a bug.) > >> > >> Proposed solution: Pretend the fsync was executed and successful. > >> Insert two lines into the fsync and fdatasync calls in fs/sync.c: > >> if (unlikely(laptop_mode)) > >> return 0; > > > > No, no, no, no, no, no, no, no, no, no. > > > > There is _absolutely no justification_ for putting people's data at > > risk like this. If you want to do make fsync/fdatasync calls > > no-ops, then go install libeatmydata on your systems. It's your > > data, and you make the decision to risk it, not us. > > 1. I thought I (may) make that decision by using laptop mode. Laptop mode does not change fsync guarantees, so no, you ar enot making a decision to throw data away when you select laptop mode. > 2. libeatmydata would _always_ be active. Your choice, really. Just use it as a preload for your apps that you want to avoid fsync for. > 3. A lib doesn't fix the ordering guarantee problem. A properly implemented filesystem will not have ordering problems just because fsyncs are not issued. > 4. It's clear that it's not the right code. (And it is a rfc and my > first one, too...) > > Any suggestions? Your problem to solve in userspace. Changing a generic, widely used kernel option like this is not allowed, and in general when there are ways in userspace of playing fast and loose with data, it's up to the user to do that in userspace, not the kernel... Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-22 0:48 ` Dave Chinner @ 2011-05-23 8:12 ` Oliver Neukum 2011-05-23 13:05 ` D. Jansen 2011-05-25 0:00 ` Dave Chinner 0 siblings, 2 replies; 53+ messages in thread From: Oliver Neukum @ 2011-05-23 8:12 UTC (permalink / raw) To: Dave Chinner; +Cc: D. Jansen, linux-kernel, akpm, tytso Am Sonntag, 22. Mai 2011, 02:48:33 schrieb Dave Chinner: > On Fri, May 20, 2011 at 08:01:17AM +0200, D. Jansen wrote: > > On Fri, May 20, 2011 at 5:39 AM, Dave Chinner <david@fromorbit.com> wrote: > > 1. I thought I (may) make that decision by using laptop mode. > > Laptop mode does not change fsync guarantees, so no, you ar enot > making a decision to throw data away when you select laptop mode. You do however decide to sync less often, resulting in a potentially larger loss of data. > > 3. A lib doesn't fix the ordering guarantee problem. > > A properly implemented filesystem will not have ordering problems > just because fsyncs are not issued. But user space will have this problem. A single task's sequence of write(); fsync(); write(); does give an implicit guarantee of ordering to user space. To keep that guarantee, you need kernel support, because only the kernel can issue the necessary barrier/flush commands. A filesystem is not required to make sure your writes hit the disk in the order you issued them, unless you use fsync, which gives an even larger guarantee, but also implies ordering. Now you may say that you want full fsync or nothing, but this is a personal preference. Regards Oliver ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-23 8:12 ` Oliver Neukum @ 2011-05-23 13:05 ` D. Jansen 2011-05-25 0:00 ` Dave Chinner 1 sibling, 0 replies; 53+ messages in thread From: D. Jansen @ 2011-05-23 13:05 UTC (permalink / raw) To: Oliver Neukum; +Cc: Dave Chinner, linux-kernel, akpm, tytso On Mon, May 23, 2011 at 10:12 AM, Oliver Neukum <oneukum@suse.de> wrote: >> > 3. A lib doesn't fix the ordering guarantee problem. >> >> A properly implemented filesystem will not have ordering problems >> just because fsyncs are not issued. > > But user space will have this problem. A single task's sequence of > write(); fsync(); write(); does give an implicit guarantee of ordering > to user space. To keep that guarantee, you need kernel support, > because only the kernel can issue the necessary barrier/flush commands. > A filesystem is not required to make sure your writes hit the disk > in the order you issued them, unless you use fsync, which gives > an even larger guarantee, but also implies ordering. If this is correct, we do need some form of kernel support. To ensure that one fsync means the write comes before another. Because how would it be implemented in user space to ensure write ordering without fsync? The library could only request the kernel to enforce write ordering. Otherwise it would have to catch and buffer all writes itself and then commit them with fsyncs in between. _That_ doesn't sound like a good idea at all... And the alternative would be that a write of 1. XXXXXXXXXXXXXXXX (file) 2. XXHHHHHXXXXXXX (write H) 3. XXHHHNNNNXXXX (write N) would become 1. XXXXXXXXXXXXXXXX (file) 2. XXXXXXNNNNXXXX (write N) 3. XXHHHHHNNXXXX (write H) And we have data that was never supposed to be on disk this way... ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-23 8:12 ` Oliver Neukum 2011-05-23 13:05 ` D. Jansen @ 2011-05-25 0:00 ` Dave Chinner 2011-05-25 6:50 ` Oliver Neukum 1 sibling, 1 reply; 53+ messages in thread From: Dave Chinner @ 2011-05-25 0:00 UTC (permalink / raw) To: Oliver Neukum; +Cc: D. Jansen, linux-kernel, akpm, tytso On Mon, May 23, 2011 at 10:12:06AM +0200, Oliver Neukum wrote: > Am Sonntag, 22. Mai 2011, 02:48:33 schrieb Dave Chinner: > > On Fri, May 20, 2011 at 08:01:17AM +0200, D. Jansen wrote: > > > On Fri, May 20, 2011 at 5:39 AM, Dave Chinner <david@fromorbit.com> wrote: > > > > 1. I thought I (may) make that decision by using laptop mode. > > > > Laptop mode does not change fsync guarantees, so no, you ar enot > > making a decision to throw data away when you select laptop mode. > > You do however decide to sync less often, resulting in a potentially > larger loss of data. > > > > 3. A lib doesn't fix the ordering guarantee problem. > > > > A properly implemented filesystem will not have ordering problems > > just because fsyncs are not issued. > > But user space will have this problem. A single task's sequence of > write(); fsync(); write(); does give an implicit guarantee of ordering > to user space. Oh, you're talking about application level write ordering. IO "ordering" in filesystem speak is about guaranteeing the order of data vs metadata writes for ensuring consistency after a crash (e.g. ext3/4 default "data=ordered" mode). that's what I was refering to, not anythign to do with applications. Besides, having to work out how to handle subtle application write ordering bugs because you changed fsync semantics is simply another reason for not changing behaviour in the first place. Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-25 0:00 ` Dave Chinner @ 2011-05-25 6:50 ` Oliver Neukum 2011-05-26 7:01 ` D. Jansen 0 siblings, 1 reply; 53+ messages in thread From: Oliver Neukum @ 2011-05-25 6:50 UTC (permalink / raw) To: Dave Chinner; +Cc: D. Jansen, linux-kernel, akpm, tytso Am Mittwoch, 25. Mai 2011, 02:00:03 schrieb Dave Chinner: > Oh, you're talking about application level write ordering. IO > "ordering" in filesystem speak is about guaranteeing the order of > data vs metadata writes for ensuring consistency after a crash > (e.g. ext3/4 default "data=ordered" mode). that's what I was > refering to, not anythign to do with applications. > > Besides, having to work out how to handle subtle application write > ordering bugs because you changed fsync semantics is simply another > reason for not changing behaviour in the first place. Sure. I'd say changing the behavior is right out. The question is whether we want an additional "superlaptop"-mode. And even in this case it seems to me that fsync() cannot be reduced to a nop because of ordering constraints. But perhaps we should then consider exporting an ordering primitive to user space. Regards Oliver -- - - - SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany - - - ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-25 6:50 ` Oliver Neukum @ 2011-05-26 7:01 ` D. Jansen 2011-05-26 10:49 ` Theodore Tso 2011-05-26 19:31 ` Matthew Garrett 0 siblings, 2 replies; 53+ messages in thread From: D. Jansen @ 2011-05-26 7:01 UTC (permalink / raw) To: Oliver Neukum; +Cc: Dave Chinner, linux-kernel, akpm, tytso On Wed, May 25, 2011 at 8:50 AM, Oliver Neukum <oneukum@suse.de> wrote: > > Am Mittwoch, 25. Mai 2011, 02:00:03 schrieb Dave Chinner: > > Oh, you're talking about application level write ordering. (...) > > Besides, having to work out how to handle subtle application write > > ordering bugs because you changed fsync semantics is simply another > > reason for not changing behaviour in the first place. > > Sure. I'd say changing the behavior is right out. The question is whether > we want an additional "superlaptop"-mode. Exactly. 0.5 Watts is a lot of energy on a modern system. And on some systems with proper link power management in sata and pcie it will be even higher (~1 Watt I think). Think not only of the work that will be lost if the system crashes (unlikely), but also of the work that could never be done because the battery was gone (a certainty). > > And even in this case it seems to me that fsync() cannot be reduced to > a nop because of ordering constraints. But perhaps we should then consider > exporting an ordering primitive to user space. That seems to be the big ordering issue. I had always assumed that user space writes (by the same app to the same file) would be committed in order. Is that really not the case? Wouldn't most app programmers assume ordering? Wouldn't that always possibly be an issue? Or do all the apps that require ordered writes use fsync. There will surely be some who require ordering but don't fsync. And without ordering, some apps won't be able to avoid fsync without data safety issues. It would seem that even ordering writes by default could not be a data safety issue. But I guess performance would be affected negatively in some cases. And behavior shouldn't be changed. So yes, it seems an ordering primitive would be a good idea. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-26 7:01 ` D. Jansen @ 2011-05-26 10:49 ` Theodore Tso [not found] ` <BANLkTimNjt4=9v33Z9Nr12xa6GmyJ-Ue5A@mail.gmail.com> 2011-05-26 19:31 ` Matthew Garrett 1 sibling, 1 reply; 53+ messages in thread From: Theodore Tso @ 2011-05-26 10:49 UTC (permalink / raw) To: D. Jansen; +Cc: Oliver Neukum, Dave Chinner, linux-kernel, akpm On May 26, 2011, at 3:01 AM, D. Jansen wrote: > That seems to be the big ordering issue. I had always assumed that > user space writes (by the same app to the same file) would be > committed in order. Is that really not the case? > > Wouldn't most app programmers assume ordering? Wouldn't that always > possibly be an issue? Or do all the apps that require ordered writes > use fsync. There will surely be some who require ordering but don't > fsync. And without ordering, some apps won't be able to avoid fsync > without data safety issues. I really don't like using the word "ordering" the way Dave used it, because it's a file system lingo that *always* confuses civilians. And "Insider" language like that isn't help for communication, unless you're certain there are only experts in the room... As Dave said earlier, "ordering" in the sense he was using it refers strictly to ensuring consistency after a crash. Now, there are two levels of consistency; one is file system level consistency, and the other is application level consistency. It used to be that desktop drives would lie about forcing data to disk in response to a FLUSH CACHE command, "yes sir, I promise the data is on the disk, sir!", because it resulted in higher WINBENCH scores. File systems engineers hated this, because a primary tool we have for assuring that file systems don't look like swiss cheese after a crash was completely unreliable. Fortunately, those disks have largely disappeared from the market place. The suggestion of making fsync a no-op is essentially asking for a knob that breaks application-level consistency the same way those broken hard drives broke file system consistency by making the FLUSH CACHE command unreliable. Maybe improving battery lifetime is a more honorable excuse than the purely mercenary goal of selling more disk drives, but it can still break applications after a crash. Now, you may think that you're prepared by that. After all, you're already prepared to say that you're willing to lose the last 15 minutes of work or whatever, right? Well, wrong. It's not so simple as that. If you're only talking about simple, flat, human-readable text files, maybe it would work that way. But what about complex, binary databases? Like sqllite databases used by Firefox and Chrome? Or MySQL databases? More and more, sophisticated applications, even desktop applications, are using these complex data stores, and the libraries which update these complex data stores rely on fsync() to prevent their database files from looking like swiss cheese. If you crash while fsync() has been disabled, the entire database file could be completely trashed, which could be hours, days, weeks, or months of work lost. So the resistance that people like Dave have to your proposal can be summed up by Confucious if you are Chinese: ""Never impose on others what you would not choose for yourself." Or if you are Jewish, the Rabbi Hillel said: "That which is hateful to you, do not do to your fellow. That is the whole Torah; the rest is the explanation; go and learn." Or if you are a Muslim, the Prophet Mohammed: "Hurt no one so that no one may hurt you." Breaking fsync() is like hard drives that break faith with file system authors by lying when they say everything is safely written to stable storage. And what are databases but complex file systems living inside a single file? -- Ted ^ permalink raw reply [flat|nested] 53+ messages in thread
[parent not found: <BANLkTimNjt4=9v33Z9Nr12xa6GmyJ-Ue5A@mail.gmail.com>]
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode [not found] ` <BANLkTimNjt4=9v33Z9Nr12xa6GmyJ-Ue5A@mail.gmail.com> @ 2011-05-26 13:31 ` Ted Ts'o 2011-05-26 16:05 ` D. Jansen 0 siblings, 1 reply; 53+ messages in thread From: Ted Ts'o @ 2011-05-26 13:31 UTC (permalink / raw) To: D. Jansen; +Cc: Oliver Neukum, akpm, linux-kernel, Dave Chinner On Thu, May 26, 2011 at 01:16:45PM +0200, D. Jansen wrote: > > exactly. so what now? This needs to be solved at the application level. So what we need is a coordinating daemon which sends a message to all applications saying, "ok, here's an opportunity to flush data to the disk; if you have dirty files, or databases that has data that needs to be flushed, send it now", and then a little while later, it sends a message to all applications, "ok, we're going to let the disk spin down now, hold your horses and don't send any data to the disk". If an application has too much data buffered in memory, and either because (a) it's getting nervous that it doesn't want to lose a huge amount of critical user data, or (b) it's about to exit, so it really needs to save its state to disk, or (c) the buffered data is taking up too much space, so it needs to flush its in-memory buffers, then the application sends a message to the coordinating daemon, saying, "please give me permission to write", and then waits for the "ok, you can write to the disk" message. That way, if you have multiple database-using applications active on the system at the same time, they can coordinate their access to the disk, and in a way such that when they stop writing, the data left in the application-level database is in a coherent state. The kernel can't do this by arbitrarily stopping writes from going to the disk because the kernel doesn't know --- and can't possibly know --- when it is a safe time to cut off write access to the disk. It can try and guess, by stopping write access after an fsync(), but it has to honor the fsync(). Consider what happens when an SMTP server sends back a 200 OK message --- that gives permission for its opposite number at the other end of the connection to delete the e-mail message, since the local SMTP server has promised that the e-mail has been saved to stable storage. It can't make that promise if fsync() has been rigged to lie. That's why at the end of the day, in order for application level data coherency, the application has to be in control. If you want to save power by coordinating access to the disk, that coordination has to happen at the application level. - Ted ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-26 13:31 ` Ted Ts'o @ 2011-05-26 16:05 ` D. Jansen 2011-05-26 16:21 ` Ted Ts'o 0 siblings, 1 reply; 53+ messages in thread From: D. Jansen @ 2011-05-26 16:05 UTC (permalink / raw) To: Ted Ts'o, D. Jansen, Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart, jens.axboe Problem: any fsync call by any application spins up the hard disk any time even in laptop_mode On Thu, May 26, 2011 at 3:31 PM, Ted Ts'o <tytso@mit.edu> wrote: > On Thu, May 26, 2011 at 01:16:45PM +0200, D. Jansen wrote: >> exactly. so what now? > This needs to be solved at the application level. So what we need is > a coordinating daemon which sends a message to all applications > saying, "ok, here's an opportunity to flush data to the disk; if you > have dirty files, or databases that has data that needs to be flushed, > send it now", and then a little while later, it sends a message to all > applications, "ok, we're going to let the disk spin down now, hold > your horses and don't send any data to the disk". I'm not sure this approach is straight forward. Because though there is no possibility to destroy data that is on disk due to non FIFO flushing of application writes queued in the kernel, which seems to be the main kernel level problem, yet new problems come up. Now there is 1) special support needed on the application side. 2) need for new out-of-kernel buffers. 3) need for inter-application write alignment nightmares. This sort of structure could cause very uncomfortable bugs that prevent writes from happening at all in cases that were not foreseen at all. 4) need for resources wasted through yet another daemon. 5) If the _application_, but not the kernel crashes, the data is safe. In my experience this is the much more likely case than that the mail server on my netbook optimized for battery time receives an email in laptop mode, sends the other server "200" and then before the next commit window my battery slips out and it's all gone. 6) _If_ there would be a special daemon I think it would make sense for that daemon to provide all the infrastructure, buffering and manage the fsync calls itself. This shouldn't and wouldn't be replicated by every single application. I think the alternative of ensuring the application writes are committed in order would make more sense: e..g a _user space library_ disables fsync etc. in laptop_mode if the user chooses to do so and kernel support for forced FIFO ordering or writes. This would fix 1) 2) 3) 4) 5) 6). Cheers, Dennis ps. So you've re-thought this "All that is necessary is a kernel patch to allow laptop_mode to disable fsync() calls(...)" (http://tytso.livejournal.com/2009/03/15/). That post had inspired my patch. pps. I know that an email getting lost is everyone's nightmare. But these things do happen, anyway. It's not the end of the world if one email is lost. (There's still a copy in the sent box. People will probably notice, eventually. I recently found out an email properly sent and filed in my gmail sent box never arrived for several of the people without any error reports. If people need to be 100% sure an email arrived they should ask for a confirmation and/or use a different medium.) This just makes it much more likely. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-26 16:05 ` D. Jansen @ 2011-05-26 16:21 ` Ted Ts'o 2011-05-27 7:12 ` D. Jansen 0 siblings, 1 reply; 53+ messages in thread From: Ted Ts'o @ 2011-05-26 16:21 UTC (permalink / raw) To: D. Jansen Cc: Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart, jens.axboe On Thu, May 26, 2011 at 06:05:43PM +0200, D. Jansen wrote: > Problem: any fsync call by any application spins up the hard disk any > time even in laptop_mode What you call a problem, I call a feature. If an application doesn't participate in the write aggregation protocol, the worst that happens is that you waste battery power. This I consider as a lesser evil than data loss. Similarly, if an application really _needs_ to write disk, and it can't contact the coordinating daemon, or the coordinating daemon doesn't respond in a reasonable amount of time, the application should feel free to write the data to disk and fsync(). This might waste a bit of power, but power is cheaper than lost data. > Because though there is no possibility to destroy data that is on disk > due to non FIFO flushing of application writes queued in the kernel, > which seems to be the main kernel level problem, yet new problems come > up. I'm not sure what you're talking about here. Buffered data can always be reordered in terms of when it is written to disk. This is considered good, and normal. If you want to guarantee that application writes are pushed out to disk, then either (a) use O_DIRECT, or (b) use fsync(). Those are your two options. If we didn't (for example) reorder writes to avoid the hard disk head from seeking all over the disk, that would actually cause more power to be consumed! > Now there is > 1) special support needed on the application side. Yep, because this is fundamentally an application-level problem, and the kernel doesn't have enough semantic information to solve the database coherency problem. > 2) need for new out-of-kernel buffers. Yes. So? > 3) need for inter-application write alignment nightmares. This sort of > structure could cause very uncomfortable bugs that prevent writes from > happening at all in cases that were not foreseen at all. Huh? I think you are talking about order that buffered writes happen, and there's no problem here. It's a feature that they can be reordered. See above. > 4) need for resources wasted through yet another daemon. A daemon doesn't have to take up much space. If it is linked with all of the GNOME libraries in the world, yeah, there'll be a problem, but there's no reason that this daemon should take more than, say a few tens of kilobytes at most. > 5) If the _application_, but not the kernel crashes, the data is safe. > In my experience this is the much more likely case than that the mail > server on my netbook optimized for battery time receives an email in > laptop mode, sends the other server "200" and then before the next > commit window my battery slips out and it's all gone. Huh? What's the problem that you're worried about here. > I think the alternative of ensuring the application writes are > committed in order would make more sense: > e..g a _user space library_ disables fsync etc. in laptop_mode if the > user chooses to do so and kernel support for forced FIFO ordering or > writes. > This would fix 1) 2) 3) 4) 5) 6). And if you do this to a mysql daemon, or to a firefox or chrome process which uses sqllite, and you crash at a wrong time, the entire database could be scrambled. You can't fix this with your solution, because you want to make fsync() lie to the database code. And so all of the extra work (and power) consumed by the database code to try to make its database writes be safe, will be compromised by making fsync() unreliable. > So you've re-thought this "All that is necessary is a kernel patch to > allow laptop_mode to disable fsync() calls(...)" > (http://tytso.livejournal.com/2009/03/15/). That post had inspired my > patch. I was thinking about things only from a file system perspective. The problem is that more and more people are running databases or other binary files which are updated in place on their laptops, and from a more holistic perspective, we have to worry about making sure that application-level databases are coherent in the face of a system crash. (For example, you drop your mobile phone, or your tablet, or your laptop, and the battery slips out.) - Ted ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-26 16:21 ` Ted Ts'o @ 2011-05-27 7:12 ` D. Jansen 2011-05-27 14:17 ` Theodore Tso 0 siblings, 1 reply; 53+ messages in thread From: D. Jansen @ 2011-05-27 7:12 UTC (permalink / raw) To: Ted Ts'o, D. Jansen, Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart On Thu, May 26, 2011 at 6:21 PM, Ted Ts'o <tytso@mit.edu> wrote: > On Thu, May 26, 2011 at 06:05:43PM +0200, D. Jansen wrote: >> Problem: any fsync call by any application spins up the hard disk any >> time even in laptop_mode > > What you call a problem, I call a feature. Problem: any fsync call by any application spins up the hard disk any time even in laptop_mode and there's nothing the user can do about it in user space - without risking that the application corrupts existing data if the kernel decides to commit the queued writes in non-FIFO order OR modifying every single application itself. >> Because though there is no possibility to destroy data that is on disk >> due to non FIFO flushing of application writes queued in the kernel, >> which seems to be the main kernel level problem, yet new problems come >> up. > > I'm not sure what you're talking about here. Buffered data can always > be reordered in terms of when it is written to disk. This is > considered good, and normal. If you want to guarantee that > application writes are pushed out to disk, then either (a) use > O_DIRECT, or (b) use fsync(). Those are your two options. That reordering is exactly what I'm talking about. It wasn't my idea. But if I understood it correctly, it's possible that the kernel commits writes of an application, _to one and the same file_, in a non-FIFO order, if the application does not fsync. And this _afaiu_ could result in the loss not only of new data, but complete corruption of previously existing data in laptop mode without fsync. But you're the expert. Is that really the case? If so, could it be avoided without the daemon and application patching? > If we didn't (for example) reorder writes to avoid the hard disk head > from seeking all over the disk, that would actually cause more power > to be consumed! Yes, probably. But I doubt if that happens only once in a commit window in laptop mode that the effect would destroy the gains. Also it is not always necessary. Only writes to one file should be committed in order. They could even be merged to one write - if they aren't already: It seems the ordering is only necessary when an fsync occurs. 1) DDD_ (write D at 0) 2) _HHH (write H at 1) (fsync) 3) DHHH (result/merged write, in order) As long as we don't end up with: 3) DDDH (out of order write, corrupt) >> Now there is (in a special write queue and coordination daemon) >> 1) special support needed on the application side. > > Yep, because this is fundamentally an application-level problem, and > the kernel doesn't have enough semantic information to solve the > database coherency problem. Well if we know that fsyncs mean the application needs the data to be committed in order, couldn't we watch out for fsync calls and then (in laptop mode when this feature specially requested by the user) switch that application to fifo per file writes? (Disregarding the write performance in that case.) Or we let the userspace eatmydata library detect the same fsync and use a kernel api to switch that write to fifo instead of fsyncing. A fifo write call might actually be useful to other applications and scenarios as well. (trojan horse!) Or the last write before the fsync is committed last. If reordering is otherwise possible, this should avoid corruption and decrease performance less. (Though we're not talking about writing hundreds of MBs in laptop mode in my average use case scenario of office applications and maybe a browser running.) > >> 2) need for new out-of-kernel buffers. > > Yes. So? Shouldn't we try to avoid replicating existing infrastructure when possible? > >> 3) need for inter-application write alignment nightmares. This sort of >> structure could cause very uncomfortable bugs that prevent writes from >> happening at all in cases that were not foreseen at all. > > Huh? I think you are talking about order that buffered writes happen, > and there's no problem here. It's a feature that they can be > reordered. See above. No, what I meant is that if there is a bug at any step of the coordination between the applications and the daemon: in the daemon, the software, their communication connection, etc., writes may not occur and we may lose data without need. >> 5) If the _application_, but not the kernel crashes, the data is safe. >> In my experience this is the much more likely case than that the mail >> server on my netbook optimized for battery time receives an email in >> laptop mode, sends the other server "200" and then before the next >> commit window my battery slips out and it's all gone. > > Huh? What's the problem that you're worried about here. Your scenario sounds like this: daemon announced when to flush data until then application buffers data in it's user space. This means if you save a file and the application crashes, e.g. segfaults and is killed, the data is still in its queue and thus lost. Without the daemon, the data would be in kernel space already and thus safe from application crashes. In my experience the kernel is very stable, applications are much less so. And I really don't see this entering many applications. They would probably say this is the task of the kernel itself or some other piece of layer in between, but not the task of every single app developer to reinvent write caching, coordination with the laptop writes daemon etc. In the end we might have one or two special "write in laptop mode" apps and as soon as I start a browser or any sqlite based app, the problem is back. >> I think the alternative of ensuring the application writes are >> committed in order would make more sense: >> e..g a _user space library_ disables fsync etc. in laptop_mode if the >> user chooses to do so and kernel support for forced FIFO ordering or >> writes. >> This would fix 1) 2) 3) 4) 5) 6). > > And if you do this to a mysql daemon, or to a firefox or chrome > process which uses sqllite, and you crash at a wrong time, the entire > database could be scrambled. Define crash at the wrong time. Because there is always a wrong time, whether with laptop mode or without, with fsync or without. > You can't fix this with your solution, because you want to make fsync() > lie to the database code. And so all > of the extra work (and power) consumed by the database code to try to > make its database writes be safe, will be compromised by making > fsync() unreliable. Yes, I would like to have the liberty of extending the decrease of safety of new data in favor of the choice of creating more new data (due to longer run time) when in laptop mode. I still want and use that safety, just not when I'm in laptop mode. > >> So you've re-thought this "All that is necessary is a kernel patch to >> allow laptop_mode to disable fsync() calls(...)" >> (http://tytso.livejournal.com/2009/03/15/). That post had inspired my >> patch. > > I was thinking about things only from a file system perspective. The > problem is that more and more people are running databases or other > binary files which are updated in place on their laptops, and from a > more holistic perspective, we have to worry about making sure that > application-level databases are coherent in the face of a system > crash. (For example, you drop your mobile phone, or your tablet, or > your laptop, and the battery slips out.) Exactly. Great example! Again, I very much agree.("Even") I don't want to end up with corrupt data. But I accept old data. Is there really no way to get there without rewriting each and every application's fsync code? Thanks for your insights! ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-27 7:12 ` D. Jansen @ 2011-05-27 14:17 ` Theodore Tso 2011-05-27 17:51 ` david 2011-05-29 10:45 ` D. Jansen 0 siblings, 2 replies; 53+ messages in thread From: Theodore Tso @ 2011-05-27 14:17 UTC (permalink / raw) To: D. Jansen; +Cc: Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart On May 27, 2011, at 3:12 AM, D. Jansen wrote: > > That reordering is exactly what I'm talking about. It wasn't my idea. > But if I understood it correctly, it's possible that the kernel > commits writes of an application, _to one and the same file_, in a > non-FIFO order, if the application does not fsync. And this _afaiu_ > could result in the loss not only of new data, but complete corruption > of previously existing data in laptop mode without fsync. No, you're not understanding the problem. All layers of the storage stack -- including the hard drive -- is allowed to reorder writes. So even if the kernel sends data to the disk in the exact same order that the application wrote it, it could still get written in a different order, because the hard drive itself can reorder writes. This is necessary for performance; if you didn't have this, the storage stack would be dog slow, and would consume even more power. So at least level, the only thing you can count upon is that if you want to make sure everything is flushed to stable store, you need to send an fsync() command at the application to file system level, or a barrier or flush command at the OS to hard drive level. So what databases do is the first write the changes they intend to make to an intent log. Then they send an fsync() command; then they write a commit block to the intent log; then they send another fsync() command; and only then now that the transaction has been committed to the commit log, do they start updating the table files. (This is a highly simplified model, but it's good enough for this discussion.) Ordering doesn't matter, because nothing, including the hard drive, guarantees ordering. What does matter is that the fsync() commands act like barriers; writes before the fsync() command are guaranteed to be written to the disk, and survive a reboot, before any writes after the fsync() are processed. See? This is why getting fsync() right is so critical; things are defined to work this way, and programs like mysql and sqllite depend on things working this way. You are proposing to break this. > (Though we're not talking about writing hundreds of > MBs in laptop mode in my average use case scenario of office > applications and maybe a browser running.) Firefox, in order to make their "awesome bar" work, is responsible for 300+ MB's worth of writes per click; so for every three clicks, you've written a gigabyte. Any other questions? > > No, what I meant is that if there is a bug at any step of the > coordination between the applications and the daemon: in the daemon, > the software, their communication connection, etc., writes may not > occur and we may lose data without need. But the application will know that, and at the end of the day, if the coordination is wrong, the application can always ignore the daemon, write the data and call fsync(). So if there is any failure, it fails safe; worst case you just waste more battery. > Your scenario sounds like this: > daemon announced when to flush data > until then application buffers data in it's user space. > > This means if you save a file and the application crashes, e.g. segfaults > and is killed, the data is still in its queue and thus lost. If the application crashes, it will always lose data. If the application thinks its flaky, it can always ignore protocol and force a disk write; as I said, that will just burn battery, which is preferable to losing data. > > Exactly. Great example! Again, I very much agree.("Even") I don't want > to end up with > corrupt data. But I accept old data. Is there really no way to get there without > rewriting each and every application's fsync code? If the application is using a binary database file format, then no, if you subvert fsync(), you can risk losing the entire database. But even if you use 100's of flat files, if you care about the relationship between the flat files as having critical meaning, then you can end up corrupting data even if you use lots of flat files. If you are willing to rewrite the *entire* database to a completely new file each time you want to write out some data, and only delete the old database once the new database has been written out, then you're fine. If the file is too big you can delay the time period between a complete writeout of the database. But then if you drop your laptop and the battery slips out, you'll lose more data. Life is full of tradeoffs. If the only editor you use is vi, and the only web browser you use is lynx, then life is much simpler. If you want more complexity, AND you want more safety, then you'll have to pay for that in terms of more battery usage. -- Ted > > Thanks for your insights! ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-27 14:17 ` Theodore Tso @ 2011-05-27 17:51 ` david 2011-05-29 10:45 ` D. Jansen 1 sibling, 0 replies; 53+ messages in thread From: david @ 2011-05-27 17:51 UTC (permalink / raw) To: Theodore Tso Cc: D. Jansen, Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart On Fri, 27 May 2011, Theodore Tso wrote: > If you are willing to rewrite the *entire* database to a completely new file each > time you want to write out some data, and only delete the old database > once the new database has been written out, then you're fine. If the file is too > big you can delay the time period between a complete writeout of the > database. But then if you drop your laptop and the battery slips out, you'll > lose more data. Life is full of tradeoffs. actually, even in this case don't you need a barrier or fsync to make sure the new file is completely written before the old file goes away? David Lang ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-27 14:17 ` Theodore Tso 2011-05-27 17:51 ` david @ 2011-05-29 10:45 ` D. Jansen 2011-05-30 1:53 ` david 1 sibling, 1 reply; 53+ messages in thread From: D. Jansen @ 2011-05-29 10:45 UTC (permalink / raw) To: Theodore Tso; +Cc: Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart On Fri, May 27, 2011 at 4:17 PM, Theodore Tso <tytso@mit.edu> wrote: > On May 27, 2011, at 3:12 AM, D. Jansen wrote: >> That reordering is exactly what I'm talking about. It wasn't my idea. >> But if I understood it correctly, it's possible that the kernel >> commits writes of an application, _to one and the same file_, in a >> non-FIFO order, if the application does not fsync. And this _afaiu_ >> could result in the loss not only of new data, but complete corruption >> of previously existing data in laptop mode without fsync. > > No, you're not understanding the problem. All layers of the storage > stack -- including the hard drive -- is allowed to reorder writes. So > even if the kernel sends data to the disk in the exact same order that > the application wrote it, it could still get written in a different order, > because the hard drive itself can reorder writes. This is necessary > for performance; if you didn't have this, the storage stack would be > dog slow, and would consume even more power. > > So at least level, the only thing you can count upon is that if you want > to make sure everything is flushed to stable store, you need to send > an fsync() command at the application to file system level, or a barrier > or flush command at the OS to hard drive level. (...) > Ordering doesn't matter, because nothing, including the hard drive, > guarantees ordering. What does matter is that the fsync() commands > act like barriers; writes before the fsync() command are guaranteed > to be written to the disk, and survive a reboot, before any writes after > the fsync() are processed. See? Ok, thanks a lot! I understand a lot better now! So we can't live without the fsyncs. So what if we would queue the fsyncs along with the writes - we would just fsync later instead of immediately, in between the writes as they came in. Then by design previous data could not be corrupted, right? We would do exactly the same thing, just later. It'd be kind of a disk write time distortion field. Thanks again for your feedback! ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-29 10:45 ` D. Jansen @ 2011-05-30 1:53 ` david 2011-05-30 7:13 ` Oliver Neukum 2011-05-30 13:55 ` D. Jansen 0 siblings, 2 replies; 53+ messages in thread From: david @ 2011-05-30 1:53 UTC (permalink / raw) To: D. Jansen Cc: Theodore Tso, Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart [-- Attachment #1: Type: TEXT/PLAIN, Size: 2640 bytes --] On Sun, 29 May 2011, D. Jansen wrote: > On Fri, May 27, 2011 at 4:17 PM, Theodore Tso <tytso@mit.edu> wrote: >> On May 27, 2011, at 3:12 AM, D. Jansen wrote: >>> That reordering is exactly what I'm talking about. It wasn't my idea. >>> But if I understood it correctly, it's possible that the kernel >>> commits writes of an application, _to one and the same file_, in a >>> non-FIFO order, if the application does not fsync. And this _afaiu_ >>> could result in the loss not only of new data, but complete corruption >>> of previously existing data in laptop mode without fsync. >> >> No, you're not understanding the problem. All layers of the storage >> stack -- including the hard drive -- is allowed to reorder writes. So >> even if the kernel sends data to the disk in the exact same order that >> the application wrote it, it could still get written in a different order, >> because the hard drive itself can reorder writes. This is necessary >> for performance; if you didn't have this, the storage stack would be >> dog slow, and would consume even more power. >> >> So at least level, the only thing you can count upon is that if you want >> to make sure everything is flushed to stable store, you need to send >> an fsync() command at the application to file system level, or a barrier >> or flush command at the OS to hard drive level. > (...) >> Ordering doesn't matter, because nothing, including the hard drive, >> guarantees ordering. What does matter is that the fsync() commands >> act like barriers; writes before the fsync() command are guaranteed >> to be written to the disk, and survive a reboot, before any writes after >> the fsync() are processed. See? > > Ok, thanks a lot! I understand a lot better now! > So we can't live without the fsyncs. > > So what if we would queue the fsyncs along with the writes - we would > just fsync later instead of immediately, in between the writes as they > came in. Then by design previous data could not be corrupted, right? > We would do exactly the same thing, just later. > It'd be kind of a disk write time distortion field. the problem is that the spec for fsync says that your program stops until fsync finishes. If you don't do that then you will corrupt and loose data. so if you delay fsync you will have your application (or desktop manager) freeze until the fsync completes. if what you are wanting is the ability to say 'these things must be written before these other things to keep them from being corrupted, but I don't care when they get written (or if they get lost in a crash)' then what you want isn't fsync, it's a barrier. David Lang ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-30 1:53 ` david @ 2011-05-30 7:13 ` Oliver Neukum 2011-05-30 12:55 ` Valdis.Kletnieks 2011-05-30 13:55 ` D. Jansen 1 sibling, 1 reply; 53+ messages in thread From: Oliver Neukum @ 2011-05-30 7:13 UTC (permalink / raw) To: david Cc: D. Jansen, Theodore Tso, akpm, linux-kernel, Dave Chinner, njs, bart Am Montag, 30. Mai 2011, 03:53:18 schrieb david@lang.hm: > > So what if we would queue the fsyncs along with the writes - we would > > just fsync later instead of immediately, in between the writes as they > > came in. Then by design previous data could not be corrupted, right? > > We would do exactly the same thing, just later. > > It'd be kind of a disk write time distortion field. > > the problem is that the spec for fsync says that your program stops until > fsync finishes. If you don't do that then you will corrupt and loose data. It is important to be precise. You will loose data, but you will not get corruption. > so if you delay fsync you will have your application (or desktop manager) > freeze until the fsync completes. > > if what you are wanting is the ability to say 'these things must be > written before these other things to keep them from being corrupted, but I > don't care when they get written (or if they get lost in a crash)' then > what you want isn't fsync, it's a barrier. Yes, but where is the problem? Regards Oliver -- - - - SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany - - - ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-30 7:13 ` Oliver Neukum @ 2011-05-30 12:55 ` Valdis.Kletnieks 2011-05-30 18:03 ` david 0 siblings, 1 reply; 53+ messages in thread From: Valdis.Kletnieks @ 2011-05-30 12:55 UTC (permalink / raw) To: Oliver Neukum Cc: david, D. Jansen, Theodore Tso, akpm, linux-kernel, Dave Chinner, njs, bart [-- Attachment #1: Type: text/plain, Size: 531 bytes --] On Mon, 30 May 2011 09:13:27 +0200, Oliver Neukum said: > Am Montag, 30. Mai 2011, 03:53:18 schrieb david@lang.hm: > > if what you are wanting is the ability to say 'these things must be > > written before these other things to keep them from being corrupted, but I > > don't care when they get written (or if they get lost in a crash)' then > > what you want isn't fsync, it's a barrier. > > Yes, but where is the problem? The problem is that *from userspace*, the closest approximation to a barrier is fsync(). [-- Attachment #2: Type: application/pgp-signature, Size: 227 bytes --] ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-30 12:55 ` Valdis.Kletnieks @ 2011-05-30 18:03 ` david 0 siblings, 0 replies; 53+ messages in thread From: david @ 2011-05-30 18:03 UTC (permalink / raw) To: Valdis.Kletnieks Cc: Oliver Neukum, D. Jansen, Theodore Tso, akpm, linux-kernel, Dave Chinner, njs, bart On Mon, 30 May 2011, Valdis.Kletnieks@vt.edu wrote: > On Mon, 30 May 2011 09:13:27 +0200, Oliver Neukum said: >> Am Montag, 30. Mai 2011, 03:53:18 schrieb david@lang.hm: > >>> if what you are wanting is the ability to say 'these things must be >>> written before these other things to keep them from being corrupted, but I >>> don't care when they get written (or if they get lost in a crash)' then >>> what you want isn't fsync, it's a barrier. >> >> Yes, but where is the problem? > > The problem is that *from userspace*, the closest approximation to a barrier > is fsync(). good point, there really should be some way of doing a barrier from userspace. David Lang ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-30 1:53 ` david 2011-05-30 7:13 ` Oliver Neukum @ 2011-05-30 13:55 ` D. Jansen 2011-05-30 18:02 ` david 1 sibling, 1 reply; 53+ messages in thread From: D. Jansen @ 2011-05-30 13:55 UTC (permalink / raw) To: david, Theodore Tso Cc: Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart On Mon, May 30, 2011 at 3:53 AM, <david@lang.hm> wrote: > On Sun, 29 May 2011, D. Jansen wrote: >> On Fri, May 27, 2011 at 4:17 PM, Theodore Tso <tytso@mit.edu> wrote: >>> On May 27, 2011, at 3:12 AM, D. Jansen wrote: >>>> That reordering is exactly what I'm talking about. It wasn't my idea. >>>> But if I understood it correctly, it's possible that the kernel >>>> commits writes of an application, _to one and the same file_, in a >>>> non-FIFO order, if the application does not fsync. And this _afaiu_ >>>> could result in the loss not only of new data, but complete corruption >>>> of previously existing data in laptop mode without fsync. >>> >>> No, you're not understanding the problem. All layers of the storage >>> stack -- including the hard drive -- is allowed to reorder writes. So >>> even if the kernel sends data to the disk in the exact same order that >>> the application wrote it, it could still get written in a different >>> order, >>> because the hard drive itself can reorder writes. This is necessary >>> for performance; if you didn't have this, the storage stack would be >>> dog slow, and would consume even more power. >>> >>> So at least level, the only thing you can count upon is that if you want >>> to make sure everything is flushed to stable store, you need to send >>> an fsync() command at the application to file system level, or a barrier >>> or flush command at the OS to hard drive level. >> (...) >>> Ordering doesn't matter, because nothing, including the hard drive, >>> guarantees ordering. What does matter is that the fsync() commands >>> act like barriers; writes before the fsync() command are guaranteed >>> to be written to the disk, and survive a reboot, before any writes after >>> the fsync() are processed. See? >> >> Ok, thanks a lot! I understand a lot better now! >> So we can't live without the fsyncs. >> >> So what if we would queue the fsyncs along with the writes - we would >> just fsync later instead of immediately, in between the writes as they >> came in. Then by design previous data could not be corrupted, right? >> We would do exactly the same thing, just later. >> It'd be kind of a disk write time distortion field. > > the problem is that the spec for fsync says that your program stops until > fsync finishes. If you don't do that then you will corrupt and loose data. > > so if you delay fsync you will have your application (or desktop manager) > freeze until the fsync completes. So that would not be an option. Freezing until the end of the write window is not what we want. Neither is ignoring the fsync because that could corrupt data, esp. in databases like sqlite. > > if what you are wanting is the ability to say 'these things must be written > before these other things to keep them from being corrupted, but I don't > care when they get written (or if they get lost in a crash)' then what you > want isn't fsync, it's a barrier. That sounds great! So an fsync call in laptop mode could be interpreted as a barrier and we would be reasonably save from corrupting old existing data? ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-30 13:55 ` D. Jansen @ 2011-05-30 18:02 ` david 2011-05-30 18:28 ` D. Jansen 0 siblings, 1 reply; 53+ messages in thread From: david @ 2011-05-30 18:02 UTC (permalink / raw) To: D. Jansen Cc: Theodore Tso, Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart [-- Attachment #1: Type: TEXT/PLAIN, Size: 3440 bytes --] On Mon, 30 May 2011, D. Jansen wrote: > On Mon, May 30, 2011 at 3:53 AM, <david@lang.hm> wrote: >> On Sun, 29 May 2011, D. Jansen wrote: >>> On Fri, May 27, 2011 at 4:17 PM, Theodore Tso <tytso@mit.edu> wrote: >>>> On May 27, 2011, at 3:12 AM, D. Jansen wrote: >>>>> That reordering is exactly what I'm talking about. It wasn't my idea. >>>>> But if I understood it correctly, it's possible that the kernel >>>>> commits writes of an application, _to one and the same file_, in a >>>>> non-FIFO order, if the application does not fsync. And this _afaiu_ >>>>> could result in the loss not only of new data, but complete corruption >>>>> of previously existing data in laptop mode without fsync. >>>> >>>> No, you're not understanding the problem. All layers of the storage >>>> stack -- including the hard drive -- is allowed to reorder writes. So >>>> even if the kernel sends data to the disk in the exact same order that >>>> the application wrote it, it could still get written in a different >>>> order, >>>> because the hard drive itself can reorder writes. This is necessary >>>> for performance; if you didn't have this, the storage stack would be >>>> dog slow, and would consume even more power. >>>> >>>> So at least level, the only thing you can count upon is that if you want >>>> to make sure everything is flushed to stable store, you need to send >>>> an fsync() command at the application to file system level, or a barrier >>>> or flush command at the OS to hard drive level. >>> (...) >>>> Ordering doesn't matter, because nothing, including the hard drive, >>>> guarantees ordering. What does matter is that the fsync() commands >>>> act like barriers; writes before the fsync() command are guaranteed >>>> to be written to the disk, and survive a reboot, before any writes after >>>> the fsync() are processed. See? >>> >>> Ok, thanks a lot! I understand a lot better now! >>> So we can't live without the fsyncs. >>> >>> So what if we would queue the fsyncs along with the writes - we would >>> just fsync later instead of immediately, in between the writes as they >>> came in. Then by design previous data could not be corrupted, right? >>> We would do exactly the same thing, just later. >>> It'd be kind of a disk write time distortion field. >> >> the problem is that the spec for fsync says that your program stops until >> fsync finishes. If you don't do that then you will corrupt and loose data. >> >> so if you delay fsync you will have your application (or desktop manager) >> freeze until the fsync completes. > > So that would not be an option. Freezing until the end of the write > window is not what we want. > Neither is ignoring the fsync because that could corrupt data, esp. in > databases like sqlite. >> >> if what you are wanting is the ability to say 'these things must be written >> before these other things to keep them from being corrupted, but I don't >> care when they get written (or if they get lost in a crash)' then what you >> want isn't fsync, it's a barrier. > > That sounds great! > So an fsync call in laptop mode could be interpreted as a barrier > and we would be reasonably save from corrupting old existing data? no, you cannot just change a fsync to a barrier, in some cases the data absolutly needs to be saved, not just ordered (remember the example of a mail server telling the other system that the data can be deleted after a fsync returns) David Lang ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-30 18:02 ` david @ 2011-05-30 18:28 ` D. Jansen 2011-05-30 18:43 ` Valdis.Kletnieks 2011-05-30 18:45 ` david 0 siblings, 2 replies; 53+ messages in thread From: D. Jansen @ 2011-05-30 18:28 UTC (permalink / raw) To: david Cc: Theodore Tso, Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart On Mon, May 30, 2011 at 8:02 PM, <david@lang.hm> wrote: > On Mon, 30 May 2011, D. Jansen wrote: >> On Mon, May 30, 2011 at 3:53 AM, <david@lang.hm> wrote: >>> On Sun, 29 May 2011, D. Jansen wrote: >>>> On Fri, May 27, 2011 at 4:17 PM, Theodore Tso <tytso@mit.edu> wrote: >>>>> On May 27, 2011, at 3:12 AM, D. Jansen wrote: >>>>>> That reordering is exactly what I'm talking about. It wasn't my idea. >>>>>> But if I understood it correctly, it's possible that the kernel >>>>>> commits writes of an application, _to one and the same file_, in a >>>>>> non-FIFO order, if the application does not fsync. And this _afaiu_ >>>>>> could result in the loss not only of new data, but complete corruption >>>>>> of previously existing data in laptop mode without fsync. >>>>> >>>>> No, you're not understanding the problem. All layers of the storage >>>>> stack -- including the hard drive -- is allowed to reorder writes. So >>>>> even if the kernel sends data to the disk in the exact same order that >>>>> the application wrote it, it could still get written in a different >>>>> order, >>>>> because the hard drive itself can reorder writes. This is necessary >>>>> for performance; if you didn't have this, the storage stack would be >>>>> dog slow, and would consume even more power. >>>>> >>>>> So at least level, the only thing you can count upon is that if you >>>>> want >>>>> to make sure everything is flushed to stable store, you need to send >>>>> an fsync() command at the application to file system level, or a >>>>> barrier >>>>> or flush command at the OS to hard drive level. >>>> >>>> (...) >>>>> >>>>> Ordering doesn't matter, because nothing, including the hard drive, >>>>> guarantees ordering. What does matter is that the fsync() commands >>>>> act like barriers; writes before the fsync() command are guaranteed >>>>> to be written to the disk, and survive a reboot, before any writes >>>>> after >>>>> the fsync() are processed. See? >>>> >>>> Ok, thanks a lot! I understand a lot better now! >>>> So we can't live without the fsyncs. >>>> >>>> So what if we would queue the fsyncs along with the writes - we would >>>> just fsync later instead of immediately, in between the writes as they >>>> came in. Then by design previous data could not be corrupted, right? >>>> We would do exactly the same thing, just later. >>>> It'd be kind of a disk write time distortion field. >>> >>> the problem is that the spec for fsync says that your program stops until >>> fsync finishes. If you don't do that then you will corrupt and loose >>> data. >>> >>> so if you delay fsync you will have your application (or desktop manager) >>> freeze until the fsync completes. >> >> So that would not be an option. Freezing until the end of the write >> window is not what we want. >> Neither is ignoring the fsync because that could corrupt data, esp. in >> databases like sqlite. >>> >>> if what you are wanting is the ability to say 'these things must be >>> written >>> before these other things to keep them from being corrupted, but I don't >>> care when they get written (or if they get lost in a crash)' then what >>> you >>> want isn't fsync, it's a barrier. >> >> That sounds great! >> So an fsync call in laptop mode could be interpreted as a barrier >> and we would be reasonably save from corrupting old existing data? > > no, you cannot just change a fsync to a barrier, in some cases the data > absolutly needs to be saved, not just ordered (remember the example of a > mail server telling the other system that the data can be deleted after a > fsync returns) I'm not really sure I why shouldn't have that choice as a user. Just because someone else could be running a mailserver on his system and configure it in a way that it doesn't behave as it should? If he really wants to do that there's really nothing we can do to stop him. I'm sure there are other ways existing kernel options can be used to make software behave different than it should. Are we going to remove them all now? The big problem is that so far only fsync existed and lots of software seemingly abuses it as an expensive write barrier. And it would really be lovely to have the choice to stop that on an opt-in basis in laptop mode. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-30 18:28 ` D. Jansen @ 2011-05-30 18:43 ` Valdis.Kletnieks 2011-05-30 19:54 ` D. Jansen 2011-05-30 18:45 ` david 1 sibling, 1 reply; 53+ messages in thread From: Valdis.Kletnieks @ 2011-05-30 18:43 UTC (permalink / raw) To: D. Jansen Cc: david, Theodore Tso, Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart [-- Attachment #1: Type: text/plain, Size: 1133 bytes --] On Mon, 30 May 2011 20:28:24 +0200, "D. Jansen" said: > I'm not really sure I why shouldn't have that choice as a user. Just > because someone else could be running a mailserver on his system and > configure it in a way that it doesn't behave as it should? It cuts both ways. If a piece of software really wants to be sure that the fsync() semantics it's expecting are actually adhered to and refuses to run otherwise, how does it tell that you're lying to it? > The big problem is that so far only fsync existed and lots of software > seemingly abuses it as an expensive write barrier. And it would really > be lovely to have the choice to stop that on an opt-in basis in laptop > mode. It's not "seemingly abuses it". That's the existing userspace API for inserting a barrier. The problem is that as defined, it will wait for the writeback to actually finish - which is actually as good as you can get without getting into the async I/O support. If there was a "insert barrier and return" API, how would it report back that the barrier had failed with EIO after it had already returned to userspace? [-- Attachment #2: Type: application/pgp-signature, Size: 227 bytes --] ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-30 18:43 ` Valdis.Kletnieks @ 2011-05-30 19:54 ` D. Jansen 0 siblings, 0 replies; 53+ messages in thread From: D. Jansen @ 2011-05-30 19:54 UTC (permalink / raw) To: Valdis.Kletnieks Cc: david, Theodore Tso, Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart On Mon, May 30, 2011 at 8:43 PM, <Valdis.Kletnieks@vt.edu> wrote: > On Mon, 30 May 2011 20:28:24 +0200, "D. Jansen" said: > >> I'm not really sure I why shouldn't have that choice as a user. Just >> because someone else could be running a mailserver on his system and >> configure it in a way that it doesn't behave as it should? > > It cuts both ways. If a piece of software really wants to be sure that the > fsync() semantics it's expecting are actually adhered to and refuses to run > otherwise, how does it tell that you're lying to it? Well if we allow to use a library to wrap fsync into write barriers that doesn't seem to matter because software doesn't need to know what happens and isn't even supposed to be able to work around it. If this was to go into the kernel as an option, the situation was different. But really, if you look at this discussion and the feedback so far I don't think that's an option we need to discuss at this point... > >> The big problem is that so far only fsync existed and lots of software >> seemingly abuses it as an expensive write barrier. And it would really >> be lovely to have the choice to stop that on an opt-in basis in laptop >> mode. > > It's not "seemingly abuses it". That's the existing userspace API for > inserting a barrier. The problem is that as defined, it will wait for the > writeback to actually finish - which is actually as good as you can get without > getting into the async I/O support. If there was a "insert barrier and return" > API, how would it report back that the barrier had failed with EIO after it had > already returned to userspace? I guess with another call some time later, to check for the success of the previous call? ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-30 18:28 ` D. Jansen 2011-05-30 18:43 ` Valdis.Kletnieks @ 2011-05-30 18:45 ` david 2011-05-30 19:49 ` D. Jansen 2011-05-31 6:48 ` Oliver Neukum 1 sibling, 2 replies; 53+ messages in thread From: david @ 2011-05-30 18:45 UTC (permalink / raw) To: D. Jansen Cc: Theodore Tso, Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart On Mon, 30 May 2011, D. Jansen wrote: > On Mon, May 30, 2011 at 8:02 PM, <david@lang.hm> wrote: >> >> no, you cannot just change a fsync to a barrier, in some cases the data >> absolutly needs to be saved, not just ordered (remember the example of a >> mail server telling the other system that the data can be deleted after a >> fsync returns) > > I'm not really sure I why shouldn't have that choice as a user. Just > because someone else could be running a mailserver on his system and > configure it in a way that it doesn't behave as it should? > If he really wants to do that there's really nothing we can do to stop > him. I'm sure there are other ways existing kernel options can be used > to make software behave different than it should. Are we going to > remove them all now? > > The big problem is that so far only fsync existed and lots of software > seemingly abuses it as an expensive write barrier. And it would really > be lovely to have the choice to stop that on an opt-in basis in laptop > mode. is the benifit of not spinning up the disk really worth the risk of loosing data? and should this really be a global across-the-board option? the problem is that most users don't know what their system is running, or what effect disaling fsync would have. those that do can probably use LD_PRELOAD to override fsync calls. it doesn't take running a mail server, even a mail client will have the same risk. If you use POP for mail (a very common option) then you download messages and tell the server to delete them. if you do not really save them (one fsync after they are all saved), then you can loose everything that you downloaded. David Lang ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-30 18:45 ` david @ 2011-05-30 19:49 ` D. Jansen 2011-05-30 20:53 ` david 2011-05-30 22:10 ` Jesper Juhl 2011-05-31 6:48 ` Oliver Neukum 1 sibling, 2 replies; 53+ messages in thread From: D. Jansen @ 2011-05-30 19:49 UTC (permalink / raw) To: david Cc: Theodore Tso, Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart On Mon, May 30, 2011 at 8:45 PM, <david@lang.hm> wrote: > On Mon, 30 May 2011, D. Jansen wrote: > >> On Mon, May 30, 2011 at 8:02 PM, <david@lang.hm> wrote: >>> >>> no, you cannot just change a fsync to a barrier, in some cases the data >>> absolutly needs to be saved, not just ordered (remember the example of a >>> mail server telling the other system that the data can be deleted after a >>> fsync returns) >> >> I'm not really sure I why shouldn't have that choice as a user. Just >> because someone else could be running a mailserver on his system and >> configure it in a way that it doesn't behave as it should? >> If he really wants to do that there's really nothing we can do to stop >> him. I'm sure there are other ways existing kernel options can be used >> to make software behave different than it should. Are we going to >> remove them all now? >> >> The big problem is that so far only fsync existed and lots of software >> seemingly abuses it as an expensive write barrier. And it would really >> be lovely to have the choice to stop that on an opt-in basis in laptop >> mode. > > is the benifit of not spinning up the disk really worth the risk of loosing > data? Is that a choice I can make myself? Is that a choice I make with laptop mode? Or is that a choice I may only make if I'm willing to modify and compile my own kernel? > > and should this really be a global across-the-board option? The point is option imo. > > the problem is that most users don't know what their system is running, or > what effect disaling fsync would have. those that do can probably use > LD_PRELOAD to override fsync calls. As we found out, they can't. But if we export barrier, I hope a library could wrap fsyncs into barriers. Is that the case? > > it doesn't take running a mail server, even a mail client will have the same > risk. If you use POP for mail (a very common option) then you download > messages and tell the server to delete them. if you do not really save them > (one fsync after they are all saved), then you can loose everything that you > downloaded. Yes, I know. It's the same argument again and again. I understand not everybody wants this. But some do. Some prefer working 10-20% longer on battery (certainty) instead of possibly losing 5 % data (possibility) or losing all your data (possibility if you use laptop mode and the hard disk wakes up again and again and eventually wears out). That's why there's laptop mode. And this would play into laptop mode and prevent the hard disk from breaking down prematurely and saving battery. Please try it out before you say that it's not necessary in your use case. It's useful and necessary in mine and other mobile use cases. Please don't assume that all distributions activate this by default and mailservers world wide use all their emails because they all crash with laptop mode active. I really don't care if it's in the kernel or elsewhere. But I care that old data is not corrupted. And it seems this is not (yet) possible with user space, e.g. a library in ld_preload. So I'm happy for any hints how to accomplish this. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-30 19:49 ` D. Jansen @ 2011-05-30 20:53 ` david 2011-05-30 21:24 ` D. Jansen 2011-05-30 22:10 ` Jesper Juhl 1 sibling, 1 reply; 53+ messages in thread From: david @ 2011-05-30 20:53 UTC (permalink / raw) To: D. Jansen Cc: Theodore Tso, Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart On Mon, 30 May 2011, D. Jansen wrote: > On Mon, May 30, 2011 at 8:45 PM, <david@lang.hm> wrote: >> >> >> the problem is that most users don't know what their system is running, or >> what effect disaling fsync would have. those that do can probably use >> LD_PRELOAD to override fsync calls. > > As we found out, they can't. But if we export barrier, I hope a > library could wrap fsyncs into barriers. Is that the case? a library can wrap fsync into anything. >> >> it doesn't take running a mail server, even a mail client will have the same >> risk. If you use POP for mail (a very common option) then you download >> messages and tell the server to delete them. if you do not really save them >> (one fsync after they are all saved), then you can loose everything that you >> downloaded. > > Yes, I know. It's the same argument again and again. I understand not > everybody wants this. But some do. Some prefer working 10-20% longer > on battery (certainty) instead of possibly losing 5 % data > (possibility) or losing all your data (possibility if you use laptop > mode and the hard disk wakes up again and again and eventually wears > out). those are some powerful numbers you are throwing around, can you back them up? do you really have so many fsync's going on that the disk spins up so much that you would gain 10-20% battery life? and what makes you think the extra spin-ups from fsyncs will cause your hard drive to fail significantly earlier? (if you have a hard drive with a limited number of spin-up cycles, you probably don't want to use laptop mode at all) why do you think it's a possibility of loosing only 5% of data? David Lang > That's why there's laptop mode. And this would play into laptop > mode and prevent the hard disk from breaking down prematurely and > saving battery. > > Please try it out before you say that it's not necessary in your use > case. It's useful and necessary in mine and other mobile use cases. > Please don't assume that all distributions activate this by default > and mailservers world wide use all their emails because they all crash > with laptop mode active. > > I really don't care if it's in the kernel or elsewhere. But I care > that old data is not corrupted. And it seems this is not (yet) > possible with user space, e.g. a library in ld_preload. So I'm happy > for any hints how to accomplish this. > ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-30 20:53 ` david @ 2011-05-30 21:24 ` D. Jansen 2011-05-30 22:41 ` david ` (2 more replies) 0 siblings, 3 replies; 53+ messages in thread From: D. Jansen @ 2011-05-30 21:24 UTC (permalink / raw) To: david Cc: Theodore Tso, Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart On Mon, May 30, 2011 at 10:53 PM, <david@lang.hm> wrote: > On Mon, 30 May 2011, D. Jansen wrote: > >> On Mon, May 30, 2011 at 8:45 PM, <david@lang.hm> wrote: >>> >>> >>> the problem is that most users don't know what their system is running, >>> or >>> what effect disaling fsync would have. those that do can probably use >>> LD_PRELOAD to override fsync calls. >> >> As we found out, they can't. But if we export barrier, I hope a >> library could wrap fsyncs into barriers. Is that the case? > > a library can wrap fsync into anything. > >>> >>> it doesn't take running a mail server, even a mail client will have the >>> same >>> risk. If you use POP for mail (a very common option) then you download >>> messages and tell the server to delete them. if you do not really save >>> them >>> (one fsync after they are all saved), then you can loose everything that >>> you >>> downloaded. >> >> Yes, I know. It's the same argument again and again. I understand not >> everybody wants this. But some do. Some prefer working 10-20% longer >> on battery (certainty) instead of possibly losing 5 % data >> (possibility) or losing all your data (possibility if you use laptop >> mode and the hard disk wakes up again and again and eventually wears >> out). > > those are some powerful numbers you are throwing around, can you back them > up? Yes. My netbook uses 4.8 W with hard disk off. 0.5 W more with hard disk on. A lot of the time, the display is off and the power consumption drops to about 3.5 W. I don't know how I got to 20 % atm. I can only do this if fsync is disabled. I tried before, but it would go on for every write. It died, very prematurely. > > do you really have so many fsync's going on that the disk spins up so much > that you would gain 10-20% battery life? Yes. Every autosave in LibreOffice triggers one. And I want autosave, but I want them in memory, not on disk. > > and what makes you think the extra spin-ups from fsyncs will cause your hard > drive to fail significantly earlier? (if you have a hard drive with a > limited number of spin-up cycles, you probably don't want to use laptop mode > at all) Experience, see above. Also, this is well described behavior. All hard disks are only designed to last a certain number of head loads and unloads. Spinning up and down even less. > > why do you think it's a possibility of loosing only 5% of data? Well, it really depends on the way your configure laptop mode. But in my case the laptop mode window is 20 minutes, the run time then is about 10-12 hrs. I can actually lose less than 5 % of the data created during the battery run time. It's a certain 1-1.5 hours (10-15 %) more work or a possible 20 mins (3.3 % actually) loss of work. I think it's a good deal. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-30 21:24 ` D. Jansen @ 2011-05-30 22:41 ` david 2011-05-31 2:03 ` Ted Ts'o 2011-05-31 8:22 ` Valdis.Kletnieks 2 siblings, 0 replies; 53+ messages in thread From: david @ 2011-05-30 22:41 UTC (permalink / raw) To: D. Jansen Cc: Theodore Tso, Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart [-- Attachment #1: Type: TEXT/PLAIN, Size: 3642 bytes --] On Mon, 30 May 2011, D. Jansen wrote: > On Mon, May 30, 2011 at 10:53 PM, <david@lang.hm> wrote: >> On Mon, 30 May 2011, D. Jansen wrote: >> >>> On Mon, May 30, 2011 at 8:45 PM, <david@lang.hm> wrote: >>>> it doesn't take running a mail server, even a mail client will have the >>>> same >>>> risk. If you use POP for mail (a very common option) then you download >>>> messages and tell the server to delete them. if you do not really save >>>> them >>>> (one fsync after they are all saved), then you can loose everything that >>>> you >>>> downloaded. >>> >>> Yes, I know. It's the same argument again and again. I understand not >>> everybody wants this. But some do. Some prefer working 10-20% longer >>> on battery (certainty) instead of possibly losing 5 % data >>> (possibility) or losing all your data (possibility if you use laptop >>> mode and the hard disk wakes up again and again and eventually wears >>> out). >> >> those are some powerful numbers you are throwing around, can you back them >> up? > > Yes. My netbook uses 4.8 W with hard disk off. 0.5 W more with hard > disk on. A lot of the time, the display is off and the power > consumption drops to about 3.5 W. I don't know how I got to 20 % atm. > I can only do this if fsync is disabled. I tried before, but it would > go on for every write. It died, very prematurely. what are you doing that's causing so many fsyncs? >> >> do you really have so many fsync's going on that the disk spins up so much >> that you would gain 10-20% battery life? > > Yes. Every autosave in LibreOffice triggers one. And I want autosave, > but I want them in memory, not on disk. are you sure that it's doing a fsync, not just a normal write? in laptop mode the drive will spin up occasionally for normal writes (to keep you from loosing too much data). I would expect that autosave is infrequent enough that (absent all other activity), each autosave will result in a spin-up >> and what makes you think the extra spin-ups from fsyncs will cause your hard >> drive to fail significantly earlier? (if you have a hard drive with a >> limited number of spin-up cycles, you probably don't want to use laptop mode >> at all) > > Experience, see above. Also, this is well described behavior. All hard > disks are only designed to last a certain number of head loads and > unloads. Spinning up and down even less. right, but the trade-off of laptop mode is that it will do _more_ spin ups on the theory that it's better to spin up the drive once ina while than to keep it running. If you are looking at total power consumption this is the case, but if you are worried about load/unload cycles this is not the case. the question I was asking is how do you know that fsync is the cause of these extra spin-ups? if you disable fsync, but still do the same number of writes, willyou really have far fewer spin-up events? I don't think that it would make nearly the difference ou think it is. >> why do you think it's a possibility of loosing only 5% of data? > > Well, it really depends on the way your configure laptop mode. But in > my case the laptop mode window is 20 minutes, the run time then is > about 10-12 hrs. I can actually lose less than 5 % of the data created > during the battery run time. It's a certain 1-1.5 hours (10-15 %) more > work or a possible 20 mins (3.3 % actually) loss of work. I think it's > a good deal. as others point out, one missing write can cause you to loose a large chunk of data, not just the data being worked on during that time. so if you end up with a corrupt file, you loose 100% of the file, not just the changes. David Lang ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-30 21:24 ` D. Jansen 2011-05-30 22:41 ` david @ 2011-05-31 2:03 ` Ted Ts'o 2011-05-31 2:26 ` david ` (2 more replies) 2011-05-31 8:22 ` Valdis.Kletnieks 2 siblings, 3 replies; 53+ messages in thread From: Ted Ts'o @ 2011-05-31 2:03 UTC (permalink / raw) To: D. Jansen Cc: david, Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart On Mon, May 30, 2011 at 11:24:03PM +0200, D. Jansen wrote: > > do you really have so many fsync's going on that the disk spins up so much > > that you would gain 10-20% battery life? > > Yes. Every autosave in LibreOffice triggers one. And I want autosave, > but I want them in memory, not on disk. What on *heck* good does an autosave to memory do? Are you afraid your X server is going to go poof, or OpenOffice is going to crash on you? I can't remember the last time this has happened to me. It's typically a system crash or a power loss that causes me to lose an OpenOffice session. > > and what makes you think the extra spin-ups from fsyncs will cause your hard > > drive to fail significantly earlier? (if you have a hard drive with a > > limited number of spin-up cycles, you probably don't want to use laptop mode > > at all) > > Experience, see above. Also, this is well described behavior. All hard > disks are only designed to last a certain number of head loads and > unloads. Spinning up and down even less. Modern laptop drives are designed for 600,000 to a million head loads and unloads. Open Office by default autosaves once every 15 minutes. So if you leave your system running (on battery?!?) 24x7, with open office open all that time, even with HDD which is only rated for 600k load cycles, that's 4.5 years. And of course, normally such a system is powered all the time, and the hard drive shouldn't be spinning down if you're on the AC mains. The real fix is that applications need to be fixed to be less write-happy. Firefox example, used to request a transactional update to a sqllite database on every web click. Laptop mode isn't the right place to fix this, since if you try to stop the writes from hitting the disk, you'll still end up burning memory that can't be released until the data is written to disk. My guess is that it's not OpenOffice which is causing you the problem, unless you've adjusted the autosave interval, in which case it's your own fault. It's probably some application like firefox, or syslog. (Syslog can be fixed by prefixing syslog.conf entries with the '-' character so that the files aren't sync'ed after every log update, for example. Or not to log as much in the first place.) - Ted ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-31 2:03 ` Ted Ts'o @ 2011-05-31 2:26 ` david 2011-05-31 13:47 ` D. Jansen 2011-06-02 7:42 ` D. Jansen 2 siblings, 0 replies; 53+ messages in thread From: david @ 2011-05-31 2:26 UTC (permalink / raw) To: Ted Ts'o Cc: D. Jansen, Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart On Mon, 30 May 2011, Ted Ts'o wrote: > On Mon, May 30, 2011 at 11:24:03PM +0200, D. Jansen wrote: > It's probably some application like firefox, or syslog. (Syslog can be > fixed by prefixing syslog.conf entries with the '-' character so that > the files aren't sync'ed after every log update, for example. Or not to > log as much in the first place.) note that rsyslog (which is the default on new versions of just about every distro) changes this default so it no longer does a fsync for each log entry. It can be configured to be ultra paranoid if you need it, but by default it's not. David Lang ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-31 2:03 ` Ted Ts'o 2011-05-31 2:26 ` david @ 2011-05-31 13:47 ` D. Jansen 2011-05-31 18:23 ` david 2011-06-02 7:42 ` D. Jansen 2 siblings, 1 reply; 53+ messages in thread From: D. Jansen @ 2011-05-31 13:47 UTC (permalink / raw) To: Ted Ts'o, D. Jansen, david, Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart On Tue, May 31, 2011 at 4:03 AM, Ted Ts'o <tytso@mit.edu> wrote: > On Mon, May 30, 2011 at 11:24:03PM +0200, D. Jansen wrote: >> > do you really have so many fsync's going on that the disk spins up so much >> > that you would gain 10-20% battery life? >> >> Yes. Every autosave in LibreOffice triggers one. And I want autosave, >> but I want them in memory, not on disk. > > What on *heck* good does an autosave to memory do? Are you afraid > your X server is going to go poof, or OpenOffice is going to crash on > you? Unfortunately, yes. This happens to me regularly, e.g. roughly every 10th resume. It's a poulsbo system. ;) Another reason is a Java extension that makes it crash happy. (Please don't tell me now that the real fix is to fix poulsbo...) >I can't remember the last time this has happened to me. It's > typically a system crash or a power loss that causes me to lose an > OpenOffice session. Well, good for you! Power loss didn't ever occur to me on the other hand, at least not on my netbook. >> > and what makes you think the extra spin-ups from fsyncs will cause your hard >> > drive to fail significantly earlier? (if you have a hard drive with a >> > limited number of spin-up cycles, you probably don't want to use laptop mode >> > at all) >> >> Experience, see above. Also, this is well described behavior. All hard >> disks are only designed to last a certain number of head loads and >> unloads. Spinning up and down even less. > > Modern laptop drives are designed for 600,000 to a million head loads > and unloads. Open Office by default autosaves once every 15 minutes. > So if you leave your system running (on battery?!?) 24x7, with open > office open all that time, even with HDD which is only rated for 600k > load cycles, that's 4.5 years. Yeah, exactly what I had thought -- enabling laptop mode for the first time. After the hard disk started to show more read errors and almost crashed so I had to exchange it, I reconsidered. I wish specifications would be more in conformity with reality... Good thing I watch my smart log and caught it in time to avoid data loss. Though I do have a solid backup routine to avoid serious issues. > And of course, normally such a system > is powered all the time, and the hard drive shouldn't be spinning down > if you're on the AC mains. Well, I use my netbook on the go mostly. I guess we just have different habits. > > The real fix is that applications need to be fixed to be less > write-happy. Firefox example, used to request a transactional update > to a sqllite database on every web click. Laptop mode isn't the right > place to fix this, since if you try to stop the writes from hitting > the disk, you'll still end up burning memory that can't be released > until the data is written to disk. (...) That is another fix that everybody can benefit from. And another reason to provide barrier support for user space. (We have a new trojan horse!) ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-31 13:47 ` D. Jansen @ 2011-05-31 18:23 ` david 2011-05-31 18:37 ` D. Jansen 0 siblings, 1 reply; 53+ messages in thread From: david @ 2011-05-31 18:23 UTC (permalink / raw) To: D. Jansen Cc: Ted Ts'o, Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart [-- Attachment #1: Type: TEXT/PLAIN, Size: 1219 bytes --] On Tue, 31 May 2011, D. Jansen wrote: > On Tue, May 31, 2011 at 4:03 AM, Ted Ts'o <tytso@mit.edu> wrote: >> On Mon, May 30, 2011 at 11:24:03PM +0200, D. Jansen wrote: >>>> do you really have so many fsync's going on that the disk spins up so much >>>> that you would gain 10-20% battery life? >>> >>> Yes. Every autosave in LibreOffice triggers one. And I want autosave, >>> but I want them in memory, not on disk. >> >> What on *heck* good does an autosave to memory do? Are you afraid >> your X server is going to go poof, or OpenOffice is going to crash on >> you? > Unfortunately, yes. This happens to me regularly, e.g. roughly every > 10th resume. It's a poulsbo system. ;) > Another reason is a Java extension that makes it crash happy. > (Please don't tell me now that the real fix is to fix poulsbo...) > >> I can't remember the last time this has happened to me. It's >> typically a system crash or a power loss that causes me to lose an >> OpenOffice session. > > Well, good for you! Power loss didn't ever occur to me on the other > hand, at least not on my netbook. failure to resume is effectivly power loss. a autosave to ram would be lost, just like with a power loss or system crash. David Lang ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-31 18:23 ` david @ 2011-05-31 18:37 ` D. Jansen 2011-05-31 18:54 ` david 0 siblings, 1 reply; 53+ messages in thread From: D. Jansen @ 2011-05-31 18:37 UTC (permalink / raw) To: david Cc: Ted Ts'o, Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart On Tue, May 31, 2011 at 8:23 PM, <david@lang.hm> wrote: > On Tue, 31 May 2011, D. Jansen wrote: > >> On Tue, May 31, 2011 at 4:03 AM, Ted Ts'o <tytso@mit.edu> wrote: >>> I can't remember the last time this has happened to me. It's >>> typically a system crash or a power loss that causes me to lose an >>> OpenOffice session. >> >> Well, good for you! Power loss didn't ever occur to me on the other >> hand, at least not on my netbook. > > failure to resume is effectivly power loss. a autosave to ram would be lost, > just like with a power loss or system crash. > Well in my case only X crashes and the autosave is saved. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-31 18:37 ` D. Jansen @ 2011-05-31 18:54 ` david 2011-05-31 19:04 ` D. Jansen 0 siblings, 1 reply; 53+ messages in thread From: david @ 2011-05-31 18:54 UTC (permalink / raw) To: D. Jansen Cc: Ted Ts'o, Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart [-- Attachment #1: Type: TEXT/PLAIN, Size: 1301 bytes --] On Tue, 31 May 2011, D. Jansen wrote: > On Tue, May 31, 2011 at 8:23 PM, <david@lang.hm> wrote: >> On Tue, 31 May 2011, D. Jansen wrote: >> >>> On Tue, May 31, 2011 at 4:03 AM, Ted Ts'o <tytso@mit.edu> wrote: > >>>> I can't remember the last time this has happened to me. It's >>>> typically a system crash or a power loss that causes me to lose an >>>> OpenOffice session. >>> >>> Well, good for you! Power loss didn't ever occur to me on the other >>> hand, at least not on my netbook. >> >> failure to resume is effectivly power loss. a autosave to ram would be lost, >> just like with a power loss or system crash. >> > Well in my case only X crashes and the autosave is saved. but you have no guarantee that it would be saved without fsync. it looks like libreoffice does 2 fsyncs per snapshot saved. This is why your data is saved. If this is only every 15 minutes (the default), and you have the disk spin-up time set to every 20 minutes, you will be spinning up the disk just about as frequently anyway. now if the system isn't flushing things to disk when it spins up for the fsync, then there is something wrong. but if it is working as designed, you get one spin-up every 15 min instead of one spin-up every 20 minutes, that should not be enough to kill anything. David Lang ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-31 18:54 ` david @ 2011-05-31 19:04 ` D. Jansen 0 siblings, 0 replies; 53+ messages in thread From: D. Jansen @ 2011-05-31 19:04 UTC (permalink / raw) To: david Cc: Ted Ts'o, Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart On Tue, May 31, 2011 at 8:54 PM, <david@lang.hm> wrote: > On Tue, 31 May 2011, D. Jansen wrote: > >> On Tue, May 31, 2011 at 8:23 PM, <david@lang.hm> wrote: >>> >>> On Tue, 31 May 2011, D. Jansen wrote: >>> >>>> On Tue, May 31, 2011 at 4:03 AM, Ted Ts'o <tytso@mit.edu> wrote: >> >>>>> I can't remember the last time this has happened to me. It's >>>>> typically a system crash or a power loss that causes me to lose an >>>>> OpenOffice session. >>>> >>>> Well, good for you! Power loss didn't ever occur to me on the other >>>> hand, at least not on my netbook. >>> >>> failure to resume is effectivly power loss. a autosave to ram would be >>> lost, >>> just like with a power loss or system crash. >>> >> Well in my case only X crashes and the autosave is saved. > > but you have no guarantee that it would be saved without fsync. But I would with barriers. > > it looks like libreoffice does 2 fsyncs per snapshot saved. This is why your > data is saved. > > If this is only every 15 minutes (the default), and you have the disk > spin-up time set to every 20 minutes, you will be spinning up the disk just > about as frequently anyway. > > now if the system isn't flushing things to disk when it spins up for the > fsync, then there is something wrong. > > but if it is working as designed, you get one spin-up every 15 min instead > of one spin-up every 20 minutes, that should not be enough to kill anything. Yes, if there was nothing else, that's probably true. But _many_ applications use fsync, even KDE's plasma shell I think. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-31 2:03 ` Ted Ts'o 2011-05-31 2:26 ` david 2011-05-31 13:47 ` D. Jansen @ 2011-06-02 7:42 ` D. Jansen 2 siblings, 0 replies; 53+ messages in thread From: D. Jansen @ 2011-06-02 7:42 UTC (permalink / raw) To: Ted Ts'o, D. Jansen, david, Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart On Tue, May 31, 2011 at 4:03 AM, Ted Ts'o <tytso@mit.edu> wrote: > The real fix is that applications need to be fixed to be less > write-happy. So do you agree to export barriers to user space? Because as you said yourself, that would be the only way for apps to avoid fsync without losing data safety. And since you want the problem fixed with the apps, we need to provide them with the means to do so, right? ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-30 21:24 ` D. Jansen 2011-05-30 22:41 ` david 2011-05-31 2:03 ` Ted Ts'o @ 2011-05-31 8:22 ` Valdis.Kletnieks 2 siblings, 0 replies; 53+ messages in thread From: Valdis.Kletnieks @ 2011-05-31 8:22 UTC (permalink / raw) To: D. Jansen Cc: david, Theodore Tso, Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart [-- Attachment #1: Type: text/plain, Size: 457 bytes --] On Mon, 30 May 2011 23:24:03 +0200, "D. Jansen" said: > > do you really have so many fsync's going on that the disk spins up so much > > that you would gain 10-20% battery life? > > Yes. Every autosave in LibreOffice triggers one. And I want autosave, > but I want them in memory, not on disk. I think what you want from autosave isn't what the LibreOffice team wants from autosave. Remember, if they're in memory, they aren't *really* saved. [-- Attachment #2: Type: application/pgp-signature, Size: 227 bytes --] ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-30 19:49 ` D. Jansen 2011-05-30 20:53 ` david @ 2011-05-30 22:10 ` Jesper Juhl 1 sibling, 0 replies; 53+ messages in thread From: Jesper Juhl @ 2011-05-30 22:10 UTC (permalink / raw) To: D. Jansen Cc: david, Theodore Tso, Oliver Neukum, akpm, linux-kernel, Dave Chinner, njs, bart [-- Attachment #1: Type: TEXT/PLAIN, Size: 4295 bytes --] On Mon, 30 May 2011, D. Jansen wrote: > On Mon, May 30, 2011 at 8:45 PM, <david@lang.hm> wrote: > > On Mon, 30 May 2011, D. Jansen wrote: > > > >> On Mon, May 30, 2011 at 8:02 PM, <david@lang.hm> wrote: > >>> > >>> no, you cannot just change a fsync to a barrier, in some cases the data > >>> absolutly needs to be saved, not just ordered (remember the example of a > >>> mail server telling the other system that the data can be deleted after a > >>> fsync returns) > >> > >> I'm not really sure I why shouldn't have that choice as a user. Just > >> because someone else could be running a mailserver on his system and > >> configure it in a way that it doesn't behave as it should? > >> If he really wants to do that there's really nothing we can do to stop > >> him. I'm sure there are other ways existing kernel options can be used > >> to make software behave different than it should. Are we going to > >> remove them all now? > >> > >> The big problem is that so far only fsync existed and lots of software > >> seemingly abuses it as an expensive write barrier. And it would really > >> be lovely to have the choice to stop that on an opt-in basis in laptop > >> mode. > > > > is the benifit of not spinning up the disk really worth the risk of loosing > > data? > > Is that a choice I can make myself? Is that a choice I make with > laptop mode? Or is that a choice I may only make if I'm willing to > modify and compile my own kernel? > > > > and should this really be a global across-the-board option? > > The point is option imo. > > > > the problem is that most users don't know what their system is running, or > > what effect disaling fsync would have. those that do can probably use > > LD_PRELOAD to override fsync calls. > > As we found out, they can't. But if we export barrier, I hope a > library could wrap fsyncs into barriers. Is that the case? > > > > it doesn't take running a mail server, even a mail client will have the same > > risk. If you use POP for mail (a very common option) then you download > > messages and tell the server to delete them. if you do not really save them > > (one fsync after they are all saved), then you can loose everything that you > > downloaded. > > Yes, I know. It's the same argument again and again. I understand not > everybody wants this. But some do. Some prefer working 10-20% longer > on battery (certainty) instead of possibly losing 5 % data > (possibility) or losing all your data (possibility if you use laptop > mode and the hard disk wakes up again and again and eventually wears > out). That's why there's laptop mode. And this would play into laptop > mode and prevent the hard disk from breaking down prematurely and > saving battery. > But how can you say that you are only risking 5% data loss or only risking your most recent data? You can't. fsync provides guarantees and applications take advantage of that guarantee and you have no way of knowing what breaking those guarantees will mean for all applications in the system (not without a *lot* of research work at least). You may be willing to lose a few emails or a few recently modified plain text files or office documents. But how would you feel if that multi megabyte or gigabyte database that has been build up over months with your work suddenly became completely unrecoverable because the application maintaining it made some assumptions based on fsync's guarantees and those guarantees were suddenly broken? My guess is you wouldn't be very happy. Some applications may only need the ordering guarantee that fsync provides, not the "data is safely on media" guarantee. So perhaps if a new barrier syscall was provided that gave applications just the lighter ordering guarantee, then over time those apps could transition to use it instead of fsync. Apps that really do need the "data is safely on media" guarantee would of course continue to call fsync, but the number of overall fsync calls in the system would decrease over time - this would help your battery life and we'd still have a working fsync for those apps that really do need it. How does that sound? -- Jesper Juhl <jj@chaosbits.net> http://www.chaosbits.net/ Don't top-post http://www.catb.org/jargon/html/T/top-post.html Plain text mails only, please. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-30 18:45 ` david 2011-05-30 19:49 ` D. Jansen @ 2011-05-31 6:48 ` Oliver Neukum 1 sibling, 0 replies; 53+ messages in thread From: Oliver Neukum @ 2011-05-31 6:48 UTC (permalink / raw) To: david Cc: D. Jansen, Theodore Tso, akpm, linux-kernel, Dave Chinner, njs, bart Am Montag, 30. Mai 2011, 20:45:42 schrieb david@lang.hm: > > The big problem is that so far only fsync existed and lots of software > > seemingly abuses it as an expensive write barrier. And it would really > > be lovely to have the choice to stop that on an opt-in basis in laptop > > mode. > > is the benifit of not spinning up the disk really worth the risk of > loosing data? That is a policy question. > and should this really be a global across-the-board option? Possibly not. However this is a secondary question. > the problem is that most users don't know what their system is running, or > what effect disaling fsync would have. those that do can probably use > LD_PRELOAD to override fsync calls. Not without a loss of ordering, as we have discussed. Regards Oliver -- - - - SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany - - - ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-26 7:01 ` D. Jansen 2011-05-26 10:49 ` Theodore Tso @ 2011-05-26 19:31 ` Matthew Garrett 2011-05-27 7:22 ` D. Jansen 1 sibling, 1 reply; 53+ messages in thread From: Matthew Garrett @ 2011-05-26 19:31 UTC (permalink / raw) To: D. Jansen; +Cc: Oliver Neukum, Dave Chinner, linux-kernel, akpm, tytso On Thu, May 26, 2011 at 09:01:49AM +0200, D. Jansen wrote: > Exactly. 0.5 Watts is a lot of energy on a modern system. And on some > systems with proper link power management in sata and pcie it will be > even higher (~1 Watt I think). Think not only of the work that will be > lost if the system crashes (unlikely), but also of the work that could > never be done because the battery was gone (a certainty). I don't think link power management is relevant. The hysteresis there is sufficiently low that it's only going to make a difference during the point where you're actually throwing data across the link. -- Matthew Garrett | mjg59@srcf.ucam.org ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-26 19:31 ` Matthew Garrett @ 2011-05-27 7:22 ` D. Jansen 0 siblings, 0 replies; 53+ messages in thread From: D. Jansen @ 2011-05-27 7:22 UTC (permalink / raw) To: Matthew Garrett; +Cc: Oliver Neukum, Dave Chinner, linux-kernel, akpm, tytso On Thu, May 26, 2011 at 9:31 PM, Matthew Garrett <mjg59@srcf.ucam.org> wrote: > On Thu, May 26, 2011 at 09:01:49AM +0200, D. Jansen wrote: > >> Exactly. 0.5 Watts is a lot of energy on a modern system. And on some >> systems with proper link power management in sata and pcie it will be >> even higher (~1 Watt I think). Think not only of the work that will be >> lost if the system crashes (unlikely), but also of the work that could >> never be done because the battery was gone (a certainty). > > I don't think link power management is relevant. The hysteresis there is > sufficiently low that it's only going to make a difference during the > point where you're actually throwing data across the link. > Ok, thanks! I'm not sure how much power could be saved in that case on different system. All I know is that it's about 0.5 Watts on my system. All I want is to help people to be able to take advantage of that extra bit to have extra battery (in my case 10-20%) for last bit of <insert activity>, in my case taking notes. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-19 13:34 [rfc] Ignore Fsync Calls in Laptop_Mode Dennis Jansen 2011-05-19 13:43 ` Alan Cox 2011-05-20 3:39 ` Dave Chinner @ 2011-05-20 15:28 ` Valdis.Kletnieks 2011-05-20 16:40 ` D. Jansen 2 siblings, 1 reply; 53+ messages in thread From: Valdis.Kletnieks @ 2011-05-20 15:28 UTC (permalink / raw) To: d.g.jansen; +Cc: linux-kernel, akpm, tytso [-- Attachment #1: Type: text/plain, Size: 1473 bytes --] On Thu, 19 May 2011 15:34:46 +0200, Dennis Jansen said: > Testing: > I've been using this workaround on my netbook for over six months now. > It works as expected for me with all software in a Ubuntu 9.10 > environment and saves me at least 0.5 Watt or roughly 10 % battery > time - without destroying my hard disk. I have seen no negative side > effects. > > What I didn't test: > I didn't test this on different platforms and environments. But as > it's generic code I expect no different behavior. I didn't benchmark > how this might affect performance due to the additional if() check for > every fsync call. But as the fsync is rather expensive and not used > *that* much, I think it should have not noticeable impact. How much destructive testing did you do? In the 6 months, how many times did the system crash (or had the battery pulled out, or whatever) while large amounts of data were still pending after apps thought they were fsync'ed? How much crash testing was done against apps that use fsync for ordering or correctness reasons? > In short: That it makes laptop_mode work as advertised IHO is no valid > point against this solution. And if, we could consider a sending a > printk the first time an fsync is skipped. That would be the same printk that the user never actually *sees* because your patch suppressed syslogd's fsync to guarantee it made it to disk, so it was lost when the system crashed soon thereafter, along with the user's work? ;) [-- Attachment #2: Type: application/pgp-signature, Size: 227 bytes --] ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-20 15:28 ` Valdis.Kletnieks @ 2011-05-20 16:40 ` D. Jansen 2011-05-20 22:03 ` torbenh 2011-05-23 8:22 ` Jesper Juhl 0 siblings, 2 replies; 53+ messages in thread From: D. Jansen @ 2011-05-20 16:40 UTC (permalink / raw) To: Valdis.Kletnieks; +Cc: linux-kernel, akpm, tytso On Fri, May 20, 2011 at 5:28 PM, <Valdis.Kletnieks@vt.edu> wrote: > On Thu, 19 May 2011 15:34:46 +0200, Dennis Jansen said: > >> Testing: >> I've been using this workaround on my netbook for over six months now. >> It works as expected for me with all software in a Ubuntu 9.10 >> environment and saves me at least 0.5 Watt or roughly 10 % battery >> time - without destroying my hard disk. I have seen no negative side >> effects. > > How much destructive testing did you do? In the 6 months, how many times did > the system crash (or had the battery pulled out, or whatever) while large > amounts of data were still pending after apps thought they were fsync'ed? How > much crash testing was done against apps that use fsync for ordering or > correctness reasons? I don't see the point in verifying the obvious. Of course applications that rely on fsync will lose data. The real problem comes with ordering correctness, which could actually _destroy previous data_ as well. In my scenario (office applications, browsing) I have not hit such a problem. Does anyone know a Linux app that actually does rely on ordering correctness? Is that one which is used on a laptop? In laptop mode? (-> when on battery?) Because so far the discussion seems to be running in circles around Alan's mailer daemon. I would just shut that down on enabling laptop mode. Problem solved. But I don't run that on my laptop, anyway, esp. in laptop mode. I wonder who would, too. > >> In short: That it makes laptop_mode work as advertised IHO is no valid >> point against this solution. And if, we could consider a sending a >> printk the first time an fsync is skipped. > > That would be the same printk that the user never actually *sees* because your > patch suppressed syslogd's fsync to guarantee it made it to disk, so it was > lost when the system crashed soon thereafter, along with the user's work? ;) That is a good point! So one last fsync after the message? :) Or a beep? But I understand something else here as well: Don't give users the choice to easily destroy their systems. It would be like having a packaged libeatmydata that a user could just download and install (like this one http://packages.ubuntu.com/oneiric/eatmydata) but with fsync disabled for all programs by default upon installation. It would be a hassle to support, because users would just enable it without understanding what they do and later complain that data is lost. Could it be that this is a big part of the reason people don't like to even think about the idea in detail? Maybe we could then discuss more about how to prevent users from using this without knowing about what they do. Apart from ordering, I don't see the big difference between data lost through normal writes (postponed in laptop mode) and fsync writes (postponed in laptop mode). Are there really apps that corrupt their data without fsync? (That you end up without the new data like new emails is a given and normal with laptop mode.) Which? Thanks, I already got the email example when Alan explained it. It's just that I doubt that the email server scenario is very likely. Email client, yes. But an email client can leave the data on the server. Let's not say "this is not possible, because a badly configured system could cause a problem." Isn't that always the case? On the other hand, one could also say that applications simply should all not use fsync in the first place, but e.g. featherstick (http://lwn.net/Articles/354861/) instead. It seems the resistance to this idea is very high, though I think more for fear of abuse and/or badly configured systems than due to normal _laptop mode_ use cases. Cheers ps. Please don't put me into any filters or ignore lists just yet, because I _am_ aware of the data loss risk. And I won't complain about data loss on that system, but just quietly enjoy my ~ 0.5 W power savings, even with sqlite-based and other fsyncing software running. Maybe someone will sometime write a patch that "featherstickyfies" fsync in laptop mode. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-20 16:40 ` D. Jansen @ 2011-05-20 22:03 ` torbenh 2011-05-21 8:23 ` D. Jansen 2011-05-23 8:22 ` Jesper Juhl 1 sibling, 1 reply; 53+ messages in thread From: torbenh @ 2011-05-20 22:03 UTC (permalink / raw) To: D. Jansen; +Cc: Valdis.Kletnieks, linux-kernel, akpm, tytso On Fri, May 20, 2011 at 06:40:49PM +0200, D. Jansen wrote: > On Fri, May 20, 2011 at 5:28 PM, <Valdis.Kletnieks@vt.edu> wrote: > > On Thu, 19 May 2011 15:34:46 +0200, Dennis Jansen said: > > > >> Testing: > >> I've been using this workaround on my netbook for over six months now. > >> It works as expected for me with all software in a Ubuntu 9.10 > >> environment and saves me at least 0.5 Watt or roughly 10 % battery > >> time - without destroying my hard disk. I have seen no negative side > >> effects. > > > > How much destructive testing did you do? In the 6 months, how many times did > > the system crash (or had the battery pulled out, or whatever) while large > > amounts of data were still pending after apps thought they were fsync'ed? How > > much crash testing was done against apps that use fsync for ordering or > > correctness reasons? > > I don't see the point in verifying the obvious. Of course applications > that rely on fsync will lose data. > The real problem comes with ordering correctness, which could actually > _destroy previous data_ as well. > In my scenario (office applications, browsing) I have not hit such a problem. how about making fsync block until the harddisk spins up ? this would also enable you to detect these apps you wouldnt be using in laptop mode anyways ? -- torben Hohn ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-20 22:03 ` torbenh @ 2011-05-21 8:23 ` D. Jansen 0 siblings, 0 replies; 53+ messages in thread From: D. Jansen @ 2011-05-21 8:23 UTC (permalink / raw) To: Valdis.Kletnieks, linux-kernel, akpm, tytso On Sat, May 21, 2011 at 12:03 AM, torbenh <torbenh@gmx.de> wrote: > On Fri, May 20, 2011 at 06:40:49PM +0200, D. Jansen wrote: >> On Fri, May 20, 2011 at 5:28 PM, <Valdis.Kletnieks@vt.edu> wrote: >> > On Thu, 19 May 2011 15:34:46 +0200, Dennis Jansen said: >> > >> >> Testing: >> >> I've been using this workaround on my netbook for over six months now. >> >> It works as expected for me with all software in a Ubuntu 9.10 >> >> environment and saves me at least 0.5 Watt or roughly 10 % battery >> >> time - without destroying my hard disk. I have seen no negative side >> >> effects. >> > >> > How much destructive testing did you do? In the 6 months, how many times did >> > the system crash (or had the battery pulled out, or whatever) while large >> > amounts of data were still pending after apps thought they were fsync'ed? How >> > much crash testing was done against apps that use fsync for ordering or >> > correctness reasons? >> >> I don't see the point in verifying the obvious. Of course applications >> that rely on fsync will lose data. >> The real problem comes with ordering correctness, which could actually >> _destroy previous data_ as well. >> In my scenario (office applications, browsing) I have not hit such a problem. > > how about making fsync block until the harddisk spins up ? > this would also enable you to detect these apps you wouldnt be using in > laptop mode anyways ? Very nice idea. I had tried that some time ago. Depending on the error codes I have, the apps either failed to save and complained to the user - or - kept trying to fsync again and again. Of course the latter wouldn't really be a problem unless a race occurs. And we have the benefit of being honest to user space and letting them know we are not fsyncing. And the fsync occurs as soon as we disable laptop mode and plug into AC. I have to try that out again. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [rfc] Ignore Fsync Calls in Laptop_Mode 2011-05-20 16:40 ` D. Jansen 2011-05-20 22:03 ` torbenh @ 2011-05-23 8:22 ` Jesper Juhl 1 sibling, 0 replies; 53+ messages in thread From: Jesper Juhl @ 2011-05-23 8:22 UTC (permalink / raw) To: D. Jansen; +Cc: Valdis.Kletnieks, linux-kernel, akpm, tytso [-- Attachment #1: Type: TEXT/PLAIN, Size: 1979 bytes --] On Fri, 20 May 2011, D. Jansen wrote: > On Fri, May 20, 2011 at 5:28 PM, <Valdis.Kletnieks@vt.edu> wrote: > > On Thu, 19 May 2011 15:34:46 +0200, Dennis Jansen said: > > > >> Testing: > >> I've been using this workaround on my netbook for over six months now. > >> It works as expected for me with all software in a Ubuntu 9.10 > >> environment and saves me at least 0.5 Watt or roughly 10 % battery > >> time - without destroying my hard disk. I have seen no negative side > >> effects. > > > > How much destructive testing did you do? In the 6 months, how many times did > > the system crash (or had the battery pulled out, or whatever) while large > > amounts of data were still pending after apps thought they were fsync'ed? How > > much crash testing was done against apps that use fsync for ordering or > > correctness reasons? > > I don't see the point in verifying the obvious. Of course applications > that rely on fsync will lose data. > The real problem comes with ordering correctness, which could actually > _destroy previous data_ as well. > In my scenario (office applications, browsing) I have not hit such a problem. > > Does anyone know a Linux app that actually does rely on ordering > correctness? Is that one which is used on a laptop? In laptop mode? > (-> when on battery?) Because so far the discussion seems to be > running in circles around Alan's mailer daemon. I would just shut that > down on enabling laptop mode. Problem solved. But I don't run that on > my laptop, anyway, esp. in laptop mode. I wonder who would, too. > PostgreSQL's Write Ahead Log (WAL) would be one such example as far as I can see ( http://www.westnet.com/~gsmith/content/postgresql/TuningPGWAL.htm ). And yes, people run that on laptops while on battery (I know that for a fact since I do that myself). -- Jesper Juhl <jj@chaosbits.net> http://www.chaosbits.net/ Don't top-post http://www.catb.org/jargon/html/T/top-post.html Plain text mails only, please. ^ permalink raw reply [flat|nested] 53+ messages in thread
end of thread, other threads:[~2011-06-02 7:42 UTC | newest] Thread overview: 53+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2011-05-19 13:34 [rfc] Ignore Fsync Calls in Laptop_Mode Dennis Jansen 2011-05-19 13:43 ` Alan Cox [not found] ` <BANLkTikFRwPY_qOQpPmCmNUJbBUsGcuGUw@mail.gmail.com> [not found] ` <20110519153928.40521b93@lxorguk.ukuu.org.uk> 2011-05-19 15:01 ` D. Jansen 2011-05-19 15:02 ` D. Jansen 2011-05-20 15:34 ` Valdis.Kletnieks 2011-05-20 3:39 ` Dave Chinner 2011-05-20 6:01 ` D. Jansen 2011-05-22 0:48 ` Dave Chinner 2011-05-23 8:12 ` Oliver Neukum 2011-05-23 13:05 ` D. Jansen 2011-05-25 0:00 ` Dave Chinner 2011-05-25 6:50 ` Oliver Neukum 2011-05-26 7:01 ` D. Jansen 2011-05-26 10:49 ` Theodore Tso [not found] ` <BANLkTimNjt4=9v33Z9Nr12xa6GmyJ-Ue5A@mail.gmail.com> 2011-05-26 13:31 ` Ted Ts'o 2011-05-26 16:05 ` D. Jansen 2011-05-26 16:21 ` Ted Ts'o 2011-05-27 7:12 ` D. Jansen 2011-05-27 14:17 ` Theodore Tso 2011-05-27 17:51 ` david 2011-05-29 10:45 ` D. Jansen 2011-05-30 1:53 ` david 2011-05-30 7:13 ` Oliver Neukum 2011-05-30 12:55 ` Valdis.Kletnieks 2011-05-30 18:03 ` david 2011-05-30 13:55 ` D. Jansen 2011-05-30 18:02 ` david 2011-05-30 18:28 ` D. Jansen 2011-05-30 18:43 ` Valdis.Kletnieks 2011-05-30 19:54 ` D. Jansen 2011-05-30 18:45 ` david 2011-05-30 19:49 ` D. Jansen 2011-05-30 20:53 ` david 2011-05-30 21:24 ` D. Jansen 2011-05-30 22:41 ` david 2011-05-31 2:03 ` Ted Ts'o 2011-05-31 2:26 ` david 2011-05-31 13:47 ` D. Jansen 2011-05-31 18:23 ` david 2011-05-31 18:37 ` D. Jansen 2011-05-31 18:54 ` david 2011-05-31 19:04 ` D. Jansen 2011-06-02 7:42 ` D. Jansen 2011-05-31 8:22 ` Valdis.Kletnieks 2011-05-30 22:10 ` Jesper Juhl 2011-05-31 6:48 ` Oliver Neukum 2011-05-26 19:31 ` Matthew Garrett 2011-05-27 7:22 ` D. Jansen 2011-05-20 15:28 ` Valdis.Kletnieks 2011-05-20 16:40 ` D. Jansen 2011-05-20 22:03 ` torbenh 2011-05-21 8:23 ` D. Jansen 2011-05-23 8:22 ` Jesper Juhl
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.