rename("a","b") succeeds multiple times race

rename("a","b") succeeds multiple times race

[Ian Jackson]

I'm running 2.2.25 on a dual PIII; I have a program that processes
mail messages which are left in a queue directory as uniquely named
files.  The queue runners each `claim' a message by renaming it away
from the initial filename, so that only one queue runner works on each
message.

However, this does not work because Linux erroneously allows several
processes to simultaneously and `successfully' rename the same file.
The filesystem in question is ext2.

I ran the system under strace, and saw (for example) the following, in
five straces of five different processes:

 02:11:47.293131 rename("q1988na-000xqY", "proc.1988na-000xqY") = 0
 02:11:47.354497 rename("q1988na-000xqY", "proc.1988na-000xqY") = 0
 02:11:47.412207 rename("q1988na-000xqY", "proc.1988na-000xqY") = 0
 02:11:47.414376 rename("q1988na-000xqY", "proc.1988na-000xqY") = 0
 02:11:47.414559 rename("q1988na-000xqY", "proc.1988na-000xqY") = 0

The q... filename was created by Exim 3.35, which did this (for
another message; I can't run the whole of the mail system under
strace):

open("/var/lib/news/mail2news2//temp.2223.chiark.greenend.org.uk", O_WRONLY|O_CREAT, 0660) = 6
[ fiddles with permissions of the file, writes data ]
stat("/var/lib/news/mail2news2//temp.2223.chiark.greenend.org.uk", {st_dev=makedev(8, 2), st_ino=201893, st_mode=S_IFREG|066
0, st_nlink=1, st_uid=9, st_gid=9, st_blksize=4096, st_blocks=0, st_size=0, st_atime=2003/04/23-11:32:14, st_mtime=2003/04/2
3-11:32:14, st_ctime=2003/04/23-11:32:14}) = 0
[...]
close(6)                                = 0
rename("/var/lib/news/mail2news2//temp.2223.chiark.greenend.org.uk", "/var/lib/news/mail2news2//q198HXy-000qWL") = 0

The q... filename is constructed by base-62-encoding the time and the
inode number.

So my questions are:

* Is this a known bug ?  Is it fixed in 2.4 ?

* I can perhaps work around it by having the queue runner rename the
  file to a name which also depends on its own pid, and then check
  that that file exists.  (This will come naturally because it only
  opens the file after renaming it.)  Will this work ?  Will it trash
  my filesystem or my kernel data structures ?

chiark:~> uname -av
Linux chiark 2.2.25 #2 SMP Wed Apr 23 13:05:23 BST 2003 i686 unknown
chiark:~> cat /proc/version
Linux version 2.2.25 (ian@chiark) (gcc version 2.7.2.3) #2 SMP Wed Apr 23 13:05:23 BST 2003
chiark:~>

My kernel is a stock 2.2.25 with patches to:
 * increase NR_TASKS to 2048
 * #define DEBUG 1 in st.c

The distribution is Debian woody; the queue runner software is my own,
written in Perl.

Ian.

Re: rename("a","b") succeeds multiple times race

[Chris Sykes]

[-- Attachment #1: Type: text/plain, Size: 1889 bytes --]

On Thu, Apr 24, 2003 at 12:57:07PM +0100, Ian Jackson wrote:
> I'm running 2.2.25 on a dual PIII; I have a program that processes
> mail messages which are left in a queue directory as uniquely named
> files.  The queue runners each `claim' a message by renaming it away
> from the initial filename, so that only one queue runner works on each
> message.
> 
> However, this does not work because Linux erroneously allows several
> processes to simultaneously and `successfully' rename the same file.
> The filesystem in question is ext2.
> 
> I ran the system under strace, and saw (for example) the following, in
> five straces of five different processes:
> 
>  02:11:47.293131 rename("q1988na-000xqY", "proc.1988na-000xqY") = 0
>  02:11:47.354497 rename("q1988na-000xqY", "proc.1988na-000xqY") = 0
>  02:11:47.412207 rename("q1988na-000xqY", "proc.1988na-000xqY") = 0
>  02:11:47.414376 rename("q1988na-000xqY", "proc.1988na-000xqY") = 0
>  02:11:47.414559 rename("q1988na-000xqY", "proc.1988na-000xqY") = 0

In rename(2):
[1] "If  newpath  already exists it will be atomically replaced
    (subject to a few conditions - see ERRORS below), so  that
    there  is  no point at which another process attempting to
    access newpath will find it missing."

 ...

[2] "However, when overwriting there will probably be a  window
    in  which both oldpath and newpath refer to the file being
    renamed."

Perhaps your program should link(2) the newpath to the oldpath, then
on success unlink(2) the oldpath.  link(2) will fail should newpath
already exist.  Of course this assumes that oldpath & newpath are on
the same filesystem.

-- 

(o-  Chris Sykes  -- GPG Key: http://www.sigsegv.plus.com/key.txt
//\       "Don't worry. Everything is getting nicely out of control ..."
V_/_                          Douglas Adams - The Salmon of Doubt


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: rename("a","b") succeeds multiple times race

[Ian Jackson]

Chris Sykes writes ("Re: rename("a","b") succeeds multiple times race"):
> In rename(2):

Linux manpages are hardly a definitive reference for the desired
behaviour of system calls !  Unfortunately SuSv2 and v3 are
unhelpfully vague.

Ian.

Re: rename("a","b") succeeds multiple times race

[Jamie Lokier]

Chris Sykes wrote:
> > I ran the system under strace, and saw (for example) the following, in
> > five straces of five different processes:
> > 
> >  02:11:47.293131 rename("q1988na-000xqY", "proc.1988na-000xqY") = 0
> >  02:11:47.354497 rename("q1988na-000xqY", "proc.1988na-000xqY") = 0
> >  02:11:47.412207 rename("q1988na-000xqY", "proc.1988na-000xqY") = 0
> >  02:11:47.414376 rename("q1988na-000xqY", "proc.1988na-000xqY") = 0
> >  02:11:47.414559 rename("q1988na-000xqY", "proc.1988na-000xqY") = 0
> 
> In rename(2):
> [1] "If  newpath  already exists it will be atomically replaced
>     (subject to a few conditions - see ERRORS below), so  that
>     there  is  no point at which another process attempting to
>     access newpath will find it missing."
> 
>  ...
> 
> [2] "However, when overwriting there will probably be a  window
>     in  which both oldpath and newpath refer to the file being
>     renamed."
> 
> Perhaps your program should link(2) the newpath to the oldpath, then
> on success unlink(2) the oldpath.  link(2) will fail should newpath
> already exist.  Of course this assumes that oldpath & newpath are on
> the same filesystem.

That would be most reliable, and you want to do that in case the
filesystem is NFS, too.  (On the other hand, there are filesystems
that support rename(2) but not link(2)).

However rename(2) should not successfully rename from the _old_ path
more than once, whether the new names are all the same or not.

-- Jamie

Re: rename("a","b") succeeds multiple times race

[viro]

On Thu, Apr 24, 2003 at 10:34:18PM +0100, Jamie Lokier wrote:
> That would be most reliable, and you want to do that in case the
> filesystem is NFS, too.  (On the other hand, there are filesystems
> that support rename(2) but not link(2)).
> 
> However rename(2) should not successfully rename from the _old_ path
> more than once, whether the new names are all the same or not.

It shouldn't, but it's a known problem with 2.2.  Unfixable without
massive change of locking scheme.

What happens is that "from" lookup of second rename() succeeds
before the first one is over.  As the result, you get an old
dentry of "a", _then_ second lookup gives you the same dentry -
renamed to "b" by then.  Then you get rename() done with
old and new dentries being identical.  Which is required to
return 0 and do nothing.

It's b0rken and 2.3/2.4/2.5 prevent that crap from happening by
saner locking rules (we keep the parent director{y,ies} locked
across the entire "do lookups for last components and proceed
with actual rename" sequence; see Documentation/filesystem/ for
more details).

2.2 is hopeless in that respect - to fix that nest of races we would
need to replace the locking of directories in namei.c and friends.
I doubt that Alan is happy with that idea...