rename("a", "b") would not always remove "a" on success. ?!!

rename("a", "b") would not always remove "a" on success. ?!!

[Denys Vlasenko]

Hi,

One of my users stumbled over a problem when power failure
hit his system at rename() and the filesystem he uses
(I don't know which) ended up having both old and new
file names in the directory. Basically, he ended up with
one file with two hardlinks pointing to it.

IOW: the scenario does not require unlucky power offs
to reproduce, just "ln a b" would do.

In his case these two particular hardlinks were pointing
to rotated log files.

When system restarted, it eventually tried to rotate files
again, via rename("a", "b").
rename succeeded, but since they are hardlinks, rename
did NOT remove "a".
Which made the logger process very confused.


The user dug into it and discovered that SUS actually
specifies this insane behavior:

http://pubs.opengroup.org/onlinepubs/9699919799/functions/rename.html

'If the old argument and the new argument resolve to either .... or different
directory entries for the same existing file, rename() shall return
successfully and perform no other action.'

It's incredible they had audacity to put such nonsense into standard.

The page says in "RATIONALE" section:

'The specification that if old and new refer to the same file is
intended to guarantee that:

rename("x", "x");

does not remove the file.'

Why didn't they just explicitly say that they actually want THIS
particular case to work correctly, not OTHER cases to be fucked up?!


Anyway. My question is, does it really need to be implemented in Linux?
It looks bogus to me, and it basically requires any program
to contain a work-around for this case. For example, mv from util-linux
apparently already has a workaround:

$ touch a; ln a b
$ strace mv a b
...
stat64("b", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
lstat64("a", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
lstat64("b", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
geteuid32()                             = 0
unlink("a")                             = 0
close(0)                                = 0
close(1)                                = 0
close(2)                                = 0
exit_group(0)                           = ?

-- 
vda

Re: rename("a", "b") would not always remove "a" on success. ?!!

[Pádraig Brady]

On 10/28/2011 04:25 PM, Denys Vlasenko wrote:
> Hi,
> 
> One of my users stumbled over a problem when power failure
> hit his system at rename() and the filesystem he uses
> (I don't know which) ended up having both old and new
> file names in the directory. Basically, he ended up with
> one file with two hardlinks pointing to it.
> 
> IOW: the scenario does not require unlucky power offs
> to reproduce, just "ln a b" would do.
> 
> In his case these two particular hardlinks were pointing
> to rotated log files.
> 
> When system restarted, it eventually tried to rotate files
> again, via rename("a", "b").
> rename succeeded, but since they are hardlinks, rename
> did NOT remove "a".
> Which made the logger process very confused.
> 
> 
> The user dug into it and discovered that SUS actually
> specifies this insane behavior:
> 
> http://pubs.opengroup.org/onlinepubs/9699919799/functions/rename.html
> 
> 'If the old argument and the new argument resolve to either .... or different
> directory entries for the same existing file, rename() shall return
> successfully and perform no other action.'
> 
> It's incredible they had audacity to put such nonsense into standard.
> 
> The page says in "RATIONALE" section:
> 
> 'The specification that if old and new refer to the same file is
> intended to guarantee that:
> 
> rename("x", "x");
> 
> does not remove the file.'
> 
> Why didn't they just explicitly say that they actually want THIS
> particular case to work correctly, not OTHER cases to be fucked up?!
> 
> 
> Anyway. My question is, does it really need to be implemented in Linux?
> It looks bogus to me, and it basically requires any program
> to contain a work-around for this case. For example, mv from util-linux
> apparently already has a workaround:
> 
> $ touch a; ln a b
> $ strace mv a b
> ...
> stat64("b", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
> lstat64("a", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
> lstat64("b", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
> geteuid32()                             = 0
> unlink("a")                             = 0
> close(0)                                = 0
> close(1)                                = 0
> close(2)                                = 0
> exit_group(0)                           = ?

mv is from coreutils BTW.
Here is the related comment from the source:

"Set *UNLINK_SRC if we've determined that the caller wants to do
`rename (a, b)' where `a' and `b' are distinct hard links to the same
file. In that case, the caller should try to unlink `a' and then return
successfully.  Ideally, we wouldn't have to do that, and we'd be
able to rely on rename to remove the source file.  However, POSIX
mistakenly requires that such a rename call do *nothing* and return
successfully."

Perhaps it could be brought up as an issue with the standards guys?

cheers,
Pádraig.

Re: rename("a", "b") would not always remove "a" on success. ?!!

[Eric Blake]

On 10/28/2011 09:32 AM, Pádraig Brady wrote:
>> http://pubs.opengroup.org/onlinepubs/9699919799/functions/rename.html
>>
>> 'If the old argument and the new argument resolve to either .... or different
>> directory entries for the same existing file, rename() shall return
>> successfully and perform no other action.'
>>
>> It's incredible they had audacity to put such nonsense into standard.
>>
>> The page says in "RATIONALE" section:
>>
>> 'The specification that if old and new refer to the same file is
>> intended to guarantee that:
>>
>> rename("x", "x");
>>
>> does not remove the file.'
>>
>> Why didn't they just explicitly say that they actually want THIS
>> particular case to work correctly, not OTHER cases to be fucked up?!

Because it is historical precedent, and changing it now would break 
software that has come to expect this behavior on hard links.

>>
>>
>> Anyway. My question is, does it really need to be implemented in Linux?
>> It looks bogus to me, and it basically requires any program
>> to contain a work-around for this case. For example, mv from util-linux
>> apparently already has a workaround:
>>
>> $ touch a; ln a b
>> $ strace mv a b
>> ...
>> stat64("b", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
>> lstat64("a", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
>> lstat64("b", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
>> geteuid32()                             = 0
>> unlink("a")                             = 0
>> close(0)                                = 0
>> close(1)                                = 0
>> close(2)                                = 0
>> exit_group(0)                           = ?
>
> mv is from coreutils BTW.
> Here is the related comment from the source:
>
> "Set *UNLINK_SRC if we've determined that the caller wants to do
> `rename (a, b)' where `a' and `b' are distinct hard links to the same
> file. In that case, the caller should try to unlink `a' and then return
> successfully.  Ideally, we wouldn't have to do that, and we'd be
> able to rely on rename to remove the source file.  However, POSIX
> mistakenly requires that such a rename call do *nothing* and return
> successfully."
>
> Perhaps it could be brought up as an issue with the standards guys?

We already have.  And POSIX 2008 already acted on that.  While you 
quoted rename(2) (which was intentionally not changed), you forgot to 
also read the POSIX wording on mv(1):

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/mv.html

2. If the source_file operand and destination path name the same 
existing file, then the destination path shall not be removed, and one 
of the following shall occur:

    a. No change is made to source_file, no error occurs, and no 
diagnostic is issued.
    b. No change is made to source_file, a diagnostic is issued to 
standard error identifying the two names, and the exit status is affected.
    c. If the source_file operand and destination path name distinct 
directory entries, then the source_file operand is removed, no error 
occurs, and no diagnostic is issued.

2a is the naive approach (using rename(2) semantics)
2b is the typical mv(1) approach (IIRC, both Solaris and BSD mv 
independently implemented this mode)
2c is the GNU coreutils approach (make mv(1) do what you meant, even 
though rename(2) is _required_ to do nothing)

Do NOT change the Linux kernel to "fix" rename(2); that will only cause 
more heartache by deviating from the standard.

-- 
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org

Re: rename("a", "b") would not always remove "a" on success. ?!!

[Eric Blake]

On 10/28/2011 09:42 AM, Eric Blake wrote:
>> Perhaps it could be brought up as an issue with the standards guys?
>
> We already have. And POSIX 2008 already acted on that. While you quoted
> rename(2) (which was intentionally not changed), you forgot to also read
> the POSIX wording on mv(1):
>
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/mv.html
>
> 2. If the source_file operand and destination path name the same
> existing file, then the destination path shall not be removed, and one
> of the following shall occur:
>
> a. No change is made to source_file, no error occurs, and no diagnostic
> is issued.
> b. No change is made to source_file, a diagnostic is issued to standard
> error identifying the two names, and the exit status is affected.
> c. If the source_file operand and destination path name distinct
> directory entries, then the source_file operand is removed, no error
> occurs, and no diagnostic is issued.

Contrast this to the POSIX 2001 wording for mv:

http://pubs.opengroup.org/onlinepubs/009695399/utilities/mv.html

2. The mv utility shall perform actions equivalent to the rename() 
function defined in the System Interfaces volume of IEEE Std 1003.1-2001,...

with no escape clause for identical files.  That is, POSIX 2008 
specifically inserted a new step 2 (renumbering the step 2 from POSIX 
2001 to step 3 in POSIX 2008) at GNU Coreutils' insistence that the 
requirements were wrong.  See also 
http://www.opengroup.org/austin/aardvark/latest/xcubug2.txt, and search 
for AI-169, which was where the change in wording was made; including 
this rationale:

  Note that ERN 88 was rejected in spite of the fact that
  http://www.opengroup.org/austin/mailarchives/austin-group-l/msg05155.html
  claims the original intent was to have rename detect identical
  directory entries rather than identical files.  One of the arguments
  presented for rejecting ERN 88 was that rename(2) has the property
  of preserving link count, whereas having rename("a", "b") unlink
  "a" when the two names are hard links would not.

Unfortunately, that link no longer works, so I can't point you to the 
historical conversation at the time where it was argued that changing 
rename(2) behavior would be detrimental.  But it does give enough 
context to provide an alternate interpretation of rename() semantics 
that fit the standardized behavior:  a successful rename("a","b") 
guarantees that "b" now has the same link count as "a" had prior to the 
rename call, even if that meant a hard-linked "a" had to be left 
untouched to avoid changing the link count of hard-linked "b".

-- 
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org

Re: rename("a", "b") would not always remove "a" on success. ?!!

[Denys Vlasenko]

On Friday 28 October 2011 17:42, Eric Blake wrote:
> On 10/28/2011 09:32 AM, Pádraig Brady wrote:
> >> http://pubs.opengroup.org/onlinepubs/9699919799/functions/rename.html
> >>
> >> 'If the old argument and the new argument resolve to either .... or different
> >> directory entries for the same existing file, rename() shall return
> >> successfully and perform no other action.'
> >>
> >> It's incredible they had audacity to put such nonsense into standard.
> >>
> >> The page says in "RATIONALE" section:
> >>
> >> 'The specification that if old and new refer to the same file is
> >> intended to guarantee that:
> >>
> >> rename("x", "x");
> >>
> >> does not remove the file.'
> >>
> >> Why didn't they just explicitly say that they actually want THIS
> >> particular case to work correctly, not OTHER cases to be fucked up?!
> 
> Because it is historical precedent, and changing it now would break 
> software that has come to expect this behavior on hard links.

There is centain level of absurdity, after which fixing a goof in standards
makes sense even if it theoretically can break some existing program.

In my opinion, the key word here is "theoritically".
Is there even one real-world program which depends on this?
I bet there is not.

-- 
vda

Re: rename("a", "b") would not always remove "a" on success. ?!!

[richard -rw- weinberger]

On Sat, Oct 29, 2011 at 3:00 AM, Denys Vlasenko
<vda.linux@googlemail.com> wrote:
> In my opinion, the key word here is "theoritically".
> Is there even one real-world program which depends on this?
> I bet there is not.
>

The best way to find such a program is breaking it by changing the
kernel's behavior. ;-)

-- 
Thanks,
//richard