Re: [PATCH] do_open(): Fix O_DIRECTORY | O_CREAT behavior

[Pedro Falcato <pedro.falcato@gmail.com>]

On Mon, Mar 20, 2023 at 5:14 PM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> On Mon, Mar 20, 2023 at 4:52 AM Christian Brauner <brauner@kernel.org> wrote:
> >
> > So before we continue down that road should we maybe treat this as a
> > chance to fix the old bug? Because this behavior of returning -ENOTDIR
> > has existed ever since v5.7 now. Since that time we had three LTS
> > releases all returning ENOTDIR even if the file was created.
>
> Ack.
>
> I think considering that the return value has been broken for so long,
> I think we can pretty much assume that there are no actual users of
> it, and we might as well clean up the semantics properly.
>
> Willing to send that patch in and we'll get it tested in the crucible
> of the real world?

That sounds good to me, I can do that. What kind of new semantics are
we talking about here?

I originally found this when testing every kind of possibly-odd edge
case in path
walking/open(2). From the systems I've tested on (not too diverse, basically
NetBSD, FreeBSD, Linux 6.2.2, and now, when looking for a regression,
a variety of kernels since 2009), 4 behaviors occurred:

1) Pre v5.7 Linux did the open-dir-if-exists-else-create-regular-file
we all know and """love""".
2) Post 5.7, we started returning this buggy -ENOTDIR error, even when
successfully creating a file.
3) NetBSD just straight up returns EINVAL on open(O_DIRECTORY | O_CREAT)
4) FreeBSD's open(O_CREAT | O_DIRECTORY) succeeds if the file exists
and is a directory. Fails with -ENOENT if it falls onto the "O_CREAT"
path (i.e it doesn't try to create the file at all, just ENOENT's;
this changed relatively recently, in 2015)

Note that all of these behaviors are allowed by POSIX (in fact, I
would not call the old Linux behavior a *bug*, just really odd
semantics).

So, again, what kind of behavior change do we want here? IMO, the best
and least destructive would probably
be to emulate FreeBSD behavior here. I don't think open(O_DIRECTORY |
O_CREAT) returning -ENOTDIR
if it doesn't exist (as Christian suggested, I think?) makes any sort
of sense here.

Just my 2c.

-- 
Pedro