fcntl file locking and pthreads

fcntl file locking and pthreads

[Mikhail Kruk]

Hi,
on 2.4 kernels fcntl-based file locking does not work with 
clone-based threads as expected (by me): two threads of the same process 
can acquire exclusive lock on a file at the same time.
flock()-based locks work as expected, i.e. only one thread can have an 
exclusive lock at a time.
What would it take to make fcntl work as flock?

Please cc me, I'm not subscribed.
Thanks

Re: fcntl file locking and pthreads

[Jamie Lokier]

Mikhail Kruk wrote:
> on 2.4 kernels fcntl-based file locking does not work with 
> clone-based threads as expected (by me): two threads of the same process 
> can acquire exclusive lock on a file at the same time.
> flock()-based locks work as expected, i.e. only one thread can have an 
> exclusive lock at a time.

Is this true even when _not_ setting CLONE_FILES?

cheers,
-- Jamie

Re: fcntl file locking and pthreads

[Mikhail Kruk]

> Mikhail Kruk wrote:
> > on 2.4 kernels fcntl-based file locking does not work with 
> > clone-based threads as expected (by me): two threads of the same process 
> > can acquire exclusive lock on a file at the same time.
> > flock()-based locks work as expected, i.e. only one thread can have an 
> > exclusive lock at a time.
> 
> Is this true even when _not_ setting CLONE_FILES?

CLONE_FILES is an argument to clone(), I'm using pthreads and I don't 
know if LinuxThreads implementation of pthreads gives me control of 
how clone is called. Anyway, if I understand what CLONE_FILES does, 
it should be given to clone, because threads do have to be able to share file 
descriptors, probably. But not the locks!

Re: fcntl file locking and pthreads

[Miquel van Smoorenburg]

In article <Pine.LNX.4.33.0305040922020.32427-100000@iole.cs.brandeis.edu>,
Mikhail Kruk  <meshko@cs.brandeis.edu> wrote:
>CLONE_FILES is an argument to clone(), I'm using pthreads and I don't 
>know if LinuxThreads implementation of pthreads gives me control of 
>how clone is called. Anyway, if I understand what CLONE_FILES does, 
>it should be given to clone, because threads do have to be able to share file 
>descriptors, probably. But not the locks!

Are you sure. I think threads /of a process/ have to share the
locks, under Linux that would translate to a thread-group.

Mike.

RE: fcntl file locking and pthreads

[David Schwartz]

> CLONE_FILES is an argument to clone(), I'm using pthreads and I don't
> know if LinuxThreads implementation of pthreads gives me control of
> how clone is called. Anyway, if I understand what CLONE_FILES does,
> it should be given to clone, because threads do have to be able
> to share file
> descriptors, probably. But not the locks!

	What if I have an application where requests are written to files. Thread A
comes along and notices a job in a file, so it locks the file and reads the
job. The job will require some network I/O, so the thread goes on to do
other things. Later on, thread B notices the network I/O has completed, so
it needs to write to the file, release the lock, and close the file.

	Making locks local to specific threads is contrary to the whole purpose of
threads. Work that needs to be done on a file descriptor is supposed to be
doable by any thread.

	This is something that can be solved with application code. Just keep a
table of all your file locks and consult the table before attempting to
acquire a lock from the system. This is no different than the housekeeping
required around 'malloc' (or more accurately 'sbrk').

	DS

RE: fcntl file locking and pthreads

[Mikhail Kruk]

> > CLONE_FILES is an argument to clone(), I'm using pthreads and I don't
> > know if LinuxThreads implementation of pthreads gives me control of
> > how clone is called. Anyway, if I understand what CLONE_FILES does,
> > it should be given to clone, because threads do have to be able
> > to share file
> > descriptors, probably. But not the locks!
> 
> 	What if I have an application where requests are written to files. Thread A
> comes along and notices a job in a file, so it locks the file and reads the
> job. The job will require some network I/O, so the thread goes on to do
> other things. Later on, thread B notices the network I/O has completed, so
> it needs to write to the file, release the lock, and close the file.

I am not persuaded by this example. Why didn't thread A close the file 
when it finished the network I/O? That would be logical time to do it. If 
it wasn't a file descriptor, but a shared memory region, would you argue 
the same about a mutex protecting that memory region?
I think this should not be a question of personal opinions or specific 
examples. It should just be consistent. Two reference platforms for 
threads are Solaris and Windows. I don't know how Solaris handles this, 
but on Windows file locks are per thread, not per process.

[Please cc]

RE: fcntl file locking and pthreads

[David Schwartz]

> > > CLONE_FILES is an argument to clone(), I'm using pthreads and I don't
> > > know if LinuxThreads implementation of pthreads gives me control of
> > > how clone is called. Anyway, if I understand what CLONE_FILES does,
> > > it should be given to clone, because threads do have to be able
> > > to share file
> > > descriptors, probably. But not the locks!

> > 	What if I have an application where requests are written to
> > files. Thread A
> > comes along and notices a job in a file, so it locks the file
> > and reads the
> > job. The job will require some network I/O, so the thread goes on to do
> > other things. Later on, thread B notices the network I/O has
> > completed, so
> > it needs to write to the file, release the lock, and close the file.

> I am not persuaded by this example. Why didn't thread A close the file
> when it finished the network I/O? That would be logical time to do it.

	That would release the process' lock on the file descriptor, but the
process is not done with the file descriptor. Surely you're not seriously
suggesting that, say, a multithreaded web server should open/close every
file each time it needs to read some data from it rather than holding the
descriptor open until it's done with it.

> If
> it wasn't a file descriptor, but a shared memory region, would you argue
> the same about a mutex protecting that memory region?

	Mutexes *are* thread resources. They're specifically used to provide
synchronization between threads. However, file descriptors are process
resources.

> I think this should not be a question of personal opinions or specific
> examples. It should just be consistent.

	Yes. File descriptors are process resources, thus everything about them
should be a process resource. The locks on a file are no different from the
file pointer.

> Two reference platforms for
> threads are Solaris and Windows. I don't know how Solaris handles this,
> but on Windows file locks are per thread, not per process.

	Surely your argument isn't that UNIX should do things a certain way because
that's how Windows does it? We can talk about two things, how things are and
how they should be. This discussion seemed to be about how things should be.
And file descriptors and the stuff associated with them should be process
resources.

	However, I think there's a simple fix to this problem. Associate the file
locks with the particular file descriptor. One can argue that the current
scheme (where closing a file descriptor releases locks associated with
another file descriptor for the same file) is as crazy as having two threads
each open the same file and wind up sharing a file pointer.

	This will allow threads to share file locks by sharing file descriptors.
However, it will not create subtle dependencies between code blocks that
happen to open/lock the same file because they'll use their own file
descriptors.

	Share a file descriptor, share locks. Open the file yourself, you have your
own locks.

	That's what makes sense.

	DS

RE: fcntl file locking and pthreads

[Mikhail Kruk]

> > Two reference platforms for
> > threads are Solaris and Windows. I don't know how Solaris handles this,
> > but on Windows file locks are per thread, not per process.
> 
> 	Surely your argument isn't that UNIX should do things a certain way because
> that's how Windows does it? We can talk about two things, how things are and
> how they should be. This discussion seemed to be about how things should be.

I mentioned Windows because it happens to have a very mature threading 
implementation and I don't see why Unix can't look at something from 
Windows as a reference.
Anyway, I understand your argument about thread vs processes resources. I 
also checked that it works the same way on Solaris (which of course is a 
better reference point for Linux), so I agree that fcntl behavior is ok. 
Thanks for your explanation!

Re: fcntl file locking and pthreads

[Mark Mielke]

On Sun, May 04, 2003 at 02:13:25AM -0400, Mikhail Kruk wrote:
> on 2.4 kernels fcntl-based file locking does not work with 
> clone-based threads as expected (by me): two threads of the same process 
> can acquire exclusive lock on a file at the same time.
> flock()-based locks work as expected, i.e. only one thread can have an 
> exclusive lock at a time.
> What would it take to make fcntl work as flock?

I don't think per-thread locks is entirely reasonable. The file descriptor
is shared between threads, which means that attributes (including locks)
attached to the file descriptor, are shared between threads.

I would suggest that system resources such as advisory locks be considered
per process, and inter-thread synchronization be performed using thread
synchronization primitives such as the mutex. Feel free to quote from POSIX
to tell me that this suggestion is wrong.

mark

-- 
mark@mielke.cc/markm@ncf.ca/markm@nortelnetworks.com __________________________
.  .  _  ._  . .   .__    .  . ._. .__ .   . . .__  | Neighbourhood Coder
|\/| |_| |_| |/    |_     |\/|  |  |_  |   |/  |_   | 
|  | | | | \ | \   |__ .  |  | .|. |__ |__ | \ |__  | Ottawa, Ontario, Canada

  One ring to rule them all, one ring to find them, one ring to bring them all
                       and in the darkness bind them...

                           http://mark.mielke.cc/

Re: fcntl file locking and pthreads

[Ville Voutilainen]

>flock()-based locks work as expected, i.e. only one thread can have an
>exclusive lock at a time.
>What would it take to make fcntl work as flock?

This is because flock locks the inode, not the file descriptor.
fcntl locks the file descriptor. This is why flock does not
work over nfs, I suppose. If you share descriptors, you also
share locks and positions (at least according to the man pages
of fcntl, dup et al). I don't know what exactly is supposed
to happen if you open the file twice in two separate threads
and then lock with fcntl in the first thread. But simply doing

open()
pthread_create()
fcntl()
			(other thread)
			fcntl()
will probably result in both threads acquiring the lock
successfully. It would be reasonable IMHO to assume that
a sequence like

open()
pthread_create()
fcntl()

			(other thread)
			open()
			fnctl() /* lock the newly opened fd */

would give you what you're after. The only problem being that
even user space manuals suggest that fcntl can only detect
that other *processes* hold a file lock. Given the muddy
nature of what is a thread/process in Linux, this requires
someone more familiar with the clone stuff to clarify.

Another issue altogether is why you are trying to sync two
threads with file locks, but I digress.

-VJV-

Re: fcntl file locking and pthreads

[Mikhail Kruk]

> open()
> pthread_create()
> fcntl()
> 			(other thread)
> 			fcntl()
> will probably result in both threads acquiring the lock
> successfully. It would be reasonable IMHO to assume that
> a sequence like
> 
> open()
> pthread_create()
> fcntl()
> 
> 			(other thread)
> 			open()
> 			fnctl() /* lock the newly opened fd */

actually what I have is:
thread 0:
pthread_create() (1)
pthread_create() (2)

thread 1:
open()
fcntl()

thread 2:
open()
fcntl()

etc

and they all succeed. Even though the file descriptors are different. 

> would give you what you're after. The only problem being that
> even user space manuals suggest that fcntl can only detect
> that other *processes* hold a file lock. Given the muddy

Unfortunately man pages I have don't even mention thread vs process.

> nature of what is a thread/process in Linux, this requires
> someone more familiar with the clone stuff to clarify.
> 
> Another issue altogether is why you are trying to sync two
> threads with file locks, but I digress.

You digress, but I feel like I have to justify myself now :)
Those threads used to be processes and now want be threads with minimal 
modifications. The files that are locked are still used by other processes 
too.

Re: fcntl file locking and pthreads

[Ville Voutilainen]

> actually what I have is:
> thread 0:
> pthread_create() (1)
> pthread_create() (2)
> thread 1:
> open()
> fcntl()
> thread 2:
> open()
> fcntl()
> and they all succeed. Even though the file descriptors are different. 

Which shows us that fcntl cannot detect a file lock held by
the current process, no matter how many times we open the same
file in order to get different file descriptors. Bloody POSIX,
might I say. :) I wonder if POSIX even says anything non-vague
wrt. to this. As in, if Linux would enable detecting
two locks on two different fds (pointing to the same file)
within one process, would Linux violate POSIX? Moreover, is
this simply so that kernel detects the situation just fine,
but the info is not carried to user space because good ole
fcntl behaviour doesn't really take threads into account?
In the marvellous scheme of things (user app -> glibc -> kernel
and back), at what point (if any) is it possible to detect
multiple locks within one process? And can it be done so that
fcntl can still be claimed to be even remotely compatible
with anything on the planet?

> You digress, but I feel like I have to justify myself now :)
> Those threads used to be processes and now want be threads with minimal 
> modifications. The files that are locked are still used by other processes 
> too.

Then, logically (I suppose you already know this, but..) you
need both the file locks and mutexes to sync with both other
threads and other processes. There, one more example of how
going multi-threaded sometimes adds (mostly locking) code.

I'll shut up now, we are only at the borderline of kernel-land
and if we aim for portability, we are probably solely and
firmly on the user-land side of the big picture.

-VJV-