experchange > programmer

Frederick Gotham (11-10-19, 08:30 PM)
I have multiple programs that use a piece of hardware (/dev/caterpillar0). The hardware device does not support simultaneous accesses -- you can't have two threads (or two processes) using it at the same time.

About half a dozen processes need to use this piece of hardware near-simultaneously, and some of these processes are multi-threaded, so I am creating a minimalistic library to achieve this and it will be made available both as a static library (.a) and dynamic library (.so).

If I was doing this in MS-Windows, I'd just create a globally named mutex:

HANDLE hMtx = CreateMutex( NULL, TRUE, "hardware_caterpillar0_lock"));

At the moment in Linux, I am creating a share memory object and then putting a pthread mutex inside it, and then using the pthread API to lock and unlock it. Here is the code I'm using for this interprocess mutex:



This interprocess mutex is actually quite clean and works quite well, but I'm able to break everything by doing this:
If I get one of the processes to perform a time-consuming operation (e.g. 6seconds), and if I hit Ctrl + C on this process when it's doing this, thenit kills the process. . . and my global mutex remains locked! So now nobody can use the hardware!

Before I started using this shared mutex, I was just creating a file and using the Linux function "flock" to lock it, but the problem with this is that 'flock' gets a lock for the current process (not the current thread) which means that it won't stop simultaneous accesses for multiple threads of the same process.

How can I get a global interprocess mutex in Linux that will release the lock if the process dies? (Even better if it will release the lock when the thread dies).
Ben Kibbey (11-10-19, 09:24 PM)
On Sun, Nov 10, 2019 at 10:30:48AM -0800, Frederick Gotham wrote:
> How can I get a global interprocess mutex in Linux that will release
> the lock if the process dies? (Even better if it will release the lock
> when the thread dies).


Try using pthread_cleanup_push/pop() to be called at function
cancellation points.
Scott Lurndal (11-11-19, 04:15 PM)
Frederick Gotham <cauldwell.thomas> writes:
[..]
> seconds), and if I hit Ctrl + C on this process when it's doing this, then=
> it kills the process. . . and my global mutex remains locked! So now nobod=
>y can use the hardware!


Did you try with the robust attribute?

PTHREAD_MUTEX_ROBUST
If the process containing the owning thread of a robust mutex
terminates while holding the mutex lock, the next thread that
acquires the mutex shall be notified about the termination by
the return value [EOWNERDEAD] from the locking function. If the
owning thread of a robust mutex terminates while holding the mutex
lock, the next thread that attempts to acquire the mutex may be notified
about the termination by the return value [EOWNERDEAD]. The notified thread
can then attempt to make the state protected by the mutex consistent again,
and if successful can mark the mutex state as consistent by calling
pthread_mutex_consistent(). After a subsequent successful call to
pthread_mutex_unlock(), the mutex lock shall be released and can be
used normally by other threads. If the mutex is unlocked without a
call to pthread_mutex_consistent(), it shall be in a permanently unusable state
Rainer Weikusat (11-11-19, 04:55 PM)
Frederick Gotham <cauldwell.thomas> writes:
> I have multiple programs that use a piece of hardware
> (/dev/caterpillar0). The hardware device does not support simultaneous
> accesses -- you can't have two threads (or two processes) using it at
> the same time.
> About half a dozen processes need to use this piece of hardware
> near-simultaneously, and some of these processes are multi-threaded,
> so I am creating a minimalistic library to achieve this and it will be
> made available both as a static library (.a) and dynamic library
> (.so).


[...]

> Before I started using this shared mutex, I was just creating a file
> and using the Linux function "flock" to lock it, but the problem with
> this is that 'flock' gets a lock for the current process (not the
> current thread) which means that it won't stop simultaneous accesses
> for multiple threads of the same process.


The obvious idea would be to use a file lock for interprocess
serialization and an ordinary mutex for interthread serialization.
Frederick Gotham (11-11-19, 06:03 PM)
On Monday, November 11, 2019 at 2:55:55 PM UTC, Rainer Weikusat wrote:

> The obvious idea would be to use a file lock for interprocess
> serialization and an ordinary mutex for interthread serialization.


I need both.
Rainer Weikusat (11-11-19, 06:53 PM)
Frederick Gotham <cauldwell.thomas> writes:
> On Monday, November 11, 2019 at 2:55:55 PM UTC, Rainer Weikusat wrote:
>> The obvious idea would be to use a file lock for interprocess
>> serialization and an ordinary mutex for interthread serialization.

> I need both.


Then use both, ie, a mutex serialize access to the file lock within a
single, multi-threaded process and the file lock to serialize accesses
from different processes. That way, if a process holding the lock gets
killed, it'll be freed automatically. This won't handle the situation
where a thread terminates unexpectedly while holding the mutex but if
that's not a programming error which should rather be fixed, one could
use a POSIX semaphore instead and unlock that from some sort of
"recovery thread".
boltar (11-11-19, 09:21 PM)
On Sun, 10 Nov 2019 10:30:48 -0800 (PST)
Frederick Gotham <cauldwell.thomas> wrote:
>Before I started using this shared mutex, I was just creating a file and us=
>ing the Linux function "flock" to lock it, but the problem with this is tha=
>t 'flock' gets a lock for the current process (not the current thread) whic=
>h means that it won't stop simultaneous accesses for multiple threads of th=
>e same process.


Tbh there are far fewer reasons to use threads on linux/unix than on Windows
due to process creation being far simpler and far less costly and IPC being
far more comprehensive on *nix. If you can remove threads from your program,
especially if its already using IPC anyway then I would recommend doing so as
the whole nightmare of deadlocking and race conditions will almost vanish.

>How can I get a global interprocess mutex in Linux that will release the lo=
>ck if the process dies? (Even better if it will release the lock when the t=
>hread dies).


If signals are the problem then set up some signal handlers to unlock the
mutex before the process dies. Otherwise use the suggestions others have made.
Casper H.S. Dik (11-12-19, 11:40 AM)
Frederick Gotham <cauldwell.thomas> writes:

>On Monday, November 11, 2019 at 2:55:55 PM UTC, Rainer Weikusat wrote:


>> The obvious idea would be to use a file lock for interprocess
>> serialization and an ordinary mutex for interthread serialization.


>I need both.


Shouldn't you use a mutex inside the device driver; it would be
shared and it guarantees that killing a process would still
unlocks the mutex (when a process is killed, the code in the
kernel will finish and the process won't be killed until the
kernel is about to return to the process).

Of course, kernel mutexes should be released before returning
from the kernel.

Casper
Kaz Kylheku (11-12-19, 02:23 PM)
On 2019-11-10, Ben Kibbey <bjk> wrote:
> On Sun, Nov 10, 2019 at 10:30:48AM -0800, Frederick Gotham wrote:
>> How can I get a global interprocess mutex in Linux that will release
>> the lock if the process dies? (Even better if it will release the lock
>> when the thread dies).

> Try using pthread_cleanup_push/pop() to be called at function
> cancellation points.


Say what? That's purely local mechanism that will not complete
its action when the process dies entirely.
Kaz Kylheku (11-12-19, 02:28 PM)
On 2019-11-10, Frederick Gotham <cauldwell.thomas> wrote:
> I have multiple programs that use a piece of hardware
> (/dev/caterpillar0). The hardware device does not support simultaneous
> accesses -- you can't have two threads (or two processes) using it at
> the same time.


Then the right thing to do is to make it a "single open" device.

The functionality for providing mutual exclusion over /dev/caterpillar0
among processes belongs in exactly one place: the kernel driver for
/dev/caterpillar0.

If a process has the device already open, then an attempt by another
process to open it should yield a -1/EBUSY.
Rainer Weikusat (11-12-19, 05:11 PM)
Kaz Kylheku <493-878-3164> writes:
> On 2019-11-10, Frederick Gotham <cauldwell.thomas> wrote:
> Then the right thing to do is to make it a "single open" device.
> The functionality for providing mutual exclusion over /dev/caterpillar0
> among processes belongs in exactly one place: the kernel driver for
> /dev/caterpillar0.
> If a process has the device already open, then an attempt by another
> process to open it should yield a -1/EBUSY.


That's exactly the place where it doesn't belong: The kernel shouldn't
be enforcing a perfectly arbitrary access policy to this device, IOW,
userspace processes (and threads within processes) ought to be free to
coordinate their activities (or not coordinate them) in whichever way
they see fit. The necessary primitives for provdigig mutual exclusion
are available, they just need to be used if mutual exclusion is desired.
Kaz Kylheku (11-13-19, 12:08 AM)
On 2019-11-12, Rainer Weikusat <rweikusat> wrote:
> Kaz Kylheku <493-878-3164> writes:
> That's exactly the place where it doesn't belong: The kernel shouldn't
> be enforcing a perfectly arbitrary access policy to this device, IOW,


See original post (quoted above):

"... [t]he hardware device does not support simultaneous accesses ..."

Therefore, it's the job of the OS to prevent that from happening,
even if user space tries to.
Rainer Weikusat (11-13-19, 01:05 AM)
Kaz Kylheku <493-878-3164> writes:
> On 2019-11-12, Rainer Weikusat <rweikusat> wrote:
> See original post (quoted above):
> "... [t]he hardware device does not support simultaneous accesses ..."
> Therefore, it's the job of the OS to prevent that from happening,
> even if user space tries to.


Sorry, but that's a non-sequitur: "Access coordination is required" does
not imply "kernel must enforce a particular policy for that", that's
just one particularly inflexible way of implementing access coordination
(and - coincidentally - one pretty close to how filesystem access works
on Windows). Eg, it's perfectly conceivable that only one special
application will ever attempt to use the device at any given time. In
this case, access synchronizatio is not needed at all (except for
"beancounter ease-of-mind", but that's not a technical problem :-).

If a given problem can be solved without pushing a specific approach for
doing so into the kernel (true for this problem), it shouldn't be pushed
into the kernel.
Ben Kibbey (11-13-19, 06:00 AM)
On Tue, Nov 12, 2019 at 12:23:46PM +0000, Kaz Kylheku wrote:
> On 2019-11-10, Ben Kibbey <bjk> wrote:
> Say what? That's purely local mechanism that will not complete
> its action when the process dies entirely.


Depending how it dies. Are you exepecting it to die in uncatchable ways?
How will your thread die?
Rainer Weikusat (11-13-19, 04:34 PM)
Rainer Weikusat <rweikusat> writes:

> Kaz Kylheku <493-878-3164> writes:
> Sorry, but that's a non-sequitur: "Access coordination is required" does
> not imply "kernel must enforce a particular policy for that",


Especially as the kernel can't even do that, at least not based on
restricting device opens: Once a file descriptor has been acquired, any
number of applications can use that for anything, including creating any
number (subject to resource limits) of additional file descriptors
referring to the same device.

Similar Threads