Circular debugging using ptrace results in deadlock due to race condition?

From: Ilja Nevolin <Ilja.Nevolin@UGent.be>
To: linux-kernel@vger.kernel.org
Subject: Circular debugging using ptrace results in deadlock due to race condition?
Date: Thu, 08 Jun 2017 16:08:54 +0200	[thread overview]
Message-ID: <20170608160854.Horde.wUKaUVyqiOZZOVp2xD1nl5A@webmail.ugent.be> (raw)

Hi guys,

As part of my master's thesis I am facing a challenging problem.
I am trying to let two processes be each other's debuggers using the  
ptrace syscall. However, my proof-of-concept implementation always  
results in a deadlock state (both processes get stuck in 't+' state as  
shown by 'ps aux').

Here is my code, it's pretty simple:
https://pastebin.com/A1iBA3nh

I have compiled and run this on an ARMv7 developer board with kernel  
version 3.0.35 (Linaro 13.08).
The output of the above code is this:

   A    waiting to continue...
   B    attachTo: 0
   B    setOptions: 0
   B    setVarData: 0
   B    setVarData: 0
   B    cont: 0
   B    waiting to continue...
   B    waiting to continue...
   A    attachTo: 0

As you can see it never reaches the "finished" printf code, and gets  
stuck as soon as the other process attempts to attach to the debugger.

I have done a similar experiment for 3 processes, such that each one  
attempts to attach to the other in a circular fashion: A -> B -> C -> A
The result in this case was exactly the same. However, here I was able  
to detect a race condition, because sometimes the code executed  
properly without getting stuck in a deadlock (but it's hard to  
reproduce).

If you wish, you can test this by using a lightweight debugger I've  
developed and three console terminals. Here's the code:
https://pastebin.com/fPJb8ZNb

Once you've compiled the above code, you simply run the binary on each  
console and enter the PID of another process to establish a 3-way  
circle.

I am far from an expert on the kernel, but I did have a look at the  
ARM specific kernel implementation which left me puzzled. I couldn't  
find where/how/why this code does not work.
Now I'm wondering if it's possible at all to make this work without a  
deadlock occurring? Does anyone have any experience with this, or can  
provide some clues/feedback?

Thank you greatly for your time, attention and effort!
Ilya Nevolin
ilja.nevolin@ugent.be