Hi,

as discussed in [1] it turned out to be a ipc-sem issue.

[ TEST-CASE / USE-CASE (reproducible) ]

Build a Linux-Next (next-20130419) kernel with 4 parallel-make-jobs
(see scripts-dir in unpacked tarball).

[ TRYOUTS ]

I tried to switch to "stable" kernel-config options (disable
intel_pstate, full-dynticks-cputime, etc.) and/or switch from slub to
slab for example.

Rik pointed to eventually missing ipc-sem patches and wanted to check
his INBOX after the conference he his visiting.
Then Davidlohr pointed to three missing ipc-sem patches [3] which did
not help here (might be some patches still missing?).

Linus helped to trigger down the problem with his "crazy rcu
double-free debug hack" [4].
Paul helped with some aid regarding towards RCU debug kconfig
(CONFIG_DEBUG_OBJECTS_RCU_HEAD=y).

Emmanuel reported same problems especially with gcc-builds (see my test-case).

( What I could not test: Turn off APPARMOR and SELINUX in
Ubuntu/precise x86-64. )

All these taken actions did not help.

[ TEMPORARY WORKAROUND ]

      Revert "ipc/sem.c: alternatives to preempt_disable()"
      Revert "ipc, sem: prevent possible deadlock"
      Revert "fix for sem_lock"
      Revert "ipc,sem: fine grained locking for semtimedop"
      Revert "ipc,sem: have only one list in struct sem_queue"
      Revert "ipcsem-open-code-and-rename-sem_lock-fix"
      Revert "ipc,sem: open code and rename sem_lock"
      Revert "ipcsem-do-not-hold-ipc-lock-more-than-necessary-fix-checkpatch-fixes"
      Revert "ipc,sem: prevent releasing RCU read lock twice in semctl_main"
      Revert "ipc,sem: do not hold ipc lock more than necessary"

NOTE: I have rebuilt 3 times successfully my latest kernel with these
revert-patchset!

As some developers attend a conference ATM, I leave this up to you experts.

Thanks to all folks with their feedback, help and patches!

Hope this pre-analyze helps a bit.

Regards,
- Sedat -

P.S.: All relevant material collected in the attached tarball.

[1] http://marc.info/?t=136631457900005&r=1&w=2
[2] http://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/log/ipc/sem.c?id=next-20130419
[3] http://marc.info/?l=linux-next&m=136640233601913&w=2
[4] http://marc.info/?l=linux-next&m=136640665503325&w=2
[5] https://lkml.org/lkml/2013/3/30/1