All of lore.kernel.org
 help / color / mirror / Atom feed
From: Sitsofe Wheeler <sitsofe@gmail.com>
To: Rob Scheepens <rob.scheepens@nutanix.com>
Cc: David Knierim <knierim@nutanix.com>,
	Rebecca Cran <rebecca@bluestop.org>, fio <fio@vger.kernel.org>,
	Jens Axboe <axboe@kernel.dk>
Subject: Re: Windows: FIO randomly hangs using attached script
Date: Thu, 8 Mar 2018 15:46:21 +0000	[thread overview]
Message-ID: <CALjAwxirkjduZof+F3prrKX0wp9YS8apS-YdK8+ru1supg1dnA@mail.gmail.com> (raw)
In-Reply-To: <CALjAwxgF-2dK+Bc9iwkVsxA4sfWKYSmHe8k3YbpUyDmBWP34ow@mail.gmail.com>

(I'm going to bottom post for now)

On 8 March 2018 at 15:13, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> OK wow that was big.
>
> The threads from 3 - 101 seem to be stuck with a backtrace that looks
> similar to this:
>
> Thread 3 (Thread 3784.0xd38):
> #0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from
> C:\Windows\SYSTEM32\ntdll.dll
> #1  0x00007ff842633acf in WaitForSingleObjectEx () from
> C:\Windows\System32\KernelBase.dll
> #2  0x000000000046c0df in pthread_mutex_lock_intern
> (timeout=4294967295, m=0x1e1880)
>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
> #3  pthread_mutex_lock (m=m@entry=0x1b0000) at
> /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
> #4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
> #5  0x0000000000446eae in sfree_pool (ptr=0x2bc6060, pool=0x55c980
> <mp>) at smalloc.c:312
> #6  sfree (ptr=0x2bc6070) at smalloc.c:336
> #7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb0d2960)
> at filesetup.c:1358
> #8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
> #9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
> #10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from
> C:\Windows\System32\msvcrt.dll
> #11 0x00007ff845a9b38c in msvcrt!_endthreadex () from
> C:\Windows\System32\msvcrt.dll
> #12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
> C:\Windows\System32\kernel32.dll
> #13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
> C:\Windows\SYSTEM32\ntdll.dll
> #14 0x0000000000000000 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
>
> From thread 102 - 201 things look like this:
>
> Thread 102 (Thread 3784.0x8d4):
> #0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from
> C:\Windows\SYSTEM32\ntdll.dll
> #1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from
> C:\Windows\System32\KernelBase.dll
> #2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0xb001bd0)
> at engines/windowsaio.c:427
> #3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
> C:\Windows\System32\kernel32.dll
> #4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
> C:\Windows\SYSTEM32\ntdll.dll
> #5  0x0000000000000000 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
>
> For completeness here's Thread 1 and 2:
>
> Thread 2 (Thread 3784.0x1574):
> #0  0x00007ff845d06bf4 in ntdll!ZwWaitForMultipleObjects () from
> C:\Windows\SYSTEM32\ntdll.dll
> #1  0x00007ff84264796f in WaitForMultipleObjectsEx () from
> C:\Windows\System32\KernelBase.dll
> #2  0x00007ff84264786e in WaitForMultipleObjects () from
> C:\Windows\System32\KernelBase.dll
> #3  0x000000000046b31f in do_sema_b_wait_intern
> (sema=sema@entry=0x8cc, nointerrupt=nointerrupt@entry=0,
>     timeout=timeout@entry=500) at
> /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:625
> #4  0x000000000046b60e in do_sema_b_wait (sema=0x8cc, nointerrupt=0,
> timeout=500, cs=0xafc9a60, val=0xafc9a88)
>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:584
> #5  0x000000000046be8e in pthread_cond_timedwait_impl (c=c@entry=0x2c2d0f8,
>     external_mutex=external_mutex@entry=0x2c2d0f0,
> t=t@entry=0x1ce9fe30, rel=rel@entry=0)
>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:504
> #6  0x000000000046bea8 in pthread_cond_timedwait (c=c@entry=0x2c2d0f8,
> m=m@entry=0x2c2d0f0, t=t@entry=0x1ce9fe30)
>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:515
> #7  0x0000000000466978 in helper_thread_main (data=0x2c2d0d0) at
> helper_thread.c:95
> #8  0x000000000046df74 in pthread_create_wrapper (args=0x0)
>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
> #9  0x00007ff845a9b2ba in msvcrt!_beginthreadex () from
> C:\Windows\System32\msvcrt.dll
> #10 0x00007ff845a9b38c in msvcrt!_endthreadex () from
> C:\Windows\System32\msvcrt.dll
> #11 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
> C:\Windows\System32\kernel32.dll
> #12 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
> C:\Windows\SYSTEM32\ntdll.dll
> #13 0x0000000000000000 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
>
> Thread 1 (Thread 3784.0x16fc):
> #0  0x00007ff845d06724 in ntdll!ZwDelayExecution () from
> C:\Windows\SYSTEM32\ntdll.dll
> #1  0x00007ff842647b47 in SleepEx () from C:\Windows\System32\KernelBase.dll
> #2  0x0000000000469fa8 in usleep (useconds=<optimized out>,
> useconds@entry=10000) at os/windows/posix.c:572
> #3  0x000000000045d03a in do_usleep (usecs=10000) at backend.c:2129
> #4  run_threads (sk_out=sk_out@entry=0x0) at backend.c:2444
> #5  0x000000000045d3da in fio_backend (sk_out=sk_out@entry=0x0) at
> backend.c:2493
> #6  0x0000000000480329 in main (argc=4, argv=0x1e13c0, envp=<optimized
> out>) at fio.c:65
>
> This could take a while to figure out!

Here a thread in among the others that looks different:

Thread 53 (Thread 3784.0xca0):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from
C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from
C:\Windows\System32\KernelBase.dll
#2  0x000000000046b4cc in do_sema_b_wait_intern (sema=sema@entry=0xa8,
nointerrupt=nointerrupt@entry=1,
    timeout=timeout@entry=4294967295) at
/usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:601
#3  0x000000000046b60e in do_sema_b_wait (sema=0xa8, nointerrupt=1,
timeout=4294967295, cs=0x1e15f0, val=0x1e1618)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:584
#4  0x000000000046b89f in pthread_cond_signal (c=<optimized out>)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:327
#5  0x00000000004412c0 in fio_mutex_up (mutex=<optimized out>) at mutex.c:246
#6  0x0000000000446feb in sfree_pool (ptr=<optimized out>,
pool=<optimized out>) at smalloc.c:317
#7  sfree (ptr=<optimized out>) at smalloc.c:336
#8  0x000000000042df59 in close_and_free_files (td=td@entry=0xb4e02a0)
at filesetup.c:1358
#9  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#10 0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#11 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from
C:\Windows\System32\msvcrt.dll
#12 0x00007ff845a9b38c in msvcrt!_endthreadex () from
C:\Windows\System32\msvcrt.dll
#13 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
C:\Windows\System32\kernel32.dll
#14 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
C:\Windows\SYSTEM32\ntdll.dll
#15 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

-- 
Sitsofe | http://sucs.org/~sits/


  parent reply	other threads:[~2018-03-08 15:46 UTC|newest]

Thread overview: 31+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-03-06  1:23 Windows: FIO randomly hangs using attached script Rebecca Cran
2018-03-06 16:35 ` Sitsofe Wheeler
2018-03-06 20:01   ` Rebecca Cran
2018-03-07  6:00     ` Sitsofe Wheeler
2018-03-07 15:33       ` David Knierim
2018-03-07 15:39         ` Rob Scheepens
2018-03-07 16:01           ` Sitsofe Wheeler
2018-03-07 16:03             ` Rebecca Cran
2018-03-07 16:05             ` Rob Scheepens
2018-03-07 16:09               ` Rebecca Cran
2018-03-08 10:51           ` Rob Scheepens
2018-03-08 12:28             ` Sitsofe Wheeler
2018-03-08 12:39               ` Rob Scheepens
2018-03-08 14:35                 ` Rob Scheepens
2018-03-08 14:38                   ` Rob Scheepens
2018-03-08 15:15                     ` Sitsofe Wheeler
2018-03-08 15:13                   ` Sitsofe Wheeler
2018-03-08 15:45                     ` Rob Scheepens
2018-03-08 15:47                       ` Sitsofe Wheeler
2018-03-08 15:46                     ` Sitsofe Wheeler [this message]
2018-03-08 15:59                       ` Sitsofe Wheeler
2018-03-08 16:18                         ` Jens Axboe
2018-03-09 14:40                           ` Sitsofe Wheeler
2018-03-09 14:55                             ` Jens Axboe
2018-03-16  8:13                               ` Sitsofe Wheeler
2018-05-01 14:41                                 ` Rob Scheepens
2018-05-01 14:42                                   ` Jens Axboe
2018-05-01 14:58                                   ` Sitsofe Wheeler
2018-03-08 22:44                         ` Sitsofe Wheeler
2018-03-06 21:53 ` Jens Axboe
2018-03-06 22:35   ` Rebecca Cran

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CALjAwxirkjduZof+F3prrKX0wp9YS8apS-YdK8+ru1supg1dnA@mail.gmail.com \
    --to=sitsofe@gmail.com \
    --cc=axboe@kernel.dk \
    --cc=fio@vger.kernel.org \
    --cc=knierim@nutanix.com \
    --cc=rebecca@bluestop.org \
    --cc=rob.scheepens@nutanix.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.