Windows: FIO randomly hangs using attached script

Windows: FIO randomly hangs using attached script

[Rebecca Cran]

[-- Attachment #1: Type: text/plain, Size: 884 bytes --]

I've had a report that FIO on Windows (at least Server 2012 R2 and 2016) 
hangs when the attached script is run. The point at which it hangs is 
apparently random, and within the condvar (pthread-win32) calls.

I've replicated the hang, but I don't have time to debug it so I was 
hoping somebody on this mailing list might have time to dig into it and 
figure out what's wrong.

The people I was talking to thought it might be due to FIO linking to 
msvcrt.dll, which is old and not supposed to be used by applications - 
they should use the CRT distributed with Visual C++, such as 
msvcr120.dll instead. However, it appears that fixing this would take 
quite a lot of work since while FIO is relatively straightforward to 
change, pthread-win32 hasn't had a full release since 2012 and doesn't 
build under a current msys environment due to duplicated symbols etc.


-- 

Rebecca


[-- Attachment #2: fiohangscript.py --]
[-- Type: text/plain, Size: 15407 bytes --]

#!/usr/bin/env python
#
# Storage Performance Profiler
# ----------------------------
#  This framework facilitates running workloads with FIO.
#  It also organises the results in easy-to-parse '.dat' files
#  for later plotting with gnuplot.
#
# Author: Felipe Franciosi <felipe@nutanix.com>
#

from ast import literal_eval
from ConfigParser import ConfigParser
from optparse import OptionParser
from os import unlink, path
from subprocess import Popen, PIPE
from sys import exit, stdout
from tempfile import NamedTemporaryFile
import datetime

# dabe

    #full_filename = ''

    #drivelist = [ '\\\.\PhysicalDrive1', '\\\.\PhysicalDrive2', '\\\.\PhysicalDrive3:\\\.\PhysicalDrive4', '\\\.\PhysicalDrive5:\\\.\PhysicalDrive6', '\\\.\PhysicalDrive7:\\\.\PhysicalDrive8' ]

    #for filename in drivelist:

      #if len(full_filename) > 1:

        #full_filename = full_filename + ':' + filename

      #else:

        #full_filename = filename

      #print full_filename

#

 

defaults = {

  # @filename: Name of device or file to profile

  #            Multiple files/devices can be specified with a ':' separator

  "filename": "\\\.\PhysicalDrive1",

 

  # @size:     Amount of data to read/write from the start of @filename(s)

  "size": "20G",

 

  # @minjob:   Starting number of I/O threads

  "minjob": "1",

 

  # @maxjob:   Maximum number of I/O threads

  "maxjob": "128",

 

  # @muljob:   Increment the number of I/O threads in multiples of @muljob.

  #            The number of threads always start with @minjob and is never

  #            higher than @maxjob. Example:

  #            minjob=1, maxjob=10, muljob=4 generates {1, 4, 8}

  #            minjob=3, maxjob=12, muljob=4 generates {3, 4, 8, 12}

  "muljob": "2",    # Increment JOBS in multiples of (eg. 1, 4, 8)

 

  # @mineqd:   Starting effective queue depth (qd x numjob)

  "mineqd": "1",

 

  # @maxeqd:   Maximum effective queue depth

  "maxeqd": "128",

 

  # @muleqd:   Increment the effective queue depth in multiples of @muleqd.

  #            This takes into consideration @muljob. If a muleqd is not a

  #            multiple of numjob x qd, the profiler will round up and use

  #            the next closest option, respecting numjob first. Examples:

  #            minjob=1, maxjob=4, muljob=2, mineqd=1, maxeqd=64, muleqd=32

  #             {efd=1 (qd=1,nj=1), efd=32 (qd=32,nj=1), efd=64 (qd=64,nj=1)

  #             {efd=2 (qd=1,nj=2), efd=32 (qd=16,nj=2), efd=64 (qd=32,nj=2)

  #             {efd=4 (qd=1,nj=4), efd=32 (qd=8,nj=4),  efd=64 (qd=16,nj=4)

  #            Note: "qd" is per thread.

  "muleqd": "1",   # Increment QD in multiples of (eg. 1, 64, 128)

 

  # @minbsz:   Minimum block size (values are always in bytes)

  "minbsz": "4096",

 

  # @maxbsz:   Maxmium block size (values are always in bytes)

  #            Note: block size is always incremented in powers of two

  "maxbsz": "1048576",

 

  # @runtime:  Runtime for each spec, always in seconds

  "runtime": "20",

 

  # @dirs:     Comma-separated list of directions. Each direction must be

  #            specified in quotes. Valid directions:

  #            "read"      Sequential read

  #            "write"     Sequential write

  #            "randread"  Random reads

  #            "randwrite" Random writes

  #            "rw"        Mixed sequential reads and writes (50/50)

  #            "randrw"    Mixed random reads and writes (50/50)

  "dirs": '"randread"',

 

  # @outdat:   Filename to write plottable text data with job results

  "outdat": "test.dat",

}

 

dirs = [

  "read",

  "write",

  "randread",

  "randwrite",

  "rw",        # or readwrite

  "randrw"

]

 

class ProfilerSpec(object):

  def __init__(self, conf):

    assert(conf.has_key('filename'))

    assert(conf.has_key('size'))

    assert(conf.has_key('runtime'))

    assert(conf.has_key('numjobs'))

    assert(conf.has_key('iodepth'))

    assert(conf.has_key('bs'))

    assert(conf.has_key('dir'))

    self.conf = conf

    self.spec = None

    self.data = {}

 

  def createSpec(self):

    fio_spec = """[global]

ioengine=windowsaio

direct=1

time_based

group_reporting

size={size}

runtime={runtime}

numjobs={numjobs}

iodepth={iodepth}

bs={bs}

rw={dir}

[job]

filename={filename}""".format(**self.conf)

 

    try:

      self.spec = NamedTemporaryFile(delete=False)

      self.spec.write(fio_spec)

      self.spec.flush()

    except:

      if self.spec:

        unlink(self.spec.name)

      raise

 

  def run(self):

    assert(self.spec)

    cmd = ["fio", self.spec.name, "--minimal", "--terse-version=3"]

    proc = Popen(cmd, stdout=PIPE, stderr=PIPE)

    out, err = proc.communicate()

 

    if err:

      raise Exception(err)

 

    res = out.split(';')

    self.data['read_bw']    = int(res[6])

    self.data['read_iops']  = int(res[7])

    self.data['write_bw']   = int(res[47])

    self.data['write_iops'] = int(res[48])

 

  def cleanup(self):

    assert(self.spec)

    try:

      unlink(self.spec.name)

    except:

      pass

    finally:

      self.spec = None

 

class ProfilerJob(object):

 def __init__(self, name, conf):

    assert(name)

    assert(conf)

    self.name = name

    self.conf = conf

    self.specs = []

    self.outdatfp = None

 

  def append(self, spec):

    self.specs.append(spec)

 

  def run(self):

    assert(len(self.specs) > 0)

    print "* Running job: '%s' (%d secs / spec)" % (self.name,

                                                    int(self.conf['runtime']))

    i=1

    for spec in self.specs:

      if i > 1:

        stdout.write("\033[F")

        stdout.flush()

      now = datetime.datetime.now()

      print "** Executing spec %d/%d at %s" % (i, len(self.specs), now)

      spec.createSpec()

      try:

        spec.run()

      except:

        raise

      finally:

        spec.cleanup()

      i = i + 1

 

  def writeDataFile(self):

    assert(self.conf['outdat'])

 

    self.outdatfp = open(self.conf['outdat'], 'w')

    self.__writeDataFile()

 

  def __writeDataFile(self):

    data = """# FIO Results for "{filename}" (size={size})

# QD   : {mineqd} -> {maxeqd} in multiples of: {muleqd}

# JOBS : {minjob} -> {maxjob} in multiples of: {muljob}

# BS   : {minbsz} -> {maxbsz} in powers of two

""".format(**self.conf)

 

    for dir in dirs:

      data = data + """

# %s:

# Eff.QD  Jobs   QD  blocksz  IOPS_rd  IOPS_wr   KB/s_rd   KB/s_wr

""" % dir

      specs_dir = [ x for x in self.specs if x.conf['dir'] == dir ]

      atleastone = False

      for spec in specs_dir:

        if spec.data:

          atleastone = True

          break

      if not atleastone:

        data = data + "0\n\n"

        continue

 

      for spec in specs_dir:

        if not spec.data:

          continue

 

        effqd = spec.conf['numjobs'] * spec.conf['iodepth']

        jobs  = spec.conf['numjobs']

        qd    = spec.conf['iodepth']

        bs    = spec.conf['bs']

        iopsr = spec.data['read_iops']

        iopsw = spec.data['write_iops']

        kbsr  = spec.data['read_bw']

        kbsw  = spec.data['write_bw']

 

        data = data + "%8d %5d %4d %8d %8d %8d %9d %9d\n" % (

               effqd, jobs, qd, bs, iopsr, iopsw, kbsr, kbsw)

 

    self.outdatfp.write(data)

    self.outdatfp.flush()

 

class ProfilerConfig(object):

  def __init__(self, configfile=None):

    self.configfile = configfile

    self.config = self.__parseConfig()

    self.jobs = self.__createJobs()

 

  def dumpConfig(self):

    assert(self.config)

    for section in self.config.sections():

      print

      print "["+section+"]"

      for option in self.config.options(section):

        if option == "dirs":

          print "%s: %s" % (option, self.__getDirs(self.config, section))

        else:

          print "%s: %s" % (option, self.config.get(section, option))

 

  def dumpSpecs(self):

    assert(self.jobs)

    for job in self.jobs:

      for spec in job.specs:

        print "%s: %s" % (job.name, spec.conf)

 

  def __parseConfig(self):

    config = ConfigParser(defaults)

 

    if self.configfile:

      config.read(self.configfile)

    else:

      # Create a single 'config' section using just defaults

      config.add_section("config")

 

    self.__validate(config)

 

    return config

 

  def __validate(self, config):

    valid_opts = set(defaults)

    valid_dirs = set(dirs + ["readwrite"])

 

    for section in config.sections():

      sect_opts = set(config.options(section))

      if sect_opts != valid_opts:

        raise Exception("Invalid options %s for section '%s'" %

                        (list(sect_opts - valid_opts), section))

 

      sect_dirs_list = self.__getDirs(config, section)

      sect_dirs = set(sect_dirs_list)

      if not sect_dirs.issubset(valid_dirs):

        raise Exception("Invalid dirs %s for section '%s'" %

                        (list(sect_dirs - valid_dirs), section))

 

      # 'rw' and 'readwrite' are equivalent in 'fio'

      if set(['rw', 'readwrite']).issubset(sect_dirs):

        sect_dirs_list.remove('readwrite')

        sect_dirs_str = str(sect_dirs_list).translate(None, "[]")

        config.set(section, "dirs", sect_dirs_str)

 

      if config.get(section, "outdat") is None:

        raise Exception("Need 'outdat' for section '%s'" % section)

 

      # TODO: Sanity check everything else (eg. bs % 512, min < max)

 

  def __createJobs(self):

    assert(self.config)

 

    jobs = []

 

    for section in self.config.sections():

      job = ProfilerJob(section, dict(self.config.items(section)))

      self.__createSpecs(job)

      jobs.append(job)

 

    return jobs

 

  def __createSpecs(self, job):

    section = job.name

    minjob = int(self.config.get(section, "minjob"))

    maxjob = int(self.config.get(section, "maxjob"))

    muljob = int(self.config.get(section, "muljob"))

    mineqd = int(self.config.get(section, "mineqd"))

    maxeqd = int(self.config.get(section, "maxeqd"))

    muleqd = int(self.config.get(section, "muleqd"))

    minbsz = int(self.config.get(section, "minbsz"))

    maxbsz = int(self.config.get(section, "maxbsz"))

 

# Hack 'er up to do what I want.   Yeah, this is not how I should do this.

# dabe

    bszcur = minbsz

    while bszcur <= maxbsz:

      filename = ''

      drivelist = [ '\\\.\PhysicalDrive1', '\\\.\PhysicalDrive2', '\\\.\PhysicalDrive3:\\\.\PhysicalDrive4', '\\\.\PhysicalDrive5:\\\.\PhysicalDrive6', '\\\.\PhysicalDrive7:\\\.\PhysicalDrive8' ]

      for tmp_filename in drivelist:

        if len(filename) > 1:

          filename = filename + ':' + tmp_filename

        else:

          filename = tmp_filename

          diskCount = 1

        #print filename

        #print len(filename)

 

        #filename = self.config.get(section, 'filename')

        size = self.config.get(section, 'size')

        runtime = self.config.get(section, 'runtime')

   

        for dir in self.__getDirs(self.config, section):

          #curjob = minjob

          #while curjob <= maxjob:

            #cureqd = mineqd if mineqd == curjob else curjob*((mineqd/curjob)+1)

          iodepth = 1

          while iodepth <= 32:

            curjob = iodepth * diskCount

            #cureqd = mineqd if mineqd == curjob else curjob*((mineqd/curjob)+1)

            #while cureqd <= maxeqd or cureqd == curjob:

              #qdperjob = cureqd/curjob

            qdperjob = 1

            conf = {'filename': filename,

                    'size': size,

                    'runtime': runtime,

                    'dir': dir,

                     'numjobs': curjob,

                     'iodepth': qdperjob,

                     'bs': bszcur}

            spec = ProfilerSpec(conf)

            job.append(spec)

 

            if iodepth == 1:

              iodepth = iodepth + 1

            else:

              iodepth = iodepth + 4

             #cureqd = muleqd*(1+(cureqd/muleqd))

             #if cureqd % curjob:

               #cureqd = curjob*((cureqd/curjob)+1)

            #curjob = muljob*(1+(curjob/muljob))

        if diskCount < 2:

          diskCount = diskCount + 1

        else:

          diskCount = diskCount + 2

 

      bszcur = bszcur*2

 

  @staticmethod

  def __getDirs(config, section):

    assert(section)

    # ConfigParser values don't cope with lists, so we store 'dirs' as a string

    return literal_eval("["+config.get(section, "dirs")+"]")

 

class Profiler(object):

  def __init__(self, configfile):

    self.config = ProfilerConfig(opts.configfile)

    # TODO: Ensure 'fio' is installed

 

  def dumpConfig(self):

    self.config.dumpConfig()

 

  def dumpSpecs(self):

    self.config.dumpSpecs()

 

  def checkOutDat(self, overwrite):

    for job in self.config.jobs:

      if path.isdir(job.conf['outdat']):

        raise Exception("Unable to write results to '%s': it's a directory")

      if path.exists(job.conf['outdat']):

        if overwrite:

          print "Warning: overwriting file '%s'" % (job.conf['outdat'],)

        else:

          raise Exception("Refusing to overwrite file '%s': use -f" %

                          (job.conf['outdat'],))

 

  def runJobs(self):

    for job in self.config.jobs:

      try:

        job.run()

      except KeyboardInterrupt:

        print "\nInterrupted by keyboard, writing partial results only"

      job.writeDataFile()

 

def main(opts):

  profiler = Profiler(opts.configfile)

 

  if opts.verbose:

    profiler.dumpConfig()

    profiler.dumpSpecs()

 

  if opts.dryrun:

    return

 

  profiler.checkOutDat(opts.overwrite)

 

  profiler.runJobs()

 

if __name__ == "__main__":

  parser = OptionParser(usage="usage: %prog [options]")

  parser.add_option("-c", "--conf", dest="configfile",

                    help="Profiler configuration file")

  parser.add_option("-v", "--verbose", dest="verbose",

                    action="store_true", default=False,

                    help="Dump config and specs")

  parser.add_option("-n", "--dryrun", dest="dryrun",

                    action="store_true", default=False,

                    help="Just parse config file, don't run profiler")

  parser.add_option("-f", "--force", dest="overwrite",

                    action="store_true", default=False,

                    help="Overwrite existing outdat files")

  (opts, args) = parser.parse_args()

 

  exit(main(opts))

Re: Windows: FIO randomly hangs using attached script

[Sitsofe Wheeler]

On 6 March 2018 at 01:23, Rebecca Cran <rebecca@bluestop.org> wrote:
> I've had a report that FIO on Windows (at least Server 2012 R2 and 2016)
> hangs when the attached script is run. The point at which it hangs is
> apparently random, and within the condvar (pthread-win32) calls.
>
> I've replicated the hang, but I don't have time to debug it so I was hoping
> somebody on this mailing list might have time to dig into it and figure out
> what's wrong.

I'd like to look but I no longer have access to Windows machines
beyond what you can get with Appveyor so I'd need quite a restricted
test case to analyse this (see below for the problems I've been
having).

> The people I was talking to thought it might be due to FIO linking to
> msvcrt.dll, which is old and not supposed to be used by applications - they
> should use the CRT distributed with Visual C++, such as msvcr120.dll
> instead. However, it appears that fixing this would take quite a lot of work
> since while FIO is relatively straightforward to change, pthread-win32
> hasn't had a full release since 2012 and doesn't build under a current msys
> environment due to duplicated symbols etc.

It's true you're not supposed to link against the system msvcrt.dll
(see https://blogs.msdn.microsoft.com/oldnewthing/20140411-00/?p=1273
and https://sourceforge.net/p/mingw-w64/wiki2/The%20case%20against%20msvcrt.dll/
) however the MinGW folks have observed it can't really go away and
seem to gear everything up for doing that and only use it for basic
functionality (e.g. see
http://mingw-users.1079350.n2.nabble.com/Which-msvcrt-dll-does-MinGW-use-tp7582968p7582969.html
and https://ghc.haskell.org/trac/ghc/ticket/14537#comment:2 ).

Since your commit back in Jan 2014
(https://github.com/axboe/fio/commit/9aa5fe3290fd49c70e498d5e072a5b27e1c3034f
) fio migrated from pthread-win32 to MinGW's
internal winpthread (which seems to be alive as there's a commit from
Jan 2018 - https://github.com/mirror/mingw-w64/commits/master/mingw-w64-libraries/winpthreads
). If we have a repeatable small test case we might be able to send it
to the MinGW folks to look at...

I tried out the python script but it seemed to be complaining about a
whitespace issue. After fixing that up it's unclear exactly what fio
command lines it runs. I think for others to dig in we'd need
something less fiddly like the raw fio command lines that generate the
hang. Is there any chance the mystery user could join the mailing list
so they can answer questions directly? Ideally we'd need a job that
works on files within a filesystem and ideally nice small files so it
could go into an AppVeyor job...

-- 
Sitsofe | http://sucs.org/~sits/

Re: Windows: FIO randomly hangs using attached script

[Rebecca Cran]

On 3/6/2018 9:35 AM, Sitsofe Wheeler wrote:
>
> I tried out the python script but it seemed to be complaining about a
> whitespace issue. After fixing that up it's unclear exactly what fio
> command lines it runs. I think for others to dig in we'd need
> something less fiddly like the raw fio command lines that generate the
> hang. Is there any chance the mystery user could join the mailing list
> so they can answer questions directly? Ideally we'd need a job that
> works on files within a filesystem and ideally nice small files so it
> could go into an AppVeyor job...
>

I've CC'd Rob and David.
You can find the FIO command lines used by running the script in verbose 
mode: it lists them all at the start, then steps through them one by 
one. Unfortunately nobody's had any luck in narrowing down a single run 
that causes the hang.
I forgot to say that the problem has been seen on FIO 3.1 - I never got 
around to trying it on version 3.5, which is now on 
https://bluestop.org/fio .

-- 
Rebecca

Re: Windows: FIO randomly hangs using attached script

[Sitsofe Wheeler]

Hi Rob, David!

On 6 March 2018 at 20:01, Rebecca Cran <rebecca@bluestop.org> wrote:
> On 3/6/2018 9:35 AM, Sitsofe Wheeler wrote:
>>
>>
>> I tried out the python script but it seemed to be complaining about a
>> whitespace issue. After fixing that up it's unclear exactly what fio
>> command lines it runs. I think for others to dig in we'd need
>> something less fiddly like the raw fio command lines that generate the
>> hang. Is there any chance the mystery user could join the mailing list
>> so they can answer questions directly? Ideally we'd need a job that
>> works on files within a filesystem and ideally nice small files so it
>> could go into an AppVeyor job...
>>
>
> I've CC'd Rob and David.
> You can find the FIO command lines used by running the script in verbose
> mode: it lists them all at the start, then steps through them one by one.
> Unfortunately nobody's had any luck in narrowing down a single run that
> causes the hang.
> I forgot to say that the problem has been seen on FIO 3.1 - I never got
> around to trying it on version 3.5, which is now on https://bluestop.org/fio
> .

Rebecca has been explaining you've generated a reproducible hang on
Windows fio. Just for reference, can you check whether fio 3.5 also
reproduces the problem?

It looks like your python script generates fio lines which are based
off config lines similar to the following:

config: {'bs': 4096, 'filename': '\\\\.\\PhysicalDrive1', 'numjobs':
1, 'runtime': '20', 'iodepth': 1, 'dir': 'randread', 'size': '20G'}
config: {'bs': 16384, 'filename':
'\\\\.\\PhysicalDrive1:\\\\.\\PhysicalDrive2:\\\\.\\PhysicalDrive3:\\\\.\\PhysicalDrive4:\\\\.\\PhysicalDrive5:\\\\.\\PhysicalDrive6:\\\\.\\PhysicalDrive7:\\\\.\\PhysicalDrive8',
'numjobs': 240, 'runtime': '20', 'iodepth': 1, 'dir': 'randread',
'size': '20G'}
config: {'bs': 1048576, 'filename': '\\\\.\\PhysicalDrive1',
'numjobs': 30, 'runtime': '20', 'iodepth': 1, 'dir': 'randread',
'size': '20G'}

Is this correct? I'm afraid I'm not in a position where I can easily
debug this using the Python script on Windows (sadly I have no
interactive access to Windows machines at the moment) but a few things
might help to narrow down the problem:

How frequently is the script able to reproduce the problem?
Can you substitute files for disks and still reproduce the problem?
What are the minimum number of filenames involved when you've seen a
hang? Does it always need more than 1?
What's the smallest size that you can use that still reproduces the problem?
What's the smallest amount of numjobs that reproduce the problem? Does
it always need more than 1?

Ideally if we can get down to the stage where we run say only two fio
lines repeatedly in a bash script and make the problem happen it will
make it easier for others to see the problem too...

-- 
Sitsofe | http://sucs.org/~sits/

Re: Windows: FIO randomly hangs using attached script

[David Knierim]

Sitsofe,
   Thanks for your interest in resolving this issue.  I no longer have a system to dig into this issue at the moment, but I will attempt to get back to this issue soon.
  That being said, I like your questions and I am happy to answer them:

>  can you check whether fio 3.5 also reproduces the problem?
When I get a system to run on, I will attempt to reproduce the issue with fio 3.5.

Yes, the example fio command line examples you showed are indeed what the script is supposed to be doing.

If it helps, I also have a bash script which generates the same fio commands as the python script which also reproduces the issue.

I have not attempted to reproduce the issue with files or less raw disks.   I can see your desire for a simplified reproducer.   When I get back to this issue, I will see what I can come up with.   I am sure something simpler will reproduce the issue, but I have not spent the time to find something simpler that works.  I will explore number of disks/files, numjobs and also determine if the working set makes any difference.

In my experience, the python script never completed when run on Windows.  Rob had better luck than I did and the script ran to completion for him several times, but it also showed the failure multiple times as well, but I don't remember what the pass percentage was for him.

When I run the same python or bash script on Linux (just updating the path to the raw disks and the ioengine), it runs 100% reliably.

Thanks again,
   David

On 3/7/18, 1:01 AM, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote:

    Hi Rob, David!
    
    On 6 March 2018 at 20:01, Rebecca Cran <rebecca@bluestop.org> wrote:
    > On 3/6/2018 9:35 AM, Sitsofe Wheeler wrote:
    >>
    >>
    >> I tried out the python script but it seemed to be complaining about a
    >> whitespace issue. After fixing that up it's unclear exactly what fio
    >> command lines it runs. I think for others to dig in we'd need
    >> something less fiddly like the raw fio command lines that generate the
    >> hang. Is there any chance the mystery user could join the mailing list
    >> so they can answer questions directly? Ideally we'd need a job that
    >> works on files within a filesystem and ideally nice small files so it
    >> could go into an AppVeyor job...
    >>
    >
    > I've CC'd Rob and David.
    > You can find the FIO command lines used by running the script in verbose
    > mode: it lists them all at the start, then steps through them one by one.
    > Unfortunately nobody's had any luck in narrowing down a single run that
    > causes the hang.
    > I forgot to say that the problem has been seen on FIO 3.1 - I never got
    > around to trying it on version 3.5, which is now on https://urldefense.proofpoint.com/v2/url?u=https-3A__bluestop.org_fio&d=DwIBaQ&c=s883GpUCOChKOHiocYtGcg&r=Bg0zthdIszvkG4nFhDtYFPUGQbQFMAIndMvXqABqJjo&m=8kyD8W-AtALPg5fmmA3JgHijeRdF0kX7Jc-0r4QyrjE&s=Z_ELgdkCUioTlDh1xSF-SCddfCe2WEoOFnFqNcN-TxQ&e=
    > .
    
    Rebecca has been explaining you've generated a reproducible hang on
    Windows fio. Just for reference, can you check whether fio 3.5 also
    reproduces the problem?
    
    It looks like your python script generates fio lines which are based
    off config lines similar to the following:
    
    config: {'bs': 4096, 'filename': '\\\\.\\PhysicalDrive1', 'numjobs':
    1, 'runtime': '20', 'iodepth': 1, 'dir': 'randread', 'size': '20G'}
    config: {'bs': 16384, 'filename':
    '\\\\.\\PhysicalDrive1:\\\\.\\PhysicalDrive2:\\\\.\\PhysicalDrive3:\\\\.\\PhysicalDrive4:\\\\.\\PhysicalDrive5:\\\\.\\PhysicalDrive6:\\\\.\\PhysicalDrive7:\\\\.\\PhysicalDrive8',
    'numjobs': 240, 'runtime': '20', 'iodepth': 1, 'dir': 'randread',
    'size': '20G'}
    config: {'bs': 1048576, 'filename': '\\\\.\\PhysicalDrive1',
    'numjobs': 30, 'runtime': '20', 'iodepth': 1, 'dir': 'randread',
    'size': '20G'}
    
    Is this correct? I'm afraid I'm not in a position where I can easily
    debug this using the Python script on Windows (sadly I have no
    interactive access to Windows machines at the moment) but a few things
    might help to narrow down the problem:
    
    How frequently is the script able to reproduce the problem?
    Can you substitute files for disks and still reproduce the problem?
    What are the minimum number of filenames involved when you've seen a
    hang? Does it always need more than 1?
    What's the smallest size that you can use that still reproduces the problem?
    What's the smallest amount of numjobs that reproduce the problem? Does
    it always need more than 1?
    
    Ideally if we can get down to the stage where we run say only two fio
    lines repeatedly in a bash script and make the problem happen it will
    make it easier for others to see the problem too...
    
    -- 
    Sitsofe | https://urldefense.proofpoint.com/v2/url?u=http-3A__sucs.org_-7Esits_&d=DwIBaQ&c=s883GpUCOChKOHiocYtGcg&r=Bg0zthdIszvkG4nFhDtYFPUGQbQFMAIndMvXqABqJjo&m=8kyD8W-AtALPg5fmmA3JgHijeRdF0kX7Jc-0r4QyrjE&s=31fBjMR9Cd95PL7Ssm7hV2W8Kr5WmeHkp-ampBKgCfM&e=
    

Re: Windows: FIO randomly hangs using attached script

[Rob Scheepens]

Adding to David's comments: my success percentage was roughly 30%. Most of the time, 2 out of 3 runs would hang.

Thanks for your involvement here Sitsofe.

\Rob


On 07/03/2018, 17:33, "David Knierim" <knierim@nutanix.com> wrote:

    Sitsofe,
       Thanks for your interest in resolving this issue.  I no longer have a system to dig into this issue at the moment, but I will attempt to get back to this issue soon.
      That being said, I like your questions and I am happy to answer them:
    
    >  can you check whether fio 3.5 also reproduces the problem?
    When I get a system to run on, I will attempt to reproduce the issue with fio 3.5.
    
    Yes, the example fio command line examples you showed are indeed what the script is supposed to be doing.
    
    If it helps, I also have a bash script which generates the same fio commands as the python script which also reproduces the issue.
    
    I have not attempted to reproduce the issue with files or less raw disks.   I can see your desire for a simplified reproducer.   When I get back to this issue, I will see what I can come up with.   I am sure something simpler will reproduce the issue, but I have not spent the time to find something simpler that works.  I will explore number of disks/files, numjobs and also determine if the working set makes any difference.
    
    In my experience, the python script never completed when run on Windows.  Rob had better luck than I did and the script ran to completion for him several times, but it also showed the failure multiple times as well, but I don't remember what the pass percentage was for him.
    
    When I run the same python or bash script on Linux (just updating the path to the raw disks and the ioengine), it runs 100% reliably.
    
    Thanks again,
       David
    
    On 3/7/18, 1:01 AM, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote:
    
        Hi Rob, David!
        
        On 6 March 2018 at 20:01, Rebecca Cran <rebecca@bluestop.org> wrote:
        > On 3/6/2018 9:35 AM, Sitsofe Wheeler wrote:
        >>
        >>
        >> I tried out the python script but it seemed to be complaining about a
        >> whitespace issue. After fixing that up it's unclear exactly what fio
        >> command lines it runs. I think for others to dig in we'd need
        >> something less fiddly like the raw fio command lines that generate the
        >> hang. Is there any chance the mystery user could join the mailing list
        >> so they can answer questions directly? Ideally we'd need a job that
        >> works on files within a filesystem and ideally nice small files so it
        >> could go into an AppVeyor job...
        >>
        >
        > I've CC'd Rob and David.
        > You can find the FIO command lines used by running the script in verbose
        > mode: it lists them all at the start, then steps through them one by one.
        > Unfortunately nobody's had any luck in narrowing down a single run that
        > causes the hang.
        > I forgot to say that the problem has been seen on FIO 3.1 - I never got
        > around to trying it on version 3.5, which is now on https://urldefense.proofpoint.com/v2/url?u=https-3A__bluestop.org_fio&d=DwIBaQ&c=s883GpUCOChKOHiocYtGcg&r=Bg0zthdIszvkG4nFhDtYFPUGQbQFMAIndMvXqABqJjo&m=8kyD8W-AtALPg5fmmA3JgHijeRdF0kX7Jc-0r4QyrjE&s=Z_ELgdkCUioTlDh1xSF-SCddfCe2WEoOFnFqNcN-TxQ&e=
        > .
        
        Rebecca has been explaining you've generated a reproducible hang on
        Windows fio. Just for reference, can you check whether fio 3.5 also
        reproduces the problem?
        
        It looks like your python script generates fio lines which are based
        off config lines similar to the following:
        
        config: {'bs': 4096, 'filename': '\\\\.\\PhysicalDrive1', 'numjobs':
        1, 'runtime': '20', 'iodepth': 1, 'dir': 'randread', 'size': '20G'}
        config: {'bs': 16384, 'filename':
        '\\\\.\\PhysicalDrive1:\\\\.\\PhysicalDrive2:\\\\.\\PhysicalDrive3:\\\\.\\PhysicalDrive4:\\\\.\\PhysicalDrive5:\\\\.\\PhysicalDrive6:\\\\.\\PhysicalDrive7:\\\\.\\PhysicalDrive8',
        'numjobs': 240, 'runtime': '20', 'iodepth': 1, 'dir': 'randread',
        'size': '20G'}
        config: {'bs': 1048576, 'filename': '\\\\.\\PhysicalDrive1',
        'numjobs': 30, 'runtime': '20', 'iodepth': 1, 'dir': 'randread',
        'size': '20G'}
        
        Is this correct? I'm afraid I'm not in a position where I can easily
        debug this using the Python script on Windows (sadly I have no
        interactive access to Windows machines at the moment) but a few things
        might help to narrow down the problem:
        
        How frequently is the script able to reproduce the problem?
        Can you substitute files for disks and still reproduce the problem?
        What are the minimum number of filenames involved when you've seen a
        hang? Does it always need more than 1?
        What's the smallest size that you can use that still reproduces the problem?
        What's the smallest amount of numjobs that reproduce the problem? Does
        it always need more than 1?
        
        Ideally if we can get down to the stage where we run say only two fio
        lines repeatedly in a bash script and make the problem happen it will
        make it easier for others to see the problem too...
        
        -- 
        Sitsofe | https://urldefense.proofpoint.com/v2/url?u=http-3A__sucs.org_-7Esits_&d=DwIBaQ&c=s883GpUCOChKOHiocYtGcg&r=Bg0zthdIszvkG4nFhDtYFPUGQbQFMAIndMvXqABqJjo&m=8kyD8W-AtALPg5fmmA3JgHijeRdF0kX7Jc-0r4QyrjE&s=31fBjMR9Cd95PL7Ssm7hV2W8Kr5WmeHkp-ampBKgCfM&e=
        
    
    

Re: Windows: FIO randomly hangs using attached script

[Sitsofe Wheeler]

Thanks for following up.

Do you happen to know how far it would get on runs where it went on to
hang? Would it be it in the middle of a job or was it still doing
setup or varied? Would it take hours to happen?

On 7 March 2018 at 15:39, Rob Scheepens <rob.scheepens@nutanix.com> wrote:
> Adding to David's comments: my success percentage was roughly 30%. Most of the time, 2 out of 3 runs would hang.
>
> Thanks for your involvement here Sitsofe.
>
> \Rob
>
>
> On 07/03/2018, 17:33, "David Knierim" <knierim@nutanix.com> wrote:
>
>     Sitsofe,
>        Thanks for your interest in resolving this issue.  I no longer have a system to dig into this issue at the moment, but I will attempt to get back to this issue soon.
>       That being said, I like your questions and I am happy to answer them:
>
>     >  can you check whether fio 3.5 also reproduces the problem?
>     When I get a system to run on, I will attempt to reproduce the issue with fio 3.5.
>
>     Yes, the example fio command line examples you showed are indeed what the script is supposed to be doing.
>
>     If it helps, I also have a bash script which generates the same fio commands as the python script which also reproduces the issue.
>
>     I have not attempted to reproduce the issue with files or less raw disks.   I can see your desire for a simplified reproducer.   When I get back to this issue, I will see what I can come up with.   I am sure something simpler will reproduce the issue, but I have not spent the time to find something simpler that works.  I will explore number of disks/files, numjobs and also determine if the working set makes any difference.
>
>     In my experience, the python script never completed when run on Windows.  Rob had better luck than I did and the script ran to completion for him several times, but it also showed the failure multiple times as well, but I don't remember what the pass percentage was for him.
>
>     When I run the same python or bash script on Linux (just updating the path to the raw disks and the ioengine), it runs 100% reliably.

-- 
Sitsofe | http://sucs.org/~sits/

Re: Windows: FIO randomly hangs using attached script

[Rebecca Cran]

On 03/07/2018 09:01 AM, Sitsofe Wheeler wrote:
> Thanks for following up.
>
> Do you happen to know how far it would get on runs where it went on to
> hang? Would it be it in the middle of a job or was it still doing
> setup or varied? Would it take hours to happen?

I saw hangs at job 130 and 134.

-- 
Rebecca

Re: Windows: FIO randomly hangs using attached script

[Rob Scheepens]

Merging since Rebecca's reply lost the thread:


From memory, my runs would also get somewhere over #100, but I'll confirm next week.

\Rob


On 03/07/2018 09:01 AM, Sitsofe Wheeler wrote:
Thanks for following up.

Do you happen to know how far it would get on runs where it went on to
hang? Would it be it in the middle of a job or was it still doing
setup or varied? Would it take hours to happen?

I saw hangs at job 130 and 134.

--
Rebecca

On 07/03/2018, 18:01, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote:

    Thanks for following up.
    
    Do you happen to know how far it would get on runs where it went on to
    hang? Would it be it in the middle of a job or was it still doing
    setup or varied? Would it take hours to happen?
    
    On 7 March 2018 at 15:39, Rob Scheepens <rob.scheepens@nutanix.com> wrote:
    > Adding to David's comments: my success percentage was roughly 30%. Most of the time, 2 out of 3 runs would hang.
    >
    > Thanks for your involvement here Sitsofe.
    >
    > \Rob
    >
    >
    > On 07/03/2018, 17:33, "David Knierim" <knierim@nutanix.com> wrote:
    >
    >     Sitsofe,
    >        Thanks for your interest in resolving this issue.  I no longer have a system to dig into this issue at the moment, but I will attempt to get back to this issue soon.
    >       That being said, I like your questions and I am happy to answer them:
    >
    >     >  can you check whether fio 3.5 also reproduces the problem?
    >     When I get a system to run on, I will attempt to reproduce the issue with fio 3.5.
    >
    >     Yes, the example fio command line examples you showed are indeed what the script is supposed to be doing.
    >
    >     If it helps, I also have a bash script which generates the same fio commands as the python script which also reproduces the issue.
    >
    >     I have not attempted to reproduce the issue with files or less raw disks.   I can see your desire for a simplified reproducer.   When I get back to this issue, I will see what I can come up with.   I am sure something simpler will reproduce the issue, but I have not spent the time to find something simpler that works.  I will explore number of disks/files, numjobs and also determine if the working set makes any difference.
    >
    >     In my experience, the python script never completed when run on Windows.  Rob had better luck than I did and the script ran to completion for him several times, but it also showed the failure multiple times as well, but I don't remember what the pass percentage was for him.
    >
    >     When I run the same python or bash script on Linux (just updating the path to the raw disks and the ioengine), it runs 100% reliably.
    
    -- 
    Sitsofe | https://urldefense.proofpoint.com/v2/url?u=http-3A__sucs.org_-7Esits_&d=DwIFaQ&c=s883GpUCOChKOHiocYtGcg&r=OMged-t_5I_fmfpUaT3vaA06lgLL_alYnDQJxHmXz64&m=927s0SaucUQFFVqXcunP1_PBnr0xEHUL8d_bKt-VIUE&s=vD6ZYXWmBRWz5kR8R4WKQF7b3l4BmLfVYl_uHRZGKq0&e=
    

Re: Windows: FIO randomly hangs using attached script

[Rebecca Cran]

On 03/07/2018 09:05 AM, Rob Scheepens wrote:
> Merging since Rebecca's reply lost the thread:

Sorry, I trimmed my reply since people should have been seeing the 
previous messages and so likely don't need new emails to contain all the 
previous content. Though I understand it's not the normal style in 
Outlook etc.

-- 
Rebecca

Re: Windows: FIO randomly hangs using attached script

[Rob Scheepens]

Just tested using fio-3.5: the first attempt already hung, at:

PS C:\Python27> python .\profiler_win_dk.py
* Running job: 'config' (20 secs / spec)
** Executing spec 176/405 at 2018-03-08 02:10:25.360000

Instead of trying an large amount of variations to reproduce, which will be time-consuming, is there a way to capture a dump using gdb and debug that? I have very limited gdb skills and when I recently tried to save it off, it gave me an error that seemed to implicate some lib was not implemented on Windows.

I left the vm in the current state, in case anyone wants me to get something from it.

\Rob


On 07/03/2018, 17:39, "Rob Scheepens" <rob.scheepens@nutanix.com> wrote:

    Adding to David's comments: my success percentage was roughly 30%. Most of the time, 2 out of 3 runs would hang.
    
    Thanks for your involvement here Sitsofe.
    
    \Rob
    
    
    On 07/03/2018, 17:33, "David Knierim" <knierim@nutanix.com> wrote:
    
        Sitsofe,
           Thanks for your interest in resolving this issue.  I no longer have a system to dig into this issue at the moment, but I will attempt to get back to this issue soon.
          That being said, I like your questions and I am happy to answer them:
        
        >  can you check whether fio 3.5 also reproduces the problem?
        When I get a system to run on, I will attempt to reproduce the issue with fio 3.5.
        
        Yes, the example fio command line examples you showed are indeed what the script is supposed to be doing.
        
        If it helps, I also have a bash script which generates the same fio commands as the python script which also reproduces the issue.
        
        I have not attempted to reproduce the issue with files or less raw disks.   I can see your desire for a simplified reproducer.   When I get back to this issue, I will see what I can come up with.   I am sure something simpler will reproduce the issue, but I have not spent the time to find something simpler that works.  I will explore number of disks/files, numjobs and also determine if the working set makes any difference.
        
        In my experience, the python script never completed when run on Windows.  Rob had better luck than I did and the script ran to completion for him several times, but it also showed the failure multiple times as well, but I don't remember what the pass percentage was for him.
        
        When I run the same python or bash script on Linux (just updating the path to the raw disks and the ioengine), it runs 100% reliably.
        
        Thanks again,
           David
        
        On 3/7/18, 1:01 AM, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote:
        
            Hi Rob, David!
            
            On 6 March 2018 at 20:01, Rebecca Cran <rebecca@bluestop.org> wrote:
            > On 3/6/2018 9:35 AM, Sitsofe Wheeler wrote:
            >>
            >>
            >> I tried out the python script but it seemed to be complaining about a
            >> whitespace issue. After fixing that up it's unclear exactly what fio
            >> command lines it runs. I think for others to dig in we'd need
            >> something less fiddly like the raw fio command lines that generate the
            >> hang. Is there any chance the mystery user could join the mailing list
            >> so they can answer questions directly? Ideally we'd need a job that
            >> works on files within a filesystem and ideally nice small files so it
            >> could go into an AppVeyor job...
            >>
            >
            > I've CC'd Rob and David.
            > You can find the FIO command lines used by running the script in verbose
            > mode: it lists them all at the start, then steps through them one by one.
            > Unfortunately nobody's had any luck in narrowing down a single run that
            > causes the hang.
            > I forgot to say that the problem has been seen on FIO 3.1 - I never got
            > around to trying it on version 3.5, which is now on https://urldefense.proofpoint.com/v2/url?u=https-3A__bluestop.org_fio&d=DwIBaQ&c=s883GpUCOChKOHiocYtGcg&r=Bg0zthdIszvkG4nFhDtYFPUGQbQFMAIndMvXqABqJjo&m=8kyD8W-AtALPg5fmmA3JgHijeRdF0kX7Jc-0r4QyrjE&s=Z_ELgdkCUioTlDh1xSF-SCddfCe2WEoOFnFqNcN-TxQ&e=
            > .
            
            Rebecca has been explaining you've generated a reproducible hang on
            Windows fio. Just for reference, can you check whether fio 3.5 also
            reproduces the problem?
            
            It looks like your python script generates fio lines which are based
            off config lines similar to the following:
            
            config: {'bs': 4096, 'filename': '\\\\.\\PhysicalDrive1', 'numjobs':
            1, 'runtime': '20', 'iodepth': 1, 'dir': 'randread', 'size': '20G'}
            config: {'bs': 16384, 'filename':
            '\\\\.\\PhysicalDrive1:\\\\.\\PhysicalDrive2:\\\\.\\PhysicalDrive3:\\\\.\\PhysicalDrive4:\\\\.\\PhysicalDrive5:\\\\.\\PhysicalDrive6:\\\\.\\PhysicalDrive7:\\\\.\\PhysicalDrive8',
            'numjobs': 240, 'runtime': '20', 'iodepth': 1, 'dir': 'randread',
            'size': '20G'}
            config: {'bs': 1048576, 'filename': '\\\\.\\PhysicalDrive1',
            'numjobs': 30, 'runtime': '20', 'iodepth': 1, 'dir': 'randread',
            'size': '20G'}
            
            Is this correct? I'm afraid I'm not in a position where I can easily
            debug this using the Python script on Windows (sadly I have no
            interactive access to Windows machines at the moment) but a few things
            might help to narrow down the problem:
            
            How frequently is the script able to reproduce the problem?
            Can you substitute files for disks and still reproduce the problem?
            What are the minimum number of filenames involved when you've seen a
            hang? Does it always need more than 1?
            What's the smallest size that you can use that still reproduces the problem?
            What's the smallest amount of numjobs that reproduce the problem? Does
            it always need more than 1?
            
            Ideally if we can get down to the stage where we run say only two fio
            lines repeatedly in a bash script and make the problem happen it will
            make it easier for others to see the problem too...
            
            -- 
            Sitsofe | https://urldefense.proofpoint.com/v2/url?u=http-3A__sucs.org_-7Esits_&d=DwIBaQ&c=s883GpUCOChKOHiocYtGcg&r=Bg0zthdIszvkG4nFhDtYFPUGQbQFMAIndMvXqABqJjo&m=8kyD8W-AtALPg5fmmA3JgHijeRdF0kX7Jc-0r4QyrjE&s=31fBjMR9Cd95PL7Ssm7hV2W8Kr5WmeHkp-ampBKgCfM&e=
            
        
        
    
    

Re: Windows: FIO randomly hangs using attached script

[Sitsofe Wheeler]

A backtrace would be a good starting point but we need your gdb to be
able to be able to produce that... If you've managed to successfully
attach gdb to fio running something like

set logging file fiobacktrace.txt
set logging on
thread apply all bt
set logging off

and copying the contents of fiobacktrace.txt to here would be helpful.
If possible also take a note of the exact job that failed and send it
over as presumably we can just keep rerunning that to produce the
problem.

On 8 March 2018 at 10:51, Rob Scheepens <rob.scheepens@nutanix.com> wrote:
> Just tested using fio-3.5: the first attempt already hung, at:
>
> PS C:\Python27> python .\profiler_win_dk.py
> * Running job: 'config' (20 secs / spec)
> ** Executing spec 176/405 at 2018-03-08 02:10:25.360000
>
> Instead of trying an large amount of variations to reproduce, which will be time-consuming, is there a way to capture a dump using gdb and debug that? I have very limited gdb skills and when I recently tried to save it off, it gave me an error that seemed to implicate some lib was not implemented on Windows.
>
> I left the vm in the current state, in case anyone wants me to get something from it.

-- 
Sitsofe | http://sucs.org/~sits/

Re: Windows: FIO randomly hangs using attached script

[Rob Scheepens]

I'll see if I can get gdb setup like that.

Below output shows spec 176 was the last job that ran ok, so you could try if it reproduces with 177. However, the job number exhibiting the hang usually is different every time.

\Rob


On 08/03/2018, 14:29, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote:

    A backtrace would be a good starting point but we need your gdb to be
    able to be able to produce that... If you've managed to successfully
    attach gdb to fio running something like
    
    set logging file fiobacktrace.txt
    set logging on
    thread apply all bt
    set logging off
    
    and copying the contents of fiobacktrace.txt to here would be helpful.
    If possible also take a note of the exact job that failed and send it
    over as presumably we can just keep rerunning that to produce the
    problem.
    
    On 8 March 2018 at 10:51, Rob Scheepens <rob.scheepens@nutanix.com> wrote:
    > Just tested using fio-3.5: the first attempt already hung, at:
    >
    > PS C:\Python27> python .\profiler_win_dk.py
    > * Running job: 'config' (20 secs / spec)
    > ** Executing spec 176/405 at 2018-03-08 02:10:25.360000
    >
    > Instead of trying an large amount of variations to reproduce, which will be time-consuming, is there a way to capture a dump using gdb and debug that? I have very limited gdb skills and when I recently tried to save it off, it gave me an error that seemed to implicate some lib was not implemented on Windows.
    >
    > I left the vm in the current state, in case anyone wants me to get something from it.
    
    -- 
    Sitsofe | https://urldefense.proofpoint.com/v2/url?u=http-3A__sucs.org_-7Esits_&d=DwIFaQ&c=s883GpUCOChKOHiocYtGcg&r=OMged-t_5I_fmfpUaT3vaA06lgLL_alYnDQJxHmXz64&m=_j-o6LW-O_G6OkO-vy7HdE7UJDsy8W_csQytzi2Vg7I&s=u17mNDv5miOlk6AgsMohHL41aIameCmzLB1vhzTlxJc&e=
    

Re: Windows: FIO randomly hangs using attached script

[Rob Scheepens]

[-- Attachment #1: Type: text/plain, Size: 2198 bytes --]

Sitsofe,

See attached for the fio threads' backtraces.

\Rob


On 08/03/2018, 14:39, "Rob Scheepens" <rob.scheepens@nutanix.com> wrote:

    I'll see if I can get gdb setup like that.
    
    Below output shows spec 176 was the last job that ran ok, so you could try if it reproduces with 177. However, the job number exhibiting the hang usually is different every time.
    
    \Rob
    
    
    On 08/03/2018, 14:29, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote:
    
        A backtrace would be a good starting point but we need your gdb to be
        able to be able to produce that... If you've managed to successfully
        attach gdb to fio running something like
        
        set logging file fiobacktrace.txt
        set logging on
        thread apply all bt
        set logging off
        
        and copying the contents of fiobacktrace.txt to here would be helpful.
        If possible also take a note of the exact job that failed and send it
        over as presumably we can just keep rerunning that to produce the
        problem.
        
        On 8 March 2018 at 10:51, Rob Scheepens <rob.scheepens@nutanix.com> wrote:
        > Just tested using fio-3.5: the first attempt already hung, at:
        >
        > PS C:\Python27> python .\profiler_win_dk.py
        > * Running job: 'config' (20 secs / spec)
        > ** Executing spec 176/405 at 2018-03-08 02:10:25.360000
        >
        > Instead of trying an large amount of variations to reproduce, which will be time-consuming, is there a way to capture a dump using gdb and debug that? I have very limited gdb skills and when I recently tried to save it off, it gave me an error that seemed to implicate some lib was not implemented on Windows.
        >
        > I left the vm in the current state, in case anyone wants me to get something from it.
        
        -- 
        Sitsofe | https://urldefense.proofpoint.com/v2/url?u=http-3A__sucs.org_-7Esits_&d=DwIFaQ&c=s883GpUCOChKOHiocYtGcg&r=OMged-t_5I_fmfpUaT3vaA06lgLL_alYnDQJxHmXz64&m=_j-o6LW-O_G6OkO-vy7HdE7UJDsy8W_csQytzi2Vg7I&s=u17mNDv5miOlk6AgsMohHL41aIameCmzLB1vhzTlxJc&e=
        
    
    


[-- Attachment #2: fiobacktrace.txt --]
[-- Type: text/plain, Size: 212648 bytes --]


Thread 201 (Thread 3784.0x1540):
#0  0x00007ff845d098f1 in ntdll!DbgBreakPoint () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff845d3192a in ntdll!DbgUiRemoteBreakin () from C:\Windows\SYSTEM32\ntdll.dll
#2  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#3  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#4  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 200 (Thread 3784.0xe70):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32b948b0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 199 (Thread 3784.0xd6c):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32b6f140) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 198 (Thread 3784.0xcd0):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32bf1810) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 197 (Thread 3784.0xb78):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32963a90) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 196 (Thread 3784.0xaa4):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32b944d0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 195 (Thread 3784.0x162c):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32706a00) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 194 (Thread 3784.0x1444):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32b94610) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 193 (Thread 3784.0x1140):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32b94690) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 192 (Thread 3784.0x1688):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32b6f020) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 191 (Thread 3784.0x12b0):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32bf1710) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 190 (Thread 3784.0x105c):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32b94390) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 189 (Thread 3784.0x10ec):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32b6ef00) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 188 (Thread 3784.0x51c):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32963950) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 187 (Thread 3784.0x9c4):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32a56b90) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 186 (Thread 3784.0xce8):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32b6ef80) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 185 (Thread 3784.0x1798):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x329639d0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 184 (Thread 3784.0xf20):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32a292b0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 183 (Thread 3784.0x12c8):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32a56a50) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 182 (Thread 3784.0xcd4):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32b438b0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 181 (Thread 3784.0x4cc):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32a293b0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 180 (Thread 3784.0x1264):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32a56ad0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 179 (Thread 3784.0x17d4):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32963e10) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 178 (Thread 3784.0xdac):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32991780) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 177 (Thread 3784.0x15ec):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32a29350) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 176 (Thread 3784.0x640):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x3297d110) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 175 (Thread 3784.0x1214):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32991720) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 174 (Thread 3784.0xd48):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x3297d350) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 173 (Thread 3784.0x10bc):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32991760) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 172 (Thread 3784.0x65c):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x327069a0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 171 (Thread 3784.0x15b0):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32963e90) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 170 (Thread 3784.0xa08):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x3297d330) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 169 (Thread 3784.0x112c):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x3280f610) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 168 (Thread 3784.0x30c):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x327068e0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 167 (Thread 3784.0x5ac):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32921070) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 166 (Thread 3784.0x1580):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x3280f590) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 165 (Thread 3784.0x129c):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2bf923e0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 164 (Thread 3784.0xb8c):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x329213f0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 163 (Thread 3784.0xcac):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x3280f8b0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 162 (Thread 3784.0x43c):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32747f80) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 161 (Thread 3784.0x1244):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32706c20) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 160 (Thread 3784.0x1684):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x3280f690) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 159 (Thread 3784.0x137c):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2bf92240) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 158 (Thread 3784.0x15d8):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32747de0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 157 (Thread 3784.0xcf0):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2e19e1b0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 156 (Thread 3784.0x1054):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2e19e1d0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 155 (Thread 3784.0x167c):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32706e50) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 154 (Thread 3784.0x1464):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32706d50) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 153 (Thread 3784.0x1788):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32706cd0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 152 (Thread 3784.0x15d4):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32706e10) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 151 (Thread 3784.0x1768):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2bf92320) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 150 (Thread 3784.0xf80):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2e19d2f0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 149 (Thread 3784.0x1170):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2e19e490) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 148 (Thread 3784.0xdc4):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2e16c330) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 147 (Thread 3784.0x9dc):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2bf925e0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 146 (Thread 3784.0x5c0):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2e19d3f0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 145 (Thread 3784.0x16f8):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2e19d4b0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 144 (Thread 3784.0x1524):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2bf92680) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 143 (Thread 3784.0x10a0):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2bf9e670) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 142 (Thread 3784.0x1610):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2e16c3f0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 141 (Thread 3784.0x248):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2e066d40) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 140 (Thread 3784.0xf90):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2bf9e810) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 139 (Thread 3784.0xcb0):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2bf91a80) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 138 (Thread 3784.0x149c):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2e16c290) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 137 (Thread 3784.0xe54):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2e066dc0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 136 (Thread 3784.0xf58):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2bf9e7f0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 135 (Thread 3784.0xd4c):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2bf94ca0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 134 (Thread 3784.0x171c):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2e066cc0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 133 (Thread 3784.0xb84):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2e031ca0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 132 (Thread 3784.0x126c):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2bf91e60) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 131 (Thread 3784.0x146c):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2df4b540) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 130 (Thread 3784.0x1718):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2e031da0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 129 (Thread 3784.0x161c):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2df4b320) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 128 (Thread 3784.0xf94):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2bf919e0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 127 (Thread 3784.0x1128):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2bf91f60) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 126 (Thread 3784.0x1324):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2df08510) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 125 (Thread 3784.0x1630):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2defc970) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 124 (Thread 3784.0x3f4):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2df085b0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 123 (Thread 3784.0x16a8):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2bf91960) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 122 (Thread 3784.0x268):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2bf85150) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 121 (Thread 3784.0xac8):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2defcb90) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 120 (Thread 3784.0xb70):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2bf91dc0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 119 (Thread 3784.0xbc):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2bf94bc0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 118 (Thread 3784.0x7c):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2bf787a0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 117 (Thread 3784.0x9b4):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2bf852f0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 116 (Thread 3784.0xe40):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2be67690) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 115 (Thread 3784.0x134):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2bf78640) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 114 (Thread 3784.0x12e4):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0xb001dd0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 113 (Thread 3784.0x7f4):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2be673f0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 112 (Thread 3784.0xd98):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0xb0ad8b0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 111 (Thread 3784.0x1490):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0xb001890) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 110 (Thread 3784.0x1784):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2be674b0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 109 (Thread 3784.0x988):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0xb0ad950) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 108 (Thread 3784.0x4b4):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0xb001830) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 107 (Thread 3784.0xfa0):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0xaff3eb0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 106 (Thread 3784.0x16d4):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0xb0ad710) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 105 (Thread 3784.0x1648):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x2bdaa2c0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 104 (Thread 3784.0x1118):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0xb001e10) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 103 (Thread 3784.0x104):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0xafe9750) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 102 (Thread 3784.0x8d4):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0xb001bd0) at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 101 (Thread 3784.0x14c0):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c2c360, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c2c370) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb82f850) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 100 (Thread 3784.0x16d0):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c2b460, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c2b470) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb81e3a0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 99 (Thread 3784.0x1538):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c2a560, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c2a570) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb80cef0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 98 (Thread 3784.0x7c0):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c29660, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c29670) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb7fba40) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 97 (Thread 3784.0x16c):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c28760, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c28770) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb7ea590) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 96 (Thread 3784.0x16cc):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c27860, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c27870) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb7d90e0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 95 (Thread 3784.0x41c):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c26960, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c26970) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb7c7c30) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 94 (Thread 3784.0x179c):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c25a60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c25a70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb7b6780) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 93 (Thread 3784.0xa9c):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c24b60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c24b70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb7a52d0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 92 (Thread 3784.0x638):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c23c60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c23c70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb793e20) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 91 (Thread 3784.0x1528):
#0  0x00007ff845d09814 in ntdll!ZwWaitForAlertByThreadId () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff845c8faa7 in ntdll!RtlpUnWaitCriticalSection () from C:\Windows\SYSTEM32\ntdll.dll
#2  0x00007ff845c8f9ae in ntdll!RtlpUnWaitCriticalSection () from C:\Windows\SYSTEM32\ntdll.dll
#3  0x00007ff845c8f83f in ntdll!RtlpUnWaitCriticalSection () from C:\Windows\SYSTEM32\ntdll.dll
#4  0x00007ff845c90d04 in ntdll!RtlEnterCriticalSection () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x00007ff845c90c30 in ntdll!RtlEnterCriticalSection () from C:\Windows\SYSTEM32\ntdll.dll
#6  0x000000000046baaf in pthread_cond_wait (c=c@entry=0x1b0008, external_mutex=external_mutex@entry=0x1b0000)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:437
#7  0x0000000000441232 in fio_mutex_down (mutex=0x1b0000) at mutex.c:224
#8  0x0000000000446eae in sfree_pool (ptr=0x2c22bc0, pool=0x55c980 <mp>) at smalloc.c:312
#9  sfree (ptr=0x2c22bd0) at smalloc.c:336
#10 0x000000000042df76 in close_and_free_files (td=td@entry=0xb782970) at filesetup.c:1367
#11 0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#12 0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#13 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#14 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#15 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#16 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#17 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 90 (Thread 3784.0x14e8):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c21e60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c21e70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb7714c0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 89 (Thread 3784.0x1070):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c20f60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c20f70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb760010) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 88 (Thread 3784.0x1654):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c20060, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c20070) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb74eb60) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 87 (Thread 3784.0xec4):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c1f160, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c1f170) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb73d6b0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 86 (Thread 3784.0x16a4):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c1e260, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c1e270) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb72c200) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 85 (Thread 3784.0x12a8):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c1d360, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c1d370) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb71ad50) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 84 (Thread 3784.0xab4):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c1c460, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c1c470) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb7098a0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 83 (Thread 3784.0x1468):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c1b560, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c1b570) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb6f83f0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 82 (Thread 3784.0x1650):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c1a660, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c1a670) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb6e6f40) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 81 (Thread 3784.0x1314):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c19760, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c19770) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb6d5a90) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 80 (Thread 3784.0x1298):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c18860, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c18870) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb6c45e0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 79 (Thread 3784.0xfa4):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c16a60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c16a70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb6a1c80) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 78 (Thread 3784.0x152c):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c15b60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c15b70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb6907d0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 77 (Thread 3784.0x111c):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c14c60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c14c70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb67f320) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 76 (Thread 3784.0xb80):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c13d60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c13d70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb66de70) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 75 (Thread 3784.0x32c):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c12e60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c12e70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb65c9c0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 74 (Thread 3784.0x2f4):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c11f60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c11f70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb64b510) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 73 (Thread 3784.0x3d0):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c11060, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c11070) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb63a060) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 72 (Thread 3784.0x1004):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c10160, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c10170) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb628bb0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 71 (Thread 3784.0x11c0):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c0f260, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c0f270) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb617700) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 70 (Thread 3784.0x770):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c0e360, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c0e370) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb606250) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 69 (Thread 3784.0x8f8):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c0d460, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c0d470) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb5f4da0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 68 (Thread 3784.0xf1c):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c0c560, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c0c570) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb5e38f0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 67 (Thread 3784.0x1058):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c0b660, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c0b670) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb5d2440) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 66 (Thread 3784.0x165c):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c0a760, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c0a770) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb5c0f90) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 65 (Thread 3784.0x1558):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c09860, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c09870) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb5afae0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 64 (Thread 3784.0xfe4):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c08960, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c08970) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb59e630) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 63 (Thread 3784.0x17cc):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c07a60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c07a70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb58d180) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 62 (Thread 3784.0x1248):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c06b60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c06b70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb57bcd0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 61 (Thread 3784.0x1334):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c05c60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c05c70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb56a820) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 60 (Thread 3784.0xfe8):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c04d60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c04d70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb559370) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 59 (Thread 3784.0x17ac):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c03e60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c03e70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb547ec0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 58 (Thread 3784.0xe44):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c02f60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c02f70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb536a10) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 57 (Thread 3784.0xf30):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c02060, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c02070) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb525560) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 56 (Thread 3784.0x3b0):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c01160, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c01170) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb5140b0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 55 (Thread 3784.0x1604):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2c00260, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2c00270) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb502c00) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 54 (Thread 3784.0x188):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bff360, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bff370) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb4f1750) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 53 (Thread 3784.0xca0):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046b4cc in do_sema_b_wait_intern (sema=sema@entry=0xa8, nointerrupt=nointerrupt@entry=1, 
    timeout=timeout@entry=4294967295) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:601
#3  0x000000000046b60e in do_sema_b_wait (sema=0xa8, nointerrupt=1, timeout=4294967295, cs=0x1e15f0, val=0x1e1618)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:584
#4  0x000000000046b89f in pthread_cond_signal (c=<optimized out>)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:327
#5  0x00000000004412c0 in fio_mutex_up (mutex=<optimized out>) at mutex.c:246
#6  0x0000000000446feb in sfree_pool (ptr=<optimized out>, pool=<optimized out>) at smalloc.c:317
#7  sfree (ptr=<optimized out>) at smalloc.c:336
#8  0x000000000042df59 in close_and_free_files (td=td@entry=0xb4e02a0) at filesetup.c:1358
#9  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#10 0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#11 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#13 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#14 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#15 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 52 (Thread 3784.0x1114):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bfd560, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bfd570) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb4cedf0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 51 (Thread 3784.0x894):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bfc660, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bfc670) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb4bd940) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 50 (Thread 3784.0xd24):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bfb760, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bfb770) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb4ac490) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 49 (Thread 3784.0xadc):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bfa860, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bfa870) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb49afe0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 48 (Thread 3784.0x8):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bf9960, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bf9970) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb489b30) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 47 (Thread 3784.0xdf4):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bf8a60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bf8a70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb478680) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 46 (Thread 3784.0x10dc):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bf7b60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bf7b70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb4671d0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 45 (Thread 3784.0xa18):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bf6c60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bf6c70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb455d20) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 44 (Thread 3784.0x154c):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bf5d60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bf5d70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb444870) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 43 (Thread 3784.0x8d0):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bf4e60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bf4e70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb4333c0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 42 (Thread 3784.0x10d8):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bf3f60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bf3f70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb421f10) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 41 (Thread 3784.0xd20):
#0  0x00007ff845d06bf4 in ntdll!ZwWaitForMultipleObjects () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264796f in WaitForMultipleObjectsEx () from C:\Windows\System32\KernelBase.dll
#2  0x00007ff84264786e in WaitForMultipleObjects () from C:\Windows\System32\KernelBase.dll
#3  0x000000000046b31f in do_sema_b_wait_intern (sema=sema@entry=0xa4, nointerrupt=nointerrupt@entry=0, 
    timeout=timeout@entry=4294967295) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:625
#4  0x000000000046b60e in do_sema_b_wait (sema=0xa4, nointerrupt=0, timeout=4294967295, cs=0x1e15c0, val=0x1e15e8)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:584
#5  0x000000000046bb91 in pthread_cond_wait (c=c@entry=0x1b0008, external_mutex=external_mutex@entry=0x1b0000)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:451
#6  0x0000000000441232 in fio_mutex_down (mutex=0x1b0000) at mutex.c:224
#7  0x0000000000446eae in sfree_pool (ptr=0x2bf3420, pool=0x55c980 <mp>) at smalloc.c:312
#8  sfree (ptr=0x2bf3430) at smalloc.c:336
#9  0x000000000042df59 in close_and_free_files (td=td@entry=0xb410a60) at filesetup.c:1358
#10 0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#11 0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#12 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#13 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#14 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#15 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#16 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 40 (Thread 3784.0x1494):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bf2160, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bf2170) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb3ff5b0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 39 (Thread 3784.0xc30):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bf1260, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bf1270) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb3ee100) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 38 (Thread 3784.0x11cc):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bf0360, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bf0370) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb3dcc50) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 37 (Thread 3784.0x17d0):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bef460, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bef470) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb3cb7a0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 36 (Thread 3784.0x1420):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bee560, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bee570) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb3ba2f0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 35 (Thread 3784.0x554):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bed660, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bed670) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb3a8e40) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 34 (Thread 3784.0x1438):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bec760, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bec770) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb397990) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 33 (Thread 3784.0x1700):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bea960, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bea970) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb375030) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 32 (Thread 3784.0x50):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2be7c60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2be7c70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb341220) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 31 (Thread 3784.0x10a8):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2be6d60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2be6d70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb32fd70) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 30 (Thread 3784.0xd94):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2be5e60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2be5e70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb31e8c0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 29 (Thread 3784.0xb3c):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2be4f60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2be4f70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb30d410) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 28 (Thread 3784.0xf64):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2be4060, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2be4070) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb2fbf60) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 27 (Thread 3784.0xb60):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2be3160, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2be3170) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb2eaab0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 26 (Thread 3784.0x15a4):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2be2260, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2be2270) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb2d9600) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 25 (Thread 3784.0x4b8):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2be0460, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2be0470) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb2b6ca0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 24 (Thread 3784.0x12e0):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bdf560, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bdf570) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb2a57f0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 23 (Thread 3784.0x1404):
#0  0x00007ff845d06bf4 in ntdll!ZwWaitForMultipleObjects () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264796f in WaitForMultipleObjectsEx () from C:\Windows\System32\KernelBase.dll
#2  0x00007ff84264786e in WaitForMultipleObjects () from C:\Windows\System32\KernelBase.dll
#3  0x000000000046b31f in do_sema_b_wait_intern (sema=sema@entry=0xa4, nointerrupt=nointerrupt@entry=0, 
    timeout=timeout@entry=4294967295) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:625
#4  0x000000000046b60e in do_sema_b_wait (sema=0xa4, nointerrupt=0, timeout=4294967295, cs=0x1e15c0, val=0x1e15e8)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:584
#5  0x000000000046bb91 in pthread_cond_wait (c=c@entry=0x1b0008, external_mutex=external_mutex@entry=0x1b0000)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:451
#6  0x0000000000441232 in fio_mutex_down (mutex=0x1b0000) at mutex.c:224
#7  0x0000000000446eae in sfree_pool (ptr=0x2bdee20, pool=0x55c980 <mp>) at smalloc.c:312
#8  sfree (ptr=0x2bdee30) at smalloc.c:336
#9  0x000000000042df76 in close_and_free_files (td=td@entry=0xb294340) at filesetup.c:1367
#10 0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#11 0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#12 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#13 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#14 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#15 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#16 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 22 (Thread 3784.0x16c4):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bdd760, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bdd770) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb282e90) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 21 (Thread 3784.0x15e4):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bdc860, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bdc870) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb2719e0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 20 (Thread 3784.0x458):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bdb960, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bdb970) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb260530) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 19 (Thread 3784.0x264):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441292 in fio_mutex_up (mutex=0x1b0000) at mutex.c:238
#5  0x0000000000446feb in sfree_pool (ptr=<optimized out>, pool=<optimized out>) at smalloc.c:317
#6  sfree (ptr=<optimized out>) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb24f080) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 18 (Thread 3784.0x16e8):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bd9b60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bd9b70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb23dbd0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 17 (Thread 3784.0x14d0):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bd8c60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bd8c70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb22c720) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 16 (Thread 3784.0x1064):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bd7d60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bd7d70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb21b270) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 15 (Thread 3784.0x1550):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bd5f60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bd5f70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb1f8910) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 14 (Thread 3784.0x1024):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bd5060, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bd5070) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb1e7460) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 13 (Thread 3784.0x9b0):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bd4160, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bd4170) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb1d5fb0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 12 (Thread 3784.0xcb4):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bd3260, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bd3270) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb1c4b00) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 11 (Thread 3784.0xae4):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bd1460, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bd1470) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb1a21a0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 10 (Thread 3784.0x17d8):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bd0560, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bd0570) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb190cf0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 9 (Thread 3784.0x10b0):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bcf660, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bcf670) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb17f840) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 8 (Thread 3784.0x4e4):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bcd860, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bcd870) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb15cee0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 7 (Thread 3784.0x14f8):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bcc960, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bcc970) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb14ba30) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 6 (Thread 3784.0xb4c):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bcba60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bcba70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb13a580) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 5 (Thread 3784.0x15c4):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bcab60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bcab70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb1290d0) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 4 (Thread 3784.0x1218):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bc6f60, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bc6f70) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb0e3e10) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 3 (Thread 3784.0xd38):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bc6060, pool=0x55c980 <mp>) at smalloc.c:312
#6  sfree (ptr=0x2bc6070) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb0d2960) at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 2 (Thread 3784.0x1574):
#0  0x00007ff845d06bf4 in ntdll!ZwWaitForMultipleObjects () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264796f in WaitForMultipleObjectsEx () from C:\Windows\System32\KernelBase.dll
#2  0x00007ff84264786e in WaitForMultipleObjects () from C:\Windows\System32\KernelBase.dll
#3  0x000000000046b31f in do_sema_b_wait_intern (sema=sema@entry=0x8cc, nointerrupt=nointerrupt@entry=0, 
    timeout=timeout@entry=500) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:625
#4  0x000000000046b60e in do_sema_b_wait (sema=0x8cc, nointerrupt=0, timeout=500, cs=0xafc9a60, val=0xafc9a88)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:584
#5  0x000000000046be8e in pthread_cond_timedwait_impl (c=c@entry=0x2c2d0f8, 
    external_mutex=external_mutex@entry=0x2c2d0f0, t=t@entry=0x1ce9fe30, rel=rel@entry=0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:504
#6  0x000000000046bea8 in pthread_cond_timedwait (c=c@entry=0x2c2d0f8, m=m@entry=0x2c2d0f0, t=t@entry=0x1ce9fe30)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:515
#7  0x0000000000466978 in helper_thread_main (data=0x2c2d0d0) at helper_thread.c:95
#8  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#9  0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#10 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#12 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#13 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 1 (Thread 3784.0x16fc):
#0  0x00007ff845d06724 in ntdll!ZwDelayExecution () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842647b47 in SleepEx () from C:\Windows\System32\KernelBase.dll
#2  0x0000000000469fa8 in usleep (useconds=<optimized out>, useconds@entry=10000) at os/windows/posix.c:572
#3  0x000000000045d03a in do_usleep (usecs=10000) at backend.c:2129
#4  run_threads (sk_out=sk_out@entry=0x0) at backend.c:2444
#5  0x000000000045d3da in fio_backend (sk_out=sk_out@entry=0x0) at backend.c:2493
#6  0x0000000000480329 in main (argc=4, argv=0x1e13c0, envp=<optimized out>) at fio.c:65

Re: Windows: FIO randomly hangs using attached script

[Rob Scheepens]

Btw.: this is the error I got when trying to generate a dump:

(gdb) generate-core-file
warning: cannot close "core.3784": Invalid operation
Can't create a corefile
(gdb)

\Rob


On 08/03/2018, 16:35, "Rob Scheepens" <rob.scheepens@nutanix.com> wrote:

    Sitsofe,
    
    See attached for the fio threads' backtraces.
    
    \Rob
    
    
    On 08/03/2018, 14:39, "Rob Scheepens" <rob.scheepens@nutanix.com> wrote:
    
        I'll see if I can get gdb setup like that.
        
        Below output shows spec 176 was the last job that ran ok, so you could try if it reproduces with 177. However, the job number exhibiting the hang usually is different every time.
        
        \Rob
        
        
        On 08/03/2018, 14:29, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote:
        
            A backtrace would be a good starting point but we need your gdb to be
            able to be able to produce that... If you've managed to successfully
            attach gdb to fio running something like
            
            set logging file fiobacktrace.txt
            set logging on
            thread apply all bt
            set logging off
            
            and copying the contents of fiobacktrace.txt to here would be helpful.
            If possible also take a note of the exact job that failed and send it
            over as presumably we can just keep rerunning that to produce the
            problem.
            
            On 8 March 2018 at 10:51, Rob Scheepens <rob.scheepens@nutanix.com> wrote:
            > Just tested using fio-3.5: the first attempt already hung, at:
            >
            > PS C:\Python27> python .\profiler_win_dk.py
            > * Running job: 'config' (20 secs / spec)
            > ** Executing spec 176/405 at 2018-03-08 02:10:25.360000
            >
            > Instead of trying an large amount of variations to reproduce, which will be time-consuming, is there a way to capture a dump using gdb and debug that? I have very limited gdb skills and when I recently tried to save it off, it gave me an error that seemed to implicate some lib was not implemented on Windows.
            >
            > I left the vm in the current state, in case anyone wants me to get something from it.
            
            -- 
            Sitsofe | https://urldefense.proofpoint.com/v2/url?u=http-3A__sucs.org_-7Esits_&d=DwIFaQ&c=s883GpUCOChKOHiocYtGcg&r=OMged-t_5I_fmfpUaT3vaA06lgLL_alYnDQJxHmXz64&m=_j-o6LW-O_G6OkO-vy7HdE7UJDsy8W_csQytzi2Vg7I&s=u17mNDv5miOlk6AgsMohHL41aIameCmzLB1vhzTlxJc&e=
            
        
        
    
    

Re: Windows: FIO randomly hangs using attached script

[Sitsofe Wheeler]

I've no good ideas on that one. Could be due to permissions or lack of
space on gdb's current working directory? Perhaps take a look with pwd
and change it with cd?

On 8 March 2018 at 14:38, Rob Scheepens <rob.scheepens@nutanix.com> wrote:
> Btw.: this is the error I got when trying to generate a dump:
>
> (gdb) generate-core-file
> warning: cannot close "core.3784": Invalid operation
> Can't create a corefile
> (gdb)

-- 
Sitsofe | http://sucs.org/~sits/

Re: Windows: FIO randomly hangs using attached script

[Sitsofe Wheeler]

OK wow that was big.

The threads from 3 - 101 seem to be stuck with a backtrace that looks
similar to this:

Thread 3 (Thread 3784.0xd38):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from
C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from
C:\Windows\System32\KernelBase.dll
#2  0x000000000046c0df in pthread_mutex_lock_intern
(timeout=4294967295, m=0x1e1880)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
#3  pthread_mutex_lock (m=m@entry=0x1b0000) at
/usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
#4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
#5  0x0000000000446eae in sfree_pool (ptr=0x2bc6060, pool=0x55c980
<mp>) at smalloc.c:312
#6  sfree (ptr=0x2bc6070) at smalloc.c:336
#7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb0d2960)
at filesetup.c:1358
#8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from
C:\Windows\System32\msvcrt.dll
#11 0x00007ff845a9b38c in msvcrt!_endthreadex () from
C:\Windows\System32\msvcrt.dll
#12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
C:\Windows\System32\kernel32.dll
#13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
C:\Windows\SYSTEM32\ntdll.dll
#14 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

From thread 102 - 201 things look like this:

Thread 102 (Thread 3784.0x8d4):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from
C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from
C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0xb001bd0)
at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

For completeness here's Thread 1 and 2:

Thread 2 (Thread 3784.0x1574):
#0  0x00007ff845d06bf4 in ntdll!ZwWaitForMultipleObjects () from
C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264796f in WaitForMultipleObjectsEx () from
C:\Windows\System32\KernelBase.dll
#2  0x00007ff84264786e in WaitForMultipleObjects () from
C:\Windows\System32\KernelBase.dll
#3  0x000000000046b31f in do_sema_b_wait_intern
(sema=sema@entry=0x8cc, nointerrupt=nointerrupt@entry=0,
    timeout=timeout@entry=500) at
/usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:625
#4  0x000000000046b60e in do_sema_b_wait (sema=0x8cc, nointerrupt=0,
timeout=500, cs=0xafc9a60, val=0xafc9a88)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:584
#5  0x000000000046be8e in pthread_cond_timedwait_impl (c=c@entry=0x2c2d0f8,
    external_mutex=external_mutex@entry=0x2c2d0f0,
t=t@entry=0x1ce9fe30, rel=rel@entry=0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:504
#6  0x000000000046bea8 in pthread_cond_timedwait (c=c@entry=0x2c2d0f8,
m=m@entry=0x2c2d0f0, t=t@entry=0x1ce9fe30)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:515
#7  0x0000000000466978 in helper_thread_main (data=0x2c2d0d0) at
helper_thread.c:95
#8  0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#9  0x00007ff845a9b2ba in msvcrt!_beginthreadex () from
C:\Windows\System32\msvcrt.dll
#10 0x00007ff845a9b38c in msvcrt!_endthreadex () from
C:\Windows\System32\msvcrt.dll
#11 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
C:\Windows\System32\kernel32.dll
#12 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
C:\Windows\SYSTEM32\ntdll.dll
#13 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 1 (Thread 3784.0x16fc):
#0  0x00007ff845d06724 in ntdll!ZwDelayExecution () from
C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842647b47 in SleepEx () from C:\Windows\System32\KernelBase.dll
#2  0x0000000000469fa8 in usleep (useconds=<optimized out>,
useconds@entry=10000) at os/windows/posix.c:572
#3  0x000000000045d03a in do_usleep (usecs=10000) at backend.c:2129
#4  run_threads (sk_out=sk_out@entry=0x0) at backend.c:2444
#5  0x000000000045d3da in fio_backend (sk_out=sk_out@entry=0x0) at
backend.c:2493
#6  0x0000000000480329 in main (argc=4, argv=0x1e13c0, envp=<optimized
out>) at fio.c:65

This could take a while to figure out!

On 8 March 2018 at 14:35, Rob Scheepens <rob.scheepens@nutanix.com> wrote:
> Sitsofe,
>
> See attached for the fio threads' backtraces.
>
> \Rob
>
>
> On 08/03/2018, 14:39, "Rob Scheepens" <rob.scheepens@nutanix.com> wrote:
>
>     I'll see if I can get gdb setup like that.
>
>     Below output shows spec 176 was the last job that ran ok, so you could try if it reproduces with 177. However, the job number exhibiting the hang usually is different every time.

-- 
Sitsofe | http://sucs.org/~sits/

Re: Windows: FIO randomly hangs using attached script

[Rob Scheepens]

Just came across what seems to be an (old) similar issue, where no resolution is mentioned (last reply 4 years ago):

deadlock in fio.. client/server:
https://www.spinics.net/lists/fio/msg03325.html

\Rob

On 08/03/2018, 17:13, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote:

    OK wow that was big.
    
    The threads from 3 - 101 seem to be stuck with a backtrace that looks
    similar to this:
    
    Thread 3 (Thread 3784.0xd38):
    #0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from
    C:\Windows\SYSTEM32\ntdll.dll
    #1  0x00007ff842633acf in WaitForSingleObjectEx () from
    C:\Windows\System32\KernelBase.dll
    #2  0x000000000046c0df in pthread_mutex_lock_intern
    (timeout=4294967295, m=0x1e1880)
        at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
    #3  pthread_mutex_lock (m=m@entry=0x1b0000) at
    /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
    #4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
    #5  0x0000000000446eae in sfree_pool (ptr=0x2bc6060, pool=0x55c980
    <mp>) at smalloc.c:312
    #6  sfree (ptr=0x2bc6070) at smalloc.c:336
    #7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb0d2960)
    at filesetup.c:1358
    #8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
    #9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
        at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
    #10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from
    C:\Windows\System32\msvcrt.dll
    #11 0x00007ff845a9b38c in msvcrt!_endthreadex () from
    C:\Windows\System32\msvcrt.dll
    #12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
    C:\Windows\System32\kernel32.dll
    #13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
    C:\Windows\SYSTEM32\ntdll.dll
    #14 0x0000000000000000 in ?? ()
    Backtrace stopped: previous frame inner to this frame (corrupt stack?)
    
    From thread 102 - 201 things look like this:
    
    Thread 102 (Thread 3784.0x8d4):
    #0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from
    C:\Windows\SYSTEM32\ntdll.dll
    #1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from
    C:\Windows\System32\KernelBase.dll
    #2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0xb001bd0)
    at engines/windowsaio.c:427
    #3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
    C:\Windows\System32\kernel32.dll
    #4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
    C:\Windows\SYSTEM32\ntdll.dll
    #5  0x0000000000000000 in ?? ()
    Backtrace stopped: previous frame inner to this frame (corrupt stack?)
    
    For completeness here's Thread 1 and 2:
    
    Thread 2 (Thread 3784.0x1574):
    #0  0x00007ff845d06bf4 in ntdll!ZwWaitForMultipleObjects () from
    C:\Windows\SYSTEM32\ntdll.dll
    #1  0x00007ff84264796f in WaitForMultipleObjectsEx () from
    C:\Windows\System32\KernelBase.dll
    #2  0x00007ff84264786e in WaitForMultipleObjects () from
    C:\Windows\System32\KernelBase.dll
    #3  0x000000000046b31f in do_sema_b_wait_intern
    (sema=sema@entry=0x8cc, nointerrupt=nointerrupt@entry=0,
        timeout=timeout@entry=500) at
    /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:625
    #4  0x000000000046b60e in do_sema_b_wait (sema=0x8cc, nointerrupt=0,
    timeout=500, cs=0xafc9a60, val=0xafc9a88)
        at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:584
    #5  0x000000000046be8e in pthread_cond_timedwait_impl (c=c@entry=0x2c2d0f8,
        external_mutex=external_mutex@entry=0x2c2d0f0,
    t=t@entry=0x1ce9fe30, rel=rel@entry=0)
        at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:504
    #6  0x000000000046bea8 in pthread_cond_timedwait (c=c@entry=0x2c2d0f8,
    m=m@entry=0x2c2d0f0, t=t@entry=0x1ce9fe30)
        at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:515
    #7  0x0000000000466978 in helper_thread_main (data=0x2c2d0d0) at
    helper_thread.c:95
    #8  0x000000000046df74 in pthread_create_wrapper (args=0x0)
        at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
    #9  0x00007ff845a9b2ba in msvcrt!_beginthreadex () from
    C:\Windows\System32\msvcrt.dll
    #10 0x00007ff845a9b38c in msvcrt!_endthreadex () from
    C:\Windows\System32\msvcrt.dll
    #11 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
    C:\Windows\System32\kernel32.dll
    #12 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
    C:\Windows\SYSTEM32\ntdll.dll
    #13 0x0000000000000000 in ?? ()
    Backtrace stopped: previous frame inner to this frame (corrupt stack?)
    
    Thread 1 (Thread 3784.0x16fc):
    #0  0x00007ff845d06724 in ntdll!ZwDelayExecution () from
    C:\Windows\SYSTEM32\ntdll.dll
    #1  0x00007ff842647b47 in SleepEx () from C:\Windows\System32\KernelBase.dll
    #2  0x0000000000469fa8 in usleep (useconds=<optimized out>,
    useconds@entry=10000) at os/windows/posix.c:572
    #3  0x000000000045d03a in do_usleep (usecs=10000) at backend.c:2129
    #4  run_threads (sk_out=sk_out@entry=0x0) at backend.c:2444
    #5  0x000000000045d3da in fio_backend (sk_out=sk_out@entry=0x0) at
    backend.c:2493
    #6  0x0000000000480329 in main (argc=4, argv=0x1e13c0, envp=<optimized
    out>) at fio.c:65
    
    This could take a while to figure out!
    
    On 8 March 2018 at 14:35, Rob Scheepens <rob.scheepens@nutanix.com> wrote:
    > Sitsofe,
    >
    > See attached for the fio threads' backtraces.
    >
    > \Rob
    >
    >
    > On 08/03/2018, 14:39, "Rob Scheepens" <rob.scheepens@nutanix.com> wrote:
    >
    >     I'll see if I can get gdb setup like that.
    >
    >     Below output shows spec 176 was the last job that ran ok, so you could try if it reproduces with 177. However, the job number exhibiting the hang usually is different every time.
    
    -- 
    Sitsofe | https://urldefense.proofpoint.com/v2/url?u=http-3A__sucs.org_-7Esits_&d=DwIFaQ&c=s883GpUCOChKOHiocYtGcg&r=OMged-t_5I_fmfpUaT3vaA06lgLL_alYnDQJxHmXz64&m=klmdlx61L5-QFKKNFgbuuwgQKtwsioCr4bZcTounUQM&s=fqSnEQXZz0hNXqRMlaIEDCOfmc15dbI0YFxXxr5aHrg&e=
    

Re: Windows: FIO randomly hangs using attached script

[Sitsofe Wheeler]

On 8 March 2018 at 15:45, Rob Scheepens <rob.scheepens@nutanix.com> wrote:
> Just came across what seems to be an (old) similar issue, where no resolution is mentioned (last reply 4 years ago):
>
> deadlock in fio.. client/server:
> https://www.spinics.net/lists/fio/msg03325.html

That looks like it got fixed: https://www.spinics.net/lists/fio/msg03345.html .

-- 
Sitsofe | http://sucs.org/~sits/

Re: Windows: FIO randomly hangs using attached script

[Sitsofe Wheeler]

(I'm going to bottom post for now)

On 8 March 2018 at 15:13, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> OK wow that was big.
>
> The threads from 3 - 101 seem to be stuck with a backtrace that looks
> similar to this:
>
> Thread 3 (Thread 3784.0xd38):
> #0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from
> C:\Windows\SYSTEM32\ntdll.dll
> #1  0x00007ff842633acf in WaitForSingleObjectEx () from
> C:\Windows\System32\KernelBase.dll
> #2  0x000000000046c0df in pthread_mutex_lock_intern
> (timeout=4294967295, m=0x1e1880)
>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
> #3  pthread_mutex_lock (m=m@entry=0x1b0000) at
> /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
> #4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
> #5  0x0000000000446eae in sfree_pool (ptr=0x2bc6060, pool=0x55c980
> <mp>) at smalloc.c:312
> #6  sfree (ptr=0x2bc6070) at smalloc.c:336
> #7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb0d2960)
> at filesetup.c:1358
> #8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
> #9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
> #10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from
> C:\Windows\System32\msvcrt.dll
> #11 0x00007ff845a9b38c in msvcrt!_endthreadex () from
> C:\Windows\System32\msvcrt.dll
> #12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
> C:\Windows\System32\kernel32.dll
> #13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
> C:\Windows\SYSTEM32\ntdll.dll
> #14 0x0000000000000000 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
>
> From thread 102 - 201 things look like this:
>
> Thread 102 (Thread 3784.0x8d4):
> #0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from
> C:\Windows\SYSTEM32\ntdll.dll
> #1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from
> C:\Windows\System32\KernelBase.dll
> #2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0xb001bd0)
> at engines/windowsaio.c:427
> #3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
> C:\Windows\System32\kernel32.dll
> #4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
> C:\Windows\SYSTEM32\ntdll.dll
> #5  0x0000000000000000 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
>
> For completeness here's Thread 1 and 2:
>
> Thread 2 (Thread 3784.0x1574):
> #0  0x00007ff845d06bf4 in ntdll!ZwWaitForMultipleObjects () from
> C:\Windows\SYSTEM32\ntdll.dll
> #1  0x00007ff84264796f in WaitForMultipleObjectsEx () from
> C:\Windows\System32\KernelBase.dll
> #2  0x00007ff84264786e in WaitForMultipleObjects () from
> C:\Windows\System32\KernelBase.dll
> #3  0x000000000046b31f in do_sema_b_wait_intern
> (sema=sema@entry=0x8cc, nointerrupt=nointerrupt@entry=0,
>     timeout=timeout@entry=500) at
> /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:625
> #4  0x000000000046b60e in do_sema_b_wait (sema=0x8cc, nointerrupt=0,
> timeout=500, cs=0xafc9a60, val=0xafc9a88)
>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:584
> #5  0x000000000046be8e in pthread_cond_timedwait_impl (c=c@entry=0x2c2d0f8,
>     external_mutex=external_mutex@entry=0x2c2d0f0,
> t=t@entry=0x1ce9fe30, rel=rel@entry=0)
>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:504
> #6  0x000000000046bea8 in pthread_cond_timedwait (c=c@entry=0x2c2d0f8,
> m=m@entry=0x2c2d0f0, t=t@entry=0x1ce9fe30)
>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:515
> #7  0x0000000000466978 in helper_thread_main (data=0x2c2d0d0) at
> helper_thread.c:95
> #8  0x000000000046df74 in pthread_create_wrapper (args=0x0)
>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
> #9  0x00007ff845a9b2ba in msvcrt!_beginthreadex () from
> C:\Windows\System32\msvcrt.dll
> #10 0x00007ff845a9b38c in msvcrt!_endthreadex () from
> C:\Windows\System32\msvcrt.dll
> #11 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
> C:\Windows\System32\kernel32.dll
> #12 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
> C:\Windows\SYSTEM32\ntdll.dll
> #13 0x0000000000000000 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
>
> Thread 1 (Thread 3784.0x16fc):
> #0  0x00007ff845d06724 in ntdll!ZwDelayExecution () from
> C:\Windows\SYSTEM32\ntdll.dll
> #1  0x00007ff842647b47 in SleepEx () from C:\Windows\System32\KernelBase.dll
> #2  0x0000000000469fa8 in usleep (useconds=<optimized out>,
> useconds@entry=10000) at os/windows/posix.c:572
> #3  0x000000000045d03a in do_usleep (usecs=10000) at backend.c:2129
> #4  run_threads (sk_out=sk_out@entry=0x0) at backend.c:2444
> #5  0x000000000045d3da in fio_backend (sk_out=sk_out@entry=0x0) at
> backend.c:2493
> #6  0x0000000000480329 in main (argc=4, argv=0x1e13c0, envp=<optimized
> out>) at fio.c:65
>
> This could take a while to figure out!

Here a thread in among the others that looks different:

Thread 53 (Thread 3784.0xca0):
#0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from
C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff842633acf in WaitForSingleObjectEx () from
C:\Windows\System32\KernelBase.dll
#2  0x000000000046b4cc in do_sema_b_wait_intern (sema=sema@entry=0xa8,
nointerrupt=nointerrupt@entry=1,
    timeout=timeout@entry=4294967295) at
/usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:601
#3  0x000000000046b60e in do_sema_b_wait (sema=0xa8, nointerrupt=1,
timeout=4294967295, cs=0x1e15f0, val=0x1e1618)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:584
#4  0x000000000046b89f in pthread_cond_signal (c=<optimized out>)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:327
#5  0x00000000004412c0 in fio_mutex_up (mutex=<optimized out>) at mutex.c:246
#6  0x0000000000446feb in sfree_pool (ptr=<optimized out>,
pool=<optimized out>) at smalloc.c:317
#7  sfree (ptr=<optimized out>) at smalloc.c:336
#8  0x000000000042df59 in close_and_free_files (td=td@entry=0xb4e02a0)
at filesetup.c:1358
#9  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#10 0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#11 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from
C:\Windows\System32\msvcrt.dll
#12 0x00007ff845a9b38c in msvcrt!_endthreadex () from
C:\Windows\System32\msvcrt.dll
#13 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
C:\Windows\System32\kernel32.dll
#14 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
C:\Windows\SYSTEM32\ntdll.dll
#15 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

-- 
Sitsofe | http://sucs.org/~sits/

Re: Windows: FIO randomly hangs using attached script

[Sitsofe Wheeler]

On 8 March 2018 at 15:46, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> (I'm going to bottom post for now)
>
> On 8 March 2018 at 15:13, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>> OK wow that was big.
>>
>> The threads from 3 - 101 seem to be stuck with a backtrace that looks
>> similar to this:
>>
>> Thread 3 (Thread 3784.0xd38):
>> #0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from
>> C:\Windows\SYSTEM32\ntdll.dll
>> #1  0x00007ff842633acf in WaitForSingleObjectEx () from
>> C:\Windows\System32\KernelBase.dll
>> #2  0x000000000046c0df in pthread_mutex_lock_intern
>> (timeout=4294967295, m=0x1e1880)
>>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
>> #3  pthread_mutex_lock (m=m@entry=0x1b0000) at
>> /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
>> #4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
>> #5  0x0000000000446eae in sfree_pool (ptr=0x2bc6060, pool=0x55c980
>> <mp>) at smalloc.c:312
>> #6  sfree (ptr=0x2bc6070) at smalloc.c:336
>> #7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb0d2960)
>> at filesetup.c:1358
>> #8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
>> #9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
>>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
>> #10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from
>> C:\Windows\System32\msvcrt.dll
>> #11 0x00007ff845a9b38c in msvcrt!_endthreadex () from
>> C:\Windows\System32\msvcrt.dll
>> #12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
>> C:\Windows\System32\kernel32.dll
>> #13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
>> C:\Windows\SYSTEM32\ntdll.dll
>> #14 0x0000000000000000 in ?? ()
>> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
>>
>> From thread 102 - 201 things look like this:
>>
>> Thread 102 (Thread 3784.0x8d4):
>> #0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from
>> C:\Windows\SYSTEM32\ntdll.dll
>> #1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from
>> C:\Windows\System32\KernelBase.dll
>> #2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0xb001bd0)
>> at engines/windowsaio.c:427
>> #3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
>> C:\Windows\System32\kernel32.dll
>> #4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
>> C:\Windows\SYSTEM32\ntdll.dll
>> #5  0x0000000000000000 in ?? ()
>> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
>>
>> For completeness here's Thread 1 and 2:
>>
>> Thread 2 (Thread 3784.0x1574):
>> #0  0x00007ff845d06bf4 in ntdll!ZwWaitForMultipleObjects () from
>> C:\Windows\SYSTEM32\ntdll.dll
>> #1  0x00007ff84264796f in WaitForMultipleObjectsEx () from
>> C:\Windows\System32\KernelBase.dll
>> #2  0x00007ff84264786e in WaitForMultipleObjects () from
>> C:\Windows\System32\KernelBase.dll
>> #3  0x000000000046b31f in do_sema_b_wait_intern
>> (sema=sema@entry=0x8cc, nointerrupt=nointerrupt@entry=0,
>>     timeout=timeout@entry=500) at
>> /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:625
>> #4  0x000000000046b60e in do_sema_b_wait (sema=0x8cc, nointerrupt=0,
>> timeout=500, cs=0xafc9a60, val=0xafc9a88)
>>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:584
>> #5  0x000000000046be8e in pthread_cond_timedwait_impl (c=c@entry=0x2c2d0f8,
>>     external_mutex=external_mutex@entry=0x2c2d0f0,
>> t=t@entry=0x1ce9fe30, rel=rel@entry=0)
>>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:504
>> #6  0x000000000046bea8 in pthread_cond_timedwait (c=c@entry=0x2c2d0f8,
>> m=m@entry=0x2c2d0f0, t=t@entry=0x1ce9fe30)
>>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:515
>> #7  0x0000000000466978 in helper_thread_main (data=0x2c2d0d0) at
>> helper_thread.c:95
>> #8  0x000000000046df74 in pthread_create_wrapper (args=0x0)
>>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
>> #9  0x00007ff845a9b2ba in msvcrt!_beginthreadex () from
>> C:\Windows\System32\msvcrt.dll
>> #10 0x00007ff845a9b38c in msvcrt!_endthreadex () from
>> C:\Windows\System32\msvcrt.dll
>> #11 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
>> C:\Windows\System32\kernel32.dll
>> #12 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
>> C:\Windows\SYSTEM32\ntdll.dll
>> #13 0x0000000000000000 in ?? ()
>> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
>>
>> Thread 1 (Thread 3784.0x16fc):
>> #0  0x00007ff845d06724 in ntdll!ZwDelayExecution () from
>> C:\Windows\SYSTEM32\ntdll.dll
>> #1  0x00007ff842647b47 in SleepEx () from C:\Windows\System32\KernelBase.dll
>> #2  0x0000000000469fa8 in usleep (useconds=<optimized out>,
>> useconds@entry=10000) at os/windows/posix.c:572
>> #3  0x000000000045d03a in do_usleep (usecs=10000) at backend.c:2129
>> #4  run_threads (sk_out=sk_out@entry=0x0) at backend.c:2444
>> #5  0x000000000045d3da in fio_backend (sk_out=sk_out@entry=0x0) at
>> backend.c:2493
>> #6  0x0000000000480329 in main (argc=4, argv=0x1e13c0, envp=<optimized
>> out>) at fio.c:65
>>
>> This could take a while to figure out!
>
> Here a thread in among the others that looks different:
>
> Thread 53 (Thread 3784.0xca0):
> #0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from
> C:\Windows\SYSTEM32\ntdll.dll
> #1  0x00007ff842633acf in WaitForSingleObjectEx () from
> C:\Windows\System32\KernelBase.dll
> #2  0x000000000046b4cc in do_sema_b_wait_intern (sema=sema@entry=0xa8,
> nointerrupt=nointerrupt@entry=1,
>     timeout=timeout@entry=4294967295) at
> /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:601
> #3  0x000000000046b60e in do_sema_b_wait (sema=0xa8, nointerrupt=1,
> timeout=4294967295, cs=0x1e15f0, val=0x1e1618)
>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:584
> #4  0x000000000046b89f in pthread_cond_signal (c=<optimized out>)
>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:327
> #5  0x00000000004412c0 in fio_mutex_up (mutex=<optimized out>) at mutex.c:246
> #6  0x0000000000446feb in sfree_pool (ptr=<optimized out>,
> pool=<optimized out>) at smalloc.c:317
> #7  sfree (ptr=<optimized out>) at smalloc.c:336
> #8  0x000000000042df59 in close_and_free_files (td=td@entry=0xb4e02a0)
> at filesetup.c:1358
> #9  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
> #10 0x000000000046df74 in pthread_create_wrapper (args=0x0)
>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
> #11 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from
> C:\Windows\System32\msvcrt.dll
> #12 0x00007ff845a9b38c in msvcrt!_endthreadex () from
> C:\Windows\System32\msvcrt.dll
> #13 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
> C:\Windows\System32\kernel32.dll
> #14 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
> C:\Windows\SYSTEM32\ntdll.dll
> #15 0x0000000000000000 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)

So it looks like the smalloc.c pool->lock was heavily contended. One
thread downed the mutex, got into the critical section, did its thing
then after unlocking the mutex got stuck trying to let one of the
people who might be waiting to wake up:

mutex.c
243         pthread_mutex_unlock(&mutex->lock);
244
245         if (do_wake)
246                 pthread_cond_signal(&mutex->cond);

This is odd. Why would a thread get stuck on that call?

-- 
Sitsofe | http://sucs.org/~sits/

Re: Windows: FIO randomly hangs using attached script

[Jens Axboe]

On 3/8/18 8:59 AM, Sitsofe Wheeler wrote:
> On 8 March 2018 at 15:46, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>> (I'm going to bottom post for now)
>>
>> On 8 March 2018 at 15:13, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>>> OK wow that was big.
>>>
>>> The threads from 3 - 101 seem to be stuck with a backtrace that looks
>>> similar to this:
>>>
>>> Thread 3 (Thread 3784.0xd38):
>>> #0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from
>>> C:\Windows\SYSTEM32\ntdll.dll
>>> #1  0x00007ff842633acf in WaitForSingleObjectEx () from
>>> C:\Windows\System32\KernelBase.dll
>>> #2  0x000000000046c0df in pthread_mutex_lock_intern
>>> (timeout=4294967295, m=0x1e1880)
>>>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
>>> #3  pthread_mutex_lock (m=m@entry=0x1b0000) at
>>> /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
>>> #4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
>>> #5  0x0000000000446eae in sfree_pool (ptr=0x2bc6060, pool=0x55c980
>>> <mp>) at smalloc.c:312
>>> #6  sfree (ptr=0x2bc6070) at smalloc.c:336
>>> #7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb0d2960)
>>> at filesetup.c:1358
>>> #8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
>>> #9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
>>>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
>>> #10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from
>>> C:\Windows\System32\msvcrt.dll
>>> #11 0x00007ff845a9b38c in msvcrt!_endthreadex () from
>>> C:\Windows\System32\msvcrt.dll
>>> #12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
>>> C:\Windows\System32\kernel32.dll
>>> #13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
>>> C:\Windows\SYSTEM32\ntdll.dll
>>> #14 0x0000000000000000 in ?? ()
>>> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
>>>
>>> From thread 102 - 201 things look like this:
>>>
>>> Thread 102 (Thread 3784.0x8d4):
>>> #0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from
>>> C:\Windows\SYSTEM32\ntdll.dll
>>> #1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from
>>> C:\Windows\System32\KernelBase.dll
>>> #2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0xb001bd0)
>>> at engines/windowsaio.c:427
>>> #3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
>>> C:\Windows\System32\kernel32.dll
>>> #4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
>>> C:\Windows\SYSTEM32\ntdll.dll
>>> #5  0x0000000000000000 in ?? ()
>>> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
>>>
>>> For completeness here's Thread 1 and 2:
>>>
>>> Thread 2 (Thread 3784.0x1574):
>>> #0  0x00007ff845d06bf4 in ntdll!ZwWaitForMultipleObjects () from
>>> C:\Windows\SYSTEM32\ntdll.dll
>>> #1  0x00007ff84264796f in WaitForMultipleObjectsEx () from
>>> C:\Windows\System32\KernelBase.dll
>>> #2  0x00007ff84264786e in WaitForMultipleObjects () from
>>> C:\Windows\System32\KernelBase.dll
>>> #3  0x000000000046b31f in do_sema_b_wait_intern
>>> (sema=sema@entry=0x8cc, nointerrupt=nointerrupt@entry=0,
>>>     timeout=timeout@entry=500) at
>>> /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:625
>>> #4  0x000000000046b60e in do_sema_b_wait (sema=0x8cc, nointerrupt=0,
>>> timeout=500, cs=0xafc9a60, val=0xafc9a88)
>>>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:584
>>> #5  0x000000000046be8e in pthread_cond_timedwait_impl (c=c@entry=0x2c2d0f8,
>>>     external_mutex=external_mutex@entry=0x2c2d0f0,
>>> t=t@entry=0x1ce9fe30, rel=rel@entry=0)
>>>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:504
>>> #6  0x000000000046bea8 in pthread_cond_timedwait (c=c@entry=0x2c2d0f8,
>>> m=m@entry=0x2c2d0f0, t=t@entry=0x1ce9fe30)
>>>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:515
>>> #7  0x0000000000466978 in helper_thread_main (data=0x2c2d0d0) at
>>> helper_thread.c:95
>>> #8  0x000000000046df74 in pthread_create_wrapper (args=0x0)
>>>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
>>> #9  0x00007ff845a9b2ba in msvcrt!_beginthreadex () from
>>> C:\Windows\System32\msvcrt.dll
>>> #10 0x00007ff845a9b38c in msvcrt!_endthreadex () from
>>> C:\Windows\System32\msvcrt.dll
>>> #11 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
>>> C:\Windows\System32\kernel32.dll
>>> #12 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
>>> C:\Windows\SYSTEM32\ntdll.dll
>>> #13 0x0000000000000000 in ?? ()
>>> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
>>>
>>> Thread 1 (Thread 3784.0x16fc):
>>> #0  0x00007ff845d06724 in ntdll!ZwDelayExecution () from
>>> C:\Windows\SYSTEM32\ntdll.dll
>>> #1  0x00007ff842647b47 in SleepEx () from C:\Windows\System32\KernelBase.dll
>>> #2  0x0000000000469fa8 in usleep (useconds=<optimized out>,
>>> useconds@entry=10000) at os/windows/posix.c:572
>>> #3  0x000000000045d03a in do_usleep (usecs=10000) at backend.c:2129
>>> #4  run_threads (sk_out=sk_out@entry=0x0) at backend.c:2444
>>> #5  0x000000000045d3da in fio_backend (sk_out=sk_out@entry=0x0) at
>>> backend.c:2493
>>> #6  0x0000000000480329 in main (argc=4, argv=0x1e13c0, envp=<optimized
>>> out>) at fio.c:65
>>>
>>> This could take a while to figure out!
>>
>> Here a thread in among the others that looks different:
>>
>> Thread 53 (Thread 3784.0xca0):
>> #0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from
>> C:\Windows\SYSTEM32\ntdll.dll
>> #1  0x00007ff842633acf in WaitForSingleObjectEx () from
>> C:\Windows\System32\KernelBase.dll
>> #2  0x000000000046b4cc in do_sema_b_wait_intern (sema=sema@entry=0xa8,
>> nointerrupt=nointerrupt@entry=1,
>>     timeout=timeout@entry=4294967295) at
>> /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:601
>> #3  0x000000000046b60e in do_sema_b_wait (sema=0xa8, nointerrupt=1,
>> timeout=4294967295, cs=0x1e15f0, val=0x1e1618)
>>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:584
>> #4  0x000000000046b89f in pthread_cond_signal (c=<optimized out>)
>>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:327
>> #5  0x00000000004412c0 in fio_mutex_up (mutex=<optimized out>) at mutex.c:246
>> #6  0x0000000000446feb in sfree_pool (ptr=<optimized out>,
>> pool=<optimized out>) at smalloc.c:317
>> #7  sfree (ptr=<optimized out>) at smalloc.c:336
>> #8  0x000000000042df59 in close_and_free_files (td=td@entry=0xb4e02a0)
>> at filesetup.c:1358
>> #9  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
>> #10 0x000000000046df74 in pthread_create_wrapper (args=0x0)
>>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
>> #11 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from
>> C:\Windows\System32\msvcrt.dll
>> #12 0x00007ff845a9b38c in msvcrt!_endthreadex () from
>> C:\Windows\System32\msvcrt.dll
>> #13 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
>> C:\Windows\System32\kernel32.dll
>> #14 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
>> C:\Windows\SYSTEM32\ntdll.dll
>> #15 0x0000000000000000 in ?? ()
>> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> 
> So it looks like the smalloc.c pool->lock was heavily contended. One
> thread downed the mutex, got into the critical section, did its thing
> then after unlocking the mutex got stuck trying to let one of the
> people who might be waiting to wake up:
> 
> mutex.c
> 243         pthread_mutex_unlock(&mutex->lock);
> 244
> 245         if (do_wake)
> 246                 pthread_cond_signal(&mutex->cond);
> 
> This is odd. Why would a thread get stuck on that call?

Does the below patch make a difference?


diff --git a/mutex.c b/mutex.c
index 63229eda09d6..acc88dc33b98 100644
--- a/mutex.c
+++ b/mutex.c
@@ -240,10 +240,11 @@ void fio_mutex_up(struct fio_mutex *mutex)
 	if (!mutex->value && mutex->waiters)
 		do_wake = 1;
 	mutex->value++;
-	pthread_mutex_unlock(&mutex->lock);
 
 	if (do_wake)
 		pthread_cond_signal(&mutex->cond);
+
+	pthread_mutex_unlock(&mutex->lock);
 }
 
 void fio_rwlock_write(struct fio_rwlock *lock)

-- 
Jens Axboe

Re: Windows: FIO randomly hangs using attached script

[Sitsofe Wheeler]

On 8 March 2018 at 16:18, Jens Axboe <axboe@kernel.dk> wrote:
> Does the below patch make a difference?
>
>
> diff --git a/mutex.c b/mutex.c
> index 63229eda09d6..acc88dc33b98 100644
> --- a/mutex.c
> +++ b/mutex.c
> @@ -240,10 +240,11 @@ void fio_mutex_up(struct fio_mutex *mutex)
>         if (!mutex->value && mutex->waiters)
>                 do_wake = 1;
>         mutex->value++;
> -       pthread_mutex_unlock(&mutex->lock);
>
>         if (do_wake)
>                 pthread_cond_signal(&mutex->cond);
> +
> +       pthread_mutex_unlock(&mutex->lock);
>  }
>
>  void fio_rwlock_write(struct fio_rwlock *lock)

It pains me to say this (because POSIX says such rejigging just
changes the scheduling order) but yes your patch makes a difference.
The following job would trigger the deadlock problem within 10 minutes
for me:

[global]
thread
ioengine=windowsaio
direct=1
iodepth=1
readwrite=read
time_based=1
filesize=1M
runtime=4
time_based
numjobs=500
group_reporting
[job1]
filename=file1:file2:file3:file4:file5:file6:file7:file8:file9:file10:file11:file12:file13:file14:file15

for i in {1..150}; do echo "Loop $i; $(date)"; ./fio --minimal
hang.fio; if [[ $? -ne 0 ]]; then echo "failure"; break; fi; done

The backtrace is always a before - most threads are waiting trying to
lock the mutex, a few are still doing IO completion port work in
windowsaio but just keep timing out and don't get events, a couple are
waiting in pthread_cond_wait() and the thread that is trying to send
pthread_cond_signal() is deadlocked. Not using direct=1 seems to make
the problem go away for some reason and having less threads also makes
the problem harder to hit.

With your patch the above seemed to be able to go to 150 loops without
issue but I haven't tested in anger.

-- 
Sitsofe | http://sucs.org/~sits/

Re: Windows: FIO randomly hangs using attached script

[Jens Axboe]

On 3/9/18 7:40 AM, Sitsofe Wheeler wrote:
> On 8 March 2018 at 16:18, Jens Axboe <axboe@kernel.dk> wrote:
>> Does the below patch make a difference?
>>
>>
>> diff --git a/mutex.c b/mutex.c
>> index 63229eda09d6..acc88dc33b98 100644
>> --- a/mutex.c
>> +++ b/mutex.c
>> @@ -240,10 +240,11 @@ void fio_mutex_up(struct fio_mutex *mutex)
>>         if (!mutex->value && mutex->waiters)
>>                 do_wake = 1;
>>         mutex->value++;
>> -       pthread_mutex_unlock(&mutex->lock);
>>
>>         if (do_wake)
>>                 pthread_cond_signal(&mutex->cond);
>> +
>> +       pthread_mutex_unlock(&mutex->lock);
>>  }
>>
>>  void fio_rwlock_write(struct fio_rwlock *lock)
> 
> It pains me to say this (because POSIX says such rejigging just
> changes the scheduling order) but yes your patch makes a difference.
> The following job would trigger the deadlock problem within 10 minutes
> for me:

In some implementations it's actually mandated to have the wakeup
within the lock, which seems to be the case here. It's a shame,
since it's clearly suboptimal (from a scalability point of view)
to have to hold the lock while issuing a wakeup for a process
that's going to grab the same lock.

I'll commit the patch. Thanks a lot for all your hard work on this
one, let's hope that was it...

-- 
Jens Axboe

Re: Windows: FIO randomly hangs using attached script

[Sitsofe Wheeler]

On 9 March 2018 at 14:55, Jens Axboe <axboe@kernel.dk> wrote:
>
> In some implementations it's actually mandated to have the wakeup
> within the lock, which seems to be the case here. It's a shame,
> since it's clearly suboptimal (from a scalability point of view)
> to have to hold the lock while issuing a wakeup for a process
> that's going to grab the same lock.
>
> I'll commit the patch. Thanks a lot for all your hard work on this
> one, let's hope that was it...

FWIW this has been accepted as a bug in the mingw-w64 project's
winpthreads library. The standalone program on
https://gist.github.com/sitsofe/abd99efec507b18234e7cc042d1baff1
reliably triggers the deadlock for me when using a mingw build on both
real Windows and under Wine but not when built for Linux with glibc.
See https://sourceforge.net/p/mingw-w64/bugs/710/ for the "Deadlock in
winpthreads' pthread_cond_signal()" bug.

Rob, David, Rebecca: do you know if the current git fio cures the
deadlock you were seeing?

-- 
Sitsofe | http://sucs.org/~sits/

Re: Windows: FIO randomly hangs using attached script

[Rob Scheepens]

Closing the loop on this: after running a couple of million times the script that used to reproduce this, I have not seen a single error with the patch.

Thanks All for the great work on this!

\Rob


On 16/03/2018, 09:14, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote:

    On 9 March 2018 at 14:55, Jens Axboe <axboe@kernel.dk> wrote:
    >
    > In some implementations it's actually mandated to have the wakeup
    > within the lock, which seems to be the case here. It's a shame,
    > since it's clearly suboptimal (from a scalability point of view)
    > to have to hold the lock while issuing a wakeup for a process
    > that's going to grab the same lock.
    >
    > I'll commit the patch. Thanks a lot for all your hard work on this
    > one, let's hope that was it...
    
    FWIW this has been accepted as a bug in the mingw-w64 project's
    winpthreads library. The standalone program on
    https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_sitsofe_abd99efec507b18234e7cc042d1baff1&d=DwIBaQ&c=s883GpUCOChKOHiocYtGcg&r=OMged-t_5I_fmfpUaT3vaA06lgLL_alYnDQJxHmXz64&m=C-SjzVSuoSBlexryZgGqFEfxhSdPbozUDu3U82Yjmqs&s=Xri6zcEiOde_B9rOwAyOAqm92oyNy-W5DHk01BPTQ_I&e=
    reliably triggers the deadlock for me when using a mingw build on both
    real Windows and under Wine but not when built for Linux with glibc.
    See https://urldefense.proofpoint.com/v2/url?u=https-3A__sourceforge.net_p_mingw-2Dw64_bugs_710_&d=DwIBaQ&c=s883GpUCOChKOHiocYtGcg&r=OMged-t_5I_fmfpUaT3vaA06lgLL_alYnDQJxHmXz64&m=C-SjzVSuoSBlexryZgGqFEfxhSdPbozUDu3U82Yjmqs&s=Ms3hnbtg0t-Ycfm-kLxkR0pvsG1BpG6Ch-oyrsk6h4Y&e= for the "Deadlock in
    winpthreads' pthread_cond_signal()" bug.
    
    Rob, David, Rebecca: do you know if the current git fio cures the
    deadlock you were seeing?
    
    -- 
    Sitsofe | https://urldefense.proofpoint.com/v2/url?u=http-3A__sucs.org_-7Esits_&d=DwIBaQ&c=s883GpUCOChKOHiocYtGcg&r=OMged-t_5I_fmfpUaT3vaA06lgLL_alYnDQJxHmXz64&m=C-SjzVSuoSBlexryZgGqFEfxhSdPbozUDu3U82Yjmqs&s=Chc6sQtGY11L4YRP_re07gL3JfHJosjt_F6tVk-WdsQ&e=
    

Re: Windows: FIO randomly hangs using attached script

[Jens Axboe]

Thanks for the update, a couple million iterations should suffice in
confidently saying it's fixed :-)


On 5/1/18 8:41 AM, Rob Scheepens wrote:
> Closing the loop on this: after running a couple of million times the script that used to reproduce this, I have not seen a single error with the patch.
> 
> Thanks All for the great work on this!
> 
> \Rob
> 
> 
> On 16/03/2018, 09:14, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote:
> 
>     On 9 March 2018 at 14:55, Jens Axboe <axboe@kernel.dk> wrote:
>     >
>     > In some implementations it's actually mandated to have the wakeup
>     > within the lock, which seems to be the case here. It's a shame,
>     > since it's clearly suboptimal (from a scalability point of view)
>     > to have to hold the lock while issuing a wakeup for a process
>     > that's going to grab the same lock.
>     >
>     > I'll commit the patch. Thanks a lot for all your hard work on this
>     > one, let's hope that was it...
>     
>     FWIW this has been accepted as a bug in the mingw-w64 project's
>     winpthreads library. The standalone program on
>     https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_sitsofe_abd99efec507b18234e7cc042d1baff1&d=DwIBaQ&c=s883GpUCOChKOHiocYtGcg&r=OMged-t_5I_fmfpUaT3vaA06lgLL_alYnDQJxHmXz64&m=C-SjzVSuoSBlexryZgGqFEfxhSdPbozUDu3U82Yjmqs&s=Xri6zcEiOde_B9rOwAyOAqm92oyNy-W5DHk01BPTQ_I&e=
>     reliably triggers the deadlock for me when using a mingw build on both
>     real Windows and under Wine but not when built for Linux with glibc.
>     See https://urldefense.proofpoint.com/v2/url?u=https-3A__sourceforge.net_p_mingw-2Dw64_bugs_710_&d=DwIBaQ&c=s883GpUCOChKOHiocYtGcg&r=OMged-t_5I_fmfpUaT3vaA06lgLL_alYnDQJxHmXz64&m=C-SjzVSuoSBlexryZgGqFEfxhSdPbozUDu3U82Yjmqs&s=Ms3hnbtg0t-Ycfm-kLxkR0pvsG1BpG6Ch-oyrsk6h4Y&e= for the "Deadlock in
>     winpthreads' pthread_cond_signal()" bug.
>     
>     Rob, David, Rebecca: do you know if the current git fio cures the
>     deadlock you were seeing?
>     
>     -- 
>     Sitsofe | https://urldefense.proofpoint.com/v2/url?u=http-3A__sucs.org_-7Esits_&d=DwIBaQ&c=s883GpUCOChKOHiocYtGcg&r=OMged-t_5I_fmfpUaT3vaA06lgLL_alYnDQJxHmXz64&m=C-SjzVSuoSBlexryZgGqFEfxhSdPbozUDu3U82Yjmqs&s=Chc6sQtGY11L4YRP_re07gL3JfHJosjt_F6tVk-WdsQ&e=
>     
> 


-- 
Jens Axboe

Re: Windows: FIO randomly hangs using attached script

[Sitsofe Wheeler]

On 1 May 2018 at 15:41, Rob Scheepens <rob.scheepens@nutanix.com> wrote:
> Closing the loop on this: after running a couple of million times the script that used to reproduce this, I have not seen a single error with the patch.
>
> Thanks All for the great work on this!

Brilliant - good to know Jens nailed it!

-- 
Sitsofe | http://sucs.org/~sits/

Re: Windows: FIO randomly hangs using attached script

[Sitsofe Wheeler]

On 8 March 2018 at 15:59, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> On 8 March 2018 at 15:46, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>> (I'm going to bottom post for now)
>>
>> On 8 March 2018 at 15:13, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>>> OK wow that was big.
>>>
>>> The threads from 3 - 101 seem to be stuck with a backtrace that looks
>>> similar to this:
>>>
>>> Thread 3 (Thread 3784.0xd38):
>>> #0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from
>>> C:\Windows\SYSTEM32\ntdll.dll
>>> #1  0x00007ff842633acf in WaitForSingleObjectEx () from
>>> C:\Windows\System32\KernelBase.dll
>>> #2  0x000000000046c0df in pthread_mutex_lock_intern
>>> (timeout=4294967295, m=0x1e1880)
>>>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167
>>> #3  pthread_mutex_lock (m=m@entry=0x1b0000) at
>>> /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188
>>> #4  0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220
>>> #5  0x0000000000446eae in sfree_pool (ptr=0x2bc6060, pool=0x55c980
>>> <mp>) at smalloc.c:312
>>> #6  sfree (ptr=0x2bc6070) at smalloc.c:336
>>> #7  0x000000000042df59 in close_and_free_files (td=td@entry=0xb0d2960)
>>> at filesetup.c:1358
>>> #8  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
>>> #9  0x000000000046df74 in pthread_create_wrapper (args=0x0)
>>>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
>>> #10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from
>>> C:\Windows\System32\msvcrt.dll
>>> #11 0x00007ff845a9b38c in msvcrt!_endthreadex () from
>>> C:\Windows\System32\msvcrt.dll
>>> #12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
>>> C:\Windows\System32\kernel32.dll
>>> #13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
>>> C:\Windows\SYSTEM32\ntdll.dll
>>> #14 0x0000000000000000 in ?? ()
>>> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
>>>
>>> From thread 102 - 201 things look like this:
>>>
>>> Thread 102 (Thread 3784.0x8d4):
>>> #0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from
>>> C:\Windows\SYSTEM32\ntdll.dll
>>> #1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from
>>> C:\Windows\System32\KernelBase.dll
>>> #2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0xb001bd0)
>>> at engines/windowsaio.c:427
>>> #3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
>>> C:\Windows\System32\kernel32.dll
>>> #4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
>>> C:\Windows\SYSTEM32\ntdll.dll
>>> #5  0x0000000000000000 in ?? ()
>>> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
>>>
>>> For completeness here's Thread 1 and 2:
>>>
>>> Thread 2 (Thread 3784.0x1574):
>>> #0  0x00007ff845d06bf4 in ntdll!ZwWaitForMultipleObjects () from
>>> C:\Windows\SYSTEM32\ntdll.dll
>>> #1  0x00007ff84264796f in WaitForMultipleObjectsEx () from
>>> C:\Windows\System32\KernelBase.dll
>>> #2  0x00007ff84264786e in WaitForMultipleObjects () from
>>> C:\Windows\System32\KernelBase.dll
>>> #3  0x000000000046b31f in do_sema_b_wait_intern
>>> (sema=sema@entry=0x8cc, nointerrupt=nointerrupt@entry=0,
>>>     timeout=timeout@entry=500) at
>>> /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:625
>>> #4  0x000000000046b60e in do_sema_b_wait (sema=0x8cc, nointerrupt=0,
>>> timeout=500, cs=0xafc9a60, val=0xafc9a88)
>>>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:584
>>> #5  0x000000000046be8e in pthread_cond_timedwait_impl (c=c@entry=0x2c2d0f8,
>>>     external_mutex=external_mutex@entry=0x2c2d0f0,
>>> t=t@entry=0x1ce9fe30, rel=rel@entry=0)
>>>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:504
>>> #6  0x000000000046bea8 in pthread_cond_timedwait (c=c@entry=0x2c2d0f8,
>>> m=m@entry=0x2c2d0f0, t=t@entry=0x1ce9fe30)
>>>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:515
>>> #7  0x0000000000466978 in helper_thread_main (data=0x2c2d0d0) at
>>> helper_thread.c:95
>>> #8  0x000000000046df74 in pthread_create_wrapper (args=0x0)
>>>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
>>> #9  0x00007ff845a9b2ba in msvcrt!_beginthreadex () from
>>> C:\Windows\System32\msvcrt.dll
>>> #10 0x00007ff845a9b38c in msvcrt!_endthreadex () from
>>> C:\Windows\System32\msvcrt.dll
>>> #11 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
>>> C:\Windows\System32\kernel32.dll
>>> #12 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
>>> C:\Windows\SYSTEM32\ntdll.dll
>>> #13 0x0000000000000000 in ?? ()
>>> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
>>>
>>> Thread 1 (Thread 3784.0x16fc):
>>> #0  0x00007ff845d06724 in ntdll!ZwDelayExecution () from
>>> C:\Windows\SYSTEM32\ntdll.dll
>>> #1  0x00007ff842647b47 in SleepEx () from C:\Windows\System32\KernelBase.dll
>>> #2  0x0000000000469fa8 in usleep (useconds=<optimized out>,
>>> useconds@entry=10000) at os/windows/posix.c:572
>>> #3  0x000000000045d03a in do_usleep (usecs=10000) at backend.c:2129
>>> #4  run_threads (sk_out=sk_out@entry=0x0) at backend.c:2444
>>> #5  0x000000000045d3da in fio_backend (sk_out=sk_out@entry=0x0) at
>>> backend.c:2493
>>> #6  0x0000000000480329 in main (argc=4, argv=0x1e13c0, envp=<optimized
>>> out>) at fio.c:65
>>>
>>> This could take a while to figure out!
>>
>> Here a thread in among the others that looks different:
>>
>> Thread 53 (Thread 3784.0xca0):
>> #0  0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from
>> C:\Windows\SYSTEM32\ntdll.dll
>> #1  0x00007ff842633acf in WaitForSingleObjectEx () from
>> C:\Windows\System32\KernelBase.dll
>> #2  0x000000000046b4cc in do_sema_b_wait_intern (sema=sema@entry=0xa8,
>> nointerrupt=nointerrupt@entry=1,
>>     timeout=timeout@entry=4294967295) at
>> /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:601
>> #3  0x000000000046b60e in do_sema_b_wait (sema=0xa8, nointerrupt=1,
>> timeout=4294967295, cs=0x1e15f0, val=0x1e1618)
>>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:584
>> #4  0x000000000046b89f in pthread_cond_signal (c=<optimized out>)
>>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:327
>> #5  0x00000000004412c0 in fio_mutex_up (mutex=<optimized out>) at mutex.c:246
>> #6  0x0000000000446feb in sfree_pool (ptr=<optimized out>,
>> pool=<optimized out>) at smalloc.c:317
>> #7  sfree (ptr=<optimized out>) at smalloc.c:336
>> #8  0x000000000042df59 in close_and_free_files (td=td@entry=0xb4e02a0)
>> at filesetup.c:1358
>> #9  0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
>> #10 0x000000000046df74 in pthread_create_wrapper (args=0x0)
>>     at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
>> #11 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from
>> C:\Windows\System32\msvcrt.dll
>> #12 0x00007ff845a9b38c in msvcrt!_endthreadex () from
>> C:\Windows\System32\msvcrt.dll
>> #13 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
>> C:\Windows\System32\kernel32.dll
>> #14 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
>> C:\Windows\SYSTEM32\ntdll.dll
>> #15 0x0000000000000000 in ?? ()
>> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
>
> So it looks like the smalloc.c pool->lock was heavily contended. One
> thread downed the mutex, got into the critical section, did its thing
> then after unlocking the mutex got stuck trying to let one of the
> people who might be waiting to wake up:
>
> mutex.c
> 243         pthread_mutex_unlock(&mutex->lock);
> 244
> 245         if (do_wake)
> 246                 pthread_cond_signal(&mutex->cond);
>
> This is odd. Why would a thread get stuck on that call?

Things I missed previously:

Thread 91 (Thread 3784.0x1528):
#0  0x00007ff845d09814 in ntdll!ZwWaitForAlertByThreadId () from
C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff845c8faa7 in ntdll!RtlpUnWaitCriticalSection () from
C:\Windows\SYSTEM32\ntdll.dll
#2  0x00007ff845c8f9ae in ntdll!RtlpUnWaitCriticalSection () from
C:\Windows\SYSTEM32\ntdll.dll
#3  0x00007ff845c8f83f in ntdll!RtlpUnWaitCriticalSection () from
C:\Windows\SYSTEM32\ntdll.dll
#4  0x00007ff845c90d04 in ntdll!RtlEnterCriticalSection () from
C:\Windows\SYSTEM32\ntdll.dll
#5  0x00007ff845c90c30 in ntdll!RtlEnterCriticalSection () from
C:\Windows\SYSTEM32\ntdll.dll
#6  0x000000000046baaf in pthread_cond_wait (c=c@entry=0x1b0008,
external_mutex=external_mutex@entry=0x1b0000)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:437
#7  0x0000000000441232 in fio_mutex_down (mutex=0x1b0000) at mutex.c:224
#8  0x0000000000446eae in sfree_pool (ptr=0x2c22bc0, pool=0x55c980
<mp>) at smalloc.c:312
#9  sfree (ptr=0x2c22bd0) at smalloc.c:336
#10 0x000000000042df76 in close_and_free_files (td=td@entry=0xb782970)
at filesetup.c:1367
#11 0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#12 0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#13 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from
C:\Windows\System32\msvcrt.dll
#14 0x00007ff845a9b38c in msvcrt!_endthreadex () from
C:\Windows\System32\msvcrt.dll
#15 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
C:\Windows\System32\kernel32.dll
#16 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
C:\Windows\SYSTEM32\ntdll.dll
#17 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 41 (Thread 3784.0xd20):
#0  0x00007ff845d06bf4 in ntdll!ZwWaitForMultipleObjects () from
C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264796f in WaitForMultipleObjectsEx () from
C:\Windows\System32\KernelBase.dll
#2  0x00007ff84264786e in WaitForMultipleObjects () from
C:\Windows\System32\KernelBase.dll
#3  0x000000000046b31f in do_sema_b_wait_intern (sema=sema@entry=0xa4,
nointerrupt=nointerrupt@entry=0,
    timeout=timeout@entry=4294967295) at
/usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:625
#4  0x000000000046b60e in do_sema_b_wait (sema=0xa4, nointerrupt=0,
timeout=4294967295, cs=0x1e15c0, val=0x1e15e8)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:584
#5  0x000000000046bb91 in pthread_cond_wait (c=c@entry=0x1b0008,
external_mutex=external_mutex@entry=0x1b0000)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:451
#6  0x0000000000441232 in fio_mutex_down (mutex=0x1b0000) at mutex.c:224
#7  0x0000000000446eae in sfree_pool (ptr=0x2bf3420, pool=0x55c980
<mp>) at smalloc.c:312
#8  sfree (ptr=0x2bf3430) at smalloc.c:336
#9  0x000000000042df59 in close_and_free_files (td=td@entry=0xb410a60)
at filesetup.c:1358
#10 0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#11 0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#12 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from
C:\Windows\System32\msvcrt.dll
#13 0x00007ff845a9b38c in msvcrt!_endthreadex () from
C:\Windows\System32\msvcrt.dll
#14 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
C:\Windows\System32\kernel32.dll
#15 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
C:\Windows\SYSTEM32\ntdll.dll
#16 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 23 (Thread 3784.0x1404):
#0  0x00007ff845d06bf4 in ntdll!ZwWaitForMultipleObjects () from
C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264796f in WaitForMultipleObjectsEx () from
C:\Windows\System32\KernelBase.dll
#2  0x00007ff84264786e in WaitForMultipleObjects () from
C:\Windows\System32\KernelBase.dll
#3  0x000000000046b31f in do_sema_b_wait_intern (sema=sema@entry=0xa4,
nointerrupt=nointerrupt@entry=0,
    timeout=timeout@entry=4294967295) at
/usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:625
#4  0x000000000046b60e in do_sema_b_wait (sema=0xa4, nointerrupt=0,
timeout=4294967295, cs=0x1e15c0, val=0x1e15e8)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:584
#5  0x000000000046bb91 in pthread_cond_wait (c=c@entry=0x1b0008,
external_mutex=external_mutex@entry=0x1b0000)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:451
#6  0x0000000000441232 in fio_mutex_down (mutex=0x1b0000) at mutex.c:224
#7  0x0000000000446eae in sfree_pool (ptr=0x2bdee20, pool=0x55c980
<mp>) at smalloc.c:312
#8  sfree (ptr=0x2bdee30) at smalloc.c:336
#9  0x000000000042df76 in close_and_free_files (td=td@entry=0xb294340)
at filesetup.c:1367
#10 0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901
#11 0x000000000046df74 in pthread_create_wrapper (args=0x0)
    at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492
#12 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from
C:\Windows\System32\msvcrt.dll
#13 0x00007ff845a9b38c in msvcrt!_endthreadex () from
C:\Windows\System32\msvcrt.dll
#14 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
C:\Windows\System32\kernel32.dll
#15 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
C:\Windows\SYSTEM32\ntdll.dll
#16 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

So three threads made it to pthread_cond_wait and presumably one of
these was trying to be woken by the pthread_cond_signal().

Something else that seems strange is why would so many threads have
stacks like this:

Thread 200 (Thread 3784.0xe70):
#0  0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from
C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from
C:\Windows\System32\KernelBase.dll
#2  0x00000000004686b0 in IoCompletionRoutine (lpParameter=0x32b948b0)
at engines/windowsaio.c:427
#3  0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from
C:\Windows\System32\kernel32.dll
#4  0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from
C:\Windows\SYSTEM32\ntdll.dll
#5  0x0000000000000000 in ?? ()

Surely the threads in windowsaio should still be able to make progress?

-- 
Sitsofe | http://sucs.org/~sits/

Re: Windows: FIO randomly hangs using attached script

[Jens Axboe]

On 3/5/18 6:23 PM, Rebecca Cran wrote:
> I've had a report that FIO on Windows (at least Server 2012 R2 and 2016) 
> hangs when the attached script is run. The point at which it hangs is 
> apparently random, and within the condvar (pthread-win32) calls.
> 
> I've replicated the hang, but I don't have time to debug it so I was 
> hoping somebody on this mailing list might have time to dig into it and 
> figure out what's wrong.
> 
> The people I was talking to thought it might be due to FIO linking to 
> msvcrt.dll, which is old and not supposed to be used by applications - 
> they should use the CRT distributed with Visual C++, such as 
> msvcr120.dll instead. However, it appears that fixing this would take 
> quite a lot of work since while FIO is relatively straightforward to 
> change, pthread-win32 hasn't had a full release since 2012 and doesn't 
> build under a current msys environment due to duplicated symbols etc.

Do you at least know what mutex it's hanging on?

-- 
Jens Axboe

Re: Windows: FIO randomly hangs using attached script

[Rebecca Cran]

On 3/6/2018 2:53 PM, Jens Axboe wrote:
> On 3/5/18 6:23 PM, Rebecca Cran wrote:
>> I've had a report that FIO on Windows (at least Server 2012 R2 and 2016)
>> hangs when the attached script is run. The point at which it hangs is
>> apparently random, and within the condvar (pthread-win32) calls.
>>
>> I've replicated the hang, but I don't have time to debug it so I was
>> hoping somebody on this mailing list might have time to dig into it and
>> figure out what's wrong.
>>
>> The people I was talking to thought it might be due to FIO linking to
>> msvcrt.dll, which is old and not supposed to be used by applications -
>> they should use the CRT distributed with Visual C++, such as
>> msvcr120.dll instead. However, it appears that fixing this would take
>> quite a lot of work since while FIO is relatively straightforward to
>> change, pthread-win32 hasn't had a full release since 2012 and doesn't
>> build under a current msys environment due to duplicated symbols etc.
> Do you at least know what mutex it's hanging on?

No, I didn't get around to checking that.

-- 
Rebecca