From mboxrd@z Thu Jan  1 00:00:00 1970
Date: Mon, 26 Apr 2021 13:34:42 -0400
From: Vivek Goyal <vgoyal@redhat.com>
Message-ID: <20210426173442.GB1741690@redhat.com>
References: <20210423211130.GA1668472@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20210423211130.GA1668472@redhat.com>
Subject: Re: [Virtio-fs] [RFC] [PATCH] virtiofsd: Auto switch between inline
 and thread-pool processing
List-Id: Development discussions about virtio-fs <virtio-fs.redhat.com>
List-Unsubscribe: <https://listman.redhat.com/mailman/options/virtio-fs>,
	<mailto:virtio-fs-request@redhat.com?subject=unsubscribe>
List-Archive: <https://listman.redhat.com/archives/virtio-fs>
List-Post: <mailto:virtio-fs@redhat.com>
List-Help: <mailto:virtio-fs-request@redhat.com?subject=help>
List-Subscribe: <https://listman.redhat.com/mailman/listinfo/virtio-fs>,
	<mailto:virtio-fs-request@redhat.com?subject=subscribe>
To: virtio-fs-list <virtio-fs@redhat.com>
Cc: Michey Mehta <mimehta@redhat.com>

On Fri, Apr 23, 2021 at 05:11:30PM -0400, Vivek Goyal wrote:
> This is just an RFC patch for now. I am still running performance numbers
> and see if this method of switching is good enough or not. I did one run
> and seemed to get higher performance on deeper queue depths. There were
> few tests where I did not match lower queue depth performance with 
> no thread pool. May be run to run variation.

Hi All,

So I ran fio jobs with auto switch enabled and uploaded all the data
here. Also pasted final results at the end of email.

https://github.com/rhvgoyal/virtiofs-tests/tree/master/performance-results/auto-switch-inline-thread-pool/april-26-2021

Looks like auto switch threshold of 3 is working pretty well for me. That
is if queue depth of 3 or less, do inline processing otherwise switch
to thread pool. Of course this will vary from system to system and
based on I/O size as well. I have only tested with bs=4K.

I think it might be worth to provide user a knob say
"--thread-pool-threshold=<nr>" where user can control it. By default it
can be 3, may be.

Thanks
Vivek

NAME                    WORKLOAD                Bandwidth       IOPS            
tpool-16                seqread-psync           62.1mb          15.5k           
no-tpool                seqread-psync           89.3mb          22.3k           
auto-switch-3           seqread-psync           89.6mb          22.4k           
auto-switch-4           seqread-psync           88.8mb          22.2k           

tpool-16                seqread-psync-multi     256.1mb         64.0k           
no-tpool                seqread-psync-multi     280.6mb         70.1k           
auto-switch-3           seqread-psync-multi     282.9mb         70.7k           
auto-switch-4           seqread-psync-multi     284.9mb         71.2k           

tpool-16                seqread-libaio          327.7mb         81.9k           
no-tpool                seqread-libaio          306.5mb         76.6k           
auto-switch-3           seqread-libaio          361.8mb         90.4k           
auto-switch-4           seqread-libaio          365.3mb         91.3k           

tpool-16                seqread-libaio-multi    354.5mb         88.6k           
no-tpool                seqread-libaio-multi    300.7mb         75.1k           
auto-switch-3           seqread-libaio-multi    375.4mb         93.8k           
auto-switch-4           seqread-libaio-multi    374.3mb         93.5k           

tpool-16                randread-psync          64.5mb          16.1k           
no-tpool                randread-psync          88.5mb          22.1k           
auto-switch-3           randread-psync          87.5mb          21.8k           
auto-switch-4           randread-psync          87.7mb          21.9k           

tpool-16                randread-psync-multi    268.0mb         67.0k           
no-tpool                randread-psync-multi    282.5mb         70.6k           
auto-switch-3           randread-psync-multi    283.3mb         70.8k           
auto-switch-4           randread-psync-multi    282.6mb         70.6k           

tpool-16                randread-libaio         325.1mb         81.2k           
no-tpool                randread-libaio         303.0mb         75.7k           
auto-switch-3           randread-libaio         360.8mb         90.2k           
auto-switch-4           randread-libaio         361.0mb         90.2k           

tpool-16                randread-libaio-multi   352.2mb         88.0k           
no-tpool                randread-libaio-multi   289.6mb         72.4k           
auto-switch-3           randread-libaio-multi   373.5mb         93.3k           
auto-switch-4           randread-libaio-multi   372.0mb         93.0k           

tpool-16                seqwrite-psync          65.7mb          16.4k           
no-tpool                seqwrite-psync          87.6mb          21.9k           
auto-switch-3           seqwrite-psync          86.5mb          21.6k           
auto-switch-4           seqwrite-psync          85.9mb          21.4k           

tpool-16                seqwrite-psync-multi    270.9mb         67.7k           
no-tpool                seqwrite-psync-multi    268.0mb         67.0k           
auto-switch-3           seqwrite-psync-multi    267.7mb         66.9k           
auto-switch-4           seqwrite-psync-multi    263.7mb         65.9k           

tpool-16                seqwrite-libaio         293.5mb         73.3k           
no-tpool                seqwrite-libaio         286.1mb         71.5k           
auto-switch-3           seqwrite-libaio         333.7mb         83.4k           
auto-switch-4           seqwrite-libaio         330.5mb         82.6k           

tpool-16                seqwrite-libaio-multi   415.0mb         103.7k          
no-tpool                seqwrite-libaio-multi   281.6mb         70.4k           
auto-switch-3           seqwrite-libaio-multi   417.4mb         104.3k          
auto-switch-4           seqwrite-libaio-multi   401.6mb         100.4k          

tpool-16                randwrite-psync         65.1mb          16.2k           
no-tpool                randwrite-psync         87.9mb          21.9k           
auto-switch-3           randwrite-psync         86.6mb          21.6k           
auto-switch-4           randwrite-psync         85.9mb          21.4k           

tpool-16                randwrite-psync-multi   268.9mb         67.2k           
no-tpool                randwrite-psync-multi   261.0mb         65.2k           
auto-switch-3           randwrite-psync-multi   261.2mb         65.3k           
auto-switch-4           randwrite-psync-multi   263.7mb         65.9k           

tpool-16                randwrite-libaio        300.8mb         75.2k           
no-tpool                randwrite-libaio        287.4mb         71.8k           
auto-switch-3           randwrite-libaio        328.2mb         82.0k           
auto-switch-4           randwrite-libaio        335.4mb         83.8k           

tpool-16                randwrite-libaio-multi  415.4mb         103.8k          
no-tpool                randwrite-libaio-multi  271.7mb         67.9k           
auto-switch-3           randwrite-libaio-multi  415.7mb         103.9k          
auto-switch-4           randwrite-libaio-multi  396.5mb         99.1k           

tpool-16                randrw-psync            35.3mb/11.7mb   9044/3019       
no-tpool                randrw-psync            52.9mb/17.7mb   13.2k/4536      
auto-switch-3           randrw-psync            52.7mb/17.6mb   13.1k/4516      
auto-switch-4           randrw-psync            52.7mb/17.6mb   13.1k/4514      

tpool-16                randrw-psync-multi      151.9mb/50.8mb  37.9k/12.7k     
no-tpool                randrw-psync-multi      160.5mb/53.7mb  40.1k/13.4k     
auto-switch-3           randrw-psync-multi      167.6mb/56.1mb  41.9k/14.0k     
auto-switch-4           randrw-psync-multi      164.6mb/55.1mb  41.1k/13.7k     

tpool-16                randrw-libaio           83.0mb/27.8mb   20.7k/7121      
no-tpool                randrw-libaio           87.2mb/29.2mb   21.7k/7479      
auto-switch-3           randrw-libaio           87.9mb/29.4mb   21.9k/7544      
auto-switch-4           randrw-libaio           86.4mb/28.9mb   21.6k/7417      

tpool-16                randrw-libaio-multi     201.6mb/67.5mb  50.4k/16.8k     
no-tpool                randrw-libaio-multi     187.5mb/62.8mb  46.8k/15.7k     
auto-switch-3           randrw-libaio-multi     218.0mb/73.0mb  54.5k/18.2k     
auto-switch-4           randrw-libaio-multi     213.4mb/71.5mb  53.3k/17.8k