Re: Stability of FILEIO as backing store?

[Forza <forza@tnonline.net>]

---- From: Mike Christie <michael.christie@oracle.com> -- Sent: 2021-02-26 - 17:12 ----

> On 2/26/21 1:19 AM, Forza wrote:
>> 
>> 
>> ---- From: michael.christie@oracle.com -- Sent: 2021-02-26 - 04:41 ----
>> 
>>> On 2/25/21 3:36 PM, Forza wrote:
>>>> Hi,
>>>>
>>>> I have a weird issue with using a file as backing store with a Win2016 server as initiator. 
>>>>
>>>> Very often if I reboot the Linux server the disk image becomes corrupt so that Windows cannot even detect the gpt partition table on it. It can happen even if I shut down the Windows machine before I reboot the Linux server.
>>>>
>>>> Initially I thought I would be write cache. But I've disabled that with no benefit to this problem. 
>>>>
>>>
>>> How are you disabling the write cache? What tools do you use? Is
>>> it targetcli or are you doing this manually via configfs?
>> 
>> I am using targetcli, using the documentation https://urldefense.com/v3/__http://linux-iscsi.org/Doc/LIO*20Admin*20Manual.pdf__;JSU!!GqivPVa7Brio!OYNbrN3Fseq8PE_-n67Mmb8_JdUUU_yWw_LcbeKvyKgKgP_iVH-X2u1vrU9RasK-Nvhz$ 
>> I created the backstore/fileio with write_back=false but this might not have disabled cache sinze I see in the saveconfig.json that it is still set to true. So I guess this can still be an issue =( 
>> 
>> 
>>>
>>> What is the output of
>>>
>>> cat /sys/kernel/config/target/core/fileio_$N/$name/info
>> 
>> Status: ACTIVATED  Max Queue Depth: 128  SectorSize: 512  HwMaxSectors: 16384
>>         TCM FILEIO ID: 0        File: /media/iscsi-tgt/dx_media_3.img  Size: 429496729600  Mode: Buffered-WCE Async: 0
>> 
> 
> I think you need to ask the people that maintain your tools (fedora or upstream
> https://github.com/open-iscsi/targetcli-fb), because for upstream's master branch
> it looks like doing write_back=false should work, but above we see "Mode: Buffered-WCE"
> and below we see emulate_write_cache=1 kike you mentioned.

I think you might be right.

In the end as I needed to get this to work, I swapped Fedora for Ubuntu Server 20.0.4.2 LTS with the HWE kernel. 

Since I changed to Ubuntu I have not had any issues. I've tested hard reboots and unclean shut downs with no issue. Go figure... 

Fedora automatically loads the saved config using "targetctl restore", while on Ubuntu automatic restore is not enabled, so I added a cron @reboot line to do "targetcli restoreconfig /etc/target/myconfig.json". Perhaps that avoids some race  during boot? 

Are there any technical differences between "targetctl restore" and "targetcli restoreconfig saveconfig.json"

There are some other changes that might be the most important. "Emulate_write_cache" is now false and /sys/config show "mode:o_dsync" instead of "buffered-wce" .

Also, the "aio" attribute  is different. Perhaps I set it manually in Fedora, but I cannot remember. It is not visible in /sys but is in the saveconfig.json.
Ubuntu:
  "storage_objects": [
    {
      "aio": false,
Fedora:
  "storage_objects": [
    {
      "aio": true,

Does "aio" mean Async I/O in this case? I could not find any documentation for this attribute. What implication would this have with false vs true? 

# grep . info
Status: ACTIVATED  Max Queue Depth: 128  SectorSize: 512  HwMaxSectors: 16384
        TCM FILEIO ID: 0        File: /media/iscsi-tgt/dx_media_3.img  Size: 429496729600  Mode: O_DSYNC Async: 0


# grep . attrib/*
attrib/alua_support:1
attrib/block_size:512
attrib/emulate_3pc:1
attrib/emulate_caw:1
attrib/emulate_dpo:1
attrib/emulate_fua_read:1
attrib/emulate_fua_write:1
attrib/emulate_model_alias:1
attrib/emulate_pr:1
attrib/emulate_rest_reord:0
attrib/emulate_tas:1
attrib/emulate_tpu:0
attrib/emulate_tpws:0
attrib/emulate_ua_intlck_ctrl:0
attrib/emulate_write_cache:0
attrib/enforce_pr_isids:1
attrib/force_pr_aptpl:0
attrib/hw_block_size:512
attrib/hw_max_sectors:16384
attrib/hw_pi_prot_type:0
attrib/hw_queue_depth:128
attrib/is_nonrot:0
attrib/max_unmap_block_desc_count:1
attrib/max_unmap_lba_count:8192
attrib/max_write_same_len:4096
attrib/optimal_sectors:16384
attrib/pgr_support:1
attrib/pi_prot_format:0
attrib/pi_prot_type:0
attrib/pi_prot_verify:0
attrib/queue_depth:128
attrib/unmap_granularity:1
attrib/unmap_granularity_alignment:0
attrib/unmap_zeroes_data:0



> 
>> 
>>> cat /sys/kernel/config/target/core/fileio_$N/$name/attrib/write_cache
>> 
>> This does not exist. The curent files are:
>> 
> 
> Sorry, I meant emulate_write_cache.
> 
>> # grep . *
>> alua_support:1
>> block_size:512
>> emulate_3pc:1
>> emulate_caw:1
>> emulate_dpo:1
>> emulate_fua_read:1
>> emulate_fua_write:1
>> emulate_model_alias:1
>> emulate_pr:1
>> emulate_rest_reord:0
>> emulate_tas:1
>> emulate_tpu:0
>> emulate_tpws:0
>> emulate_ua_intlck_ctrl:0
>> emulate_write_cache:1
>> enforce_pr_isids:1
>> force_pr_aptpl:0
>> hw_block_size:512
>> hw_max_sectors:16384
>> hw_pi_prot_type:0
>> hw_queue_depth:128
>> is_nonrot:0
>> max_unmap_block_desc_count:1
>> max_unmap_lba_count:8192
>> max_write_same_len:4096
>> optimal_sectors:16384
>> pgr_support:1
>> pi_prot_format:0
>> pi_prot_type:0
>> pi_prot_verify:0
>> queue_depth:128
>> unmap_granularity:1
>> unmap_granularity_alignment:0
>> unmap_zeroes_data:0


Thank you all for your inputs. I still would like to understand what wasn't right in the Fedora setup so that I might learn and avoid the specific pitfall in the future. 

Regards, 
Forza