linux-cifs.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Tom Talpey <ttalpey@microsoft.com>
To: ronnie sahlberg <ronniesahlberg@gmail.com>,
	linux-cifs <linux-cifs@vger.kernel.org>
Subject: RE: FSCTL_QUERY_ALLOCATED_RANGES issue with Windows2016
Date: Thu, 15 Aug 2019 15:15:01 +0000	[thread overview]
Message-ID: <MWHPR2101MB073136FE90E5C6FCD9F21CA8A0AC0@MWHPR2101MB0731.namprd21.prod.outlook.com> (raw)
In-Reply-To: <CAN05THT0OkbAoNu8ZVSHF-xY7w0ZW4q4i-jTxjNManrnz0OMfg@mail.gmail.com>

> -----Original Message-----
> From: linux-cifs-owner@vger.kernel.org <linux-cifs-owner@vger.kernel.org> On
> Behalf Of ronnie sahlberg
> Sent: Wednesday, August 14, 2019 11:31 PM
> To: linux-cifs <linux-cifs@vger.kernel.org>
> Subject: FSCTL_QUERY_ALLOCATED_RANGES issue with Windows2016
> 
> I am seeing issues with how FSCTL_QUERY_ALLOCATED_RANGES behaves
> under windows,
> in particular that it is inconsistent so we can't run certain xfstests
> against windows servers.
> 
> 
> The behavior can be triggered using the following command from xfstests :
>   ./src/seek_sanity_test -s 19 -e 19 /mnt/file
> (where /mnt is an smb share mounted from a windows 2016 server.)
> 
> In cifs.ko we use this FSCTL to implement both SEEK_HOLE/SEEK_DATA as well
> as
> fiemap(). But since the behavior of this FSCTL is not deteministic it often
> cause the tests to fail.
> 
> 
> This is a test tool for SEEK_DATA and SEEK_HOLE. As part of this it performs
> a check to try to discover if the filesystem supports sparse files or not.
> For non-sparse files SEEK_DATA is basically a no-op and SEEK_HOLE just returns
> the file size.
> Later during the test, the result of whether the file is "sparse" or not
> will affect what the expected results are. If this check gets the
> sparse-supported wrong the test cases later will fail.
> 
> 
> How the check works:
> ====================
> The actual check for whether the file supports being sparse or not is the
> following snippet :
>         ftruncate(fd, 0);
>         bufsz = alloc_size * 2;
>         filsz = bufsz * 2;
> 
>         buf = do_malloc(bufsz);
>         if (!buf)
>                 goto out;
>         memset(buf, 'a', bufsz);
> 
>         /* File with 2 allocated blocks.... */
>         ret = do_pwrite(fd, buf, bufsz, 0);
>         if (ret)
>                 goto out;
> 
>         /* followed by a hole... */
>         ret = do_truncate(fd, filsz);
>         if (ret)
>                 goto out;
> 
>         /* Is SEEK_DATA and SEEK_HOLE supported in the kernel? */
>         pos = lseek(fd, 0, SEEK_HOLE);
>         if (pos == -1) {
>                 fprintf(stderr, "Kernel does not support llseek(2) extension "
>                         "SEEK_HOLE. Aborting.\n");
>                 ret = -1;
>                 goto out;
>         }
> 
>         if (pos == filsz) {
>                 default_behavior = 1;
>                 fprintf(stderr, "File system supports the default behavior.\n")\
> ;
>         }
> 
> I.e.
> 1, ftruncate to 0
> 2, write 2M to the start of the file
> 3, ftruncate the file to be 4Mb
> 4, SEEK_HOLE and check if it returns 4Mb or no.
> If it returns 4Mb then we switch back to default_behavior and we allow
> the semantics as if the file is not sparse.
> I.e. allow SEEK_DATA to behave as if the file is either sparse or not-sparse.
> 
> Also note that if it looks like the sparse-file is not supported then
> it prints ""File system supports the default behavior." which may help when
> running the test tool.
> 
> Strace for this check (when the check failed.)
> =============================================
> 18:22:14.949612 ftruncate(3, 0)         = 0 <0.011513>
> 18:22:14.963725 mmap(NULL, 2101248, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe8d9f14000 <0.0
> 00192>
> 18:22:14.970334 pwrite64(3,
> "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
> aaaaaaaaaaaaaaaaaa
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"...,
> 2097152, 0) = 2097152 <0.002127>
> 18:22:14.972620 ftruncate(3, 4194304)   = 0 <0.582491>
> 18:22:15.557308 lseek(3, 0, SEEK_HOLE)  = 4194304 <0.012791>
> 18:22:15.572457 write(2, "File system supports the default
> behavior.\n", 43) = 43 <0.000318>
> 
> 
> Example of when the check goes "Wrong". Capture seek_good.cap
> =============================================================
> The relevant packets here are
> 1820, set end of file == 0
> (1822/1823 update timestamps)
> 2930, write 2Mb to offset 0
> 3008, set the file sparse
> 3010, set end of file to 4M
> 3019, query-allocated-ranges
> 
> In 3019/3020 the server responds that the full 0-4M range is allocated.
> This is wrong since file should only be allocated in the first 2Mb at
> this stage.
> And this makes the test tool think that we don't support sparse files,
> and sets default_behavior to 1.
> 
> A different run when the check is successful (seek_bad.cap)
> ============================================================
> In the seek_bad capture you can see the same sequence in packets
> 1869, set end of file == 0
> 2990, write the first 2M of the file
> 3067, set file sparse
> 3069, set end of file to 4M
> 3078, query-allocated-ranges
> But this time, query-allocated-ranges report that only 0-2M is mapped,
> which is the correct range,
> and thus the test tool assumes that we can handle holes properly.
> 
> The captures are ~5MByte each unfiltered so too big for the list.
> Email me directly and I will send them to you.
> 
> 
> So the question here is what is the actual semantics for sparse files
> and query-allocated-ranges on windows?

I'll try to get you an answer, but in the meantime just a question...
Does the behavior change if the file is opened with FILE_FLAG_WRITE_THROUGH?

Tom.

  reply	other threads:[~2019-08-15 15:15 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-08-15  6:30 FSCTL_QUERY_ALLOCATED_RANGES issue with Windows2016 ronnie sahlberg
2019-08-15 15:15 ` Tom Talpey [this message]
2019-08-16  2:17   ` ronnie sahlberg
2019-08-15 15:48 ` David Disseldorp
2019-08-16  2:07   ` ronnie sahlberg
2019-08-16  4:36     ` ronnie sahlberg
2019-08-19 16:31       ` David Disseldorp
2019-08-19 22:03         ` David Disseldorp
2019-08-20 17:47           ` Tom Talpey
2019-08-22 22:46             ` ronnie sahlberg
2019-08-16  9:07     ` David Disseldorp

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=MWHPR2101MB073136FE90E5C6FCD9F21CA8A0AC0@MWHPR2101MB0731.namprd21.prod.outlook.com \
    --to=ttalpey@microsoft.com \
    --cc=linux-cifs@vger.kernel.org \
    --cc=ronniesahlberg@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).