Is thin provisioning still experimental?

Is thin provisioning still experimental?

[Drew Hastings]

[-- Attachment #1.1: Type: text/plain, Size: 2180 bytes --]

I love all of the work you guys do @dm-devel . Thanks for taking the time
to read this.


I would like to use thin provisioning targets in production, but it's hard
to ignore the warning in the documentation. It seems like, with an
understanding of how thin provisioning works, it should be safe to use.


Quoting from this page:
https://raw.githubusercontent.com/torvalds/linux/master/Documentation/device-mapper/thin-provisioning.txt

"These targets are very much still in the EXPERIMENTAL state.  Please do
not yet rely on them in production.  But do experiment and offer us
feedback.  Different use cases will have different performance
characteristics, for example due to fragmentation of the data volume."

I believe that similar suggestions have been in place for more than 7 years
now, based on this post:
https://www.redhat.com/archives/dm-devel/2011-July/msg00012.html


Is it still the case that thin provisioning should never be used in
production? There are plenty of articles online (including from redhat)
which make no mention of the fact that the underlying technology is unsafe.
LVM does not offer any warnings about the inherit risk of using thin
provisioning.

Is that advisory still necessary? If so, can you provide some insight
specifically into what is unsafe about the thin targets at this point? Is
the concern that the underlying software device drivers are still not
tested well enough to avoid bugs? Is this more a situation where someone
might run into extremely poor performance for a very bad implementation of
the targets? Is it more about people directly using the targets instead of
using LVM, and maybe it's not bullet-proof for bad integrations?


If the metadata and data device for the thin pool have enough space and are
both error free, the kernel has plenty of free RAM, block sizes are set
large enough to never run into performance issues (64 MiB), all of the
underlying hardware is redundant on high performance NVME (no worries of
fragmentation of data volume)... is it still unsafe for production? If so,
can you shed some light on why that is?


Thin provisioning is so cool. It would be a shame to not use it!


Thank you so much!

[-- Attachment #1.2: Type: text/html, Size: 2620 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Is thin provisioning still experimental?

[Mike Snitzer]

On Mon, Jul 23 2018 at  1:06am -0400,
Drew Hastings <dhastings@crucialwebhost.com> wrote:

>    I love all of the work you guys do @dm-devel . Thanks for taking the time
>    to read this.
>    I would like to use thin provisioning targets in production, but it's hard
>    to ignore the warning in the documentation. It seems like, with an
>    understanding of how thin provisioning works, it should be safe to use.

It is stale.  I just committed this update that'll go upstream for the
4.19 merge window, see:
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-4.19&id=f88a3f746ff0047c92e8312646247b08264daf35

>    If the metadata and data device for the thin pool have enough space and
>    are both error free, the kernel has plenty of free RAM, block sizes are
>    set large enough to never run into performance issues (64 MiB), all of the
>    underlying hardware is redundant on high performance NVME (no worries of
>    fragmentation of data volume)... is it still unsafe for production? If so,
>    can you shed some light on why that is?

It is safe.  You do just want to make sure to not run out of space.  We
now handle that event favorably but it is best to tempt fate.  No idea
what, if any, filesystem you intend to run but XFS has the ability to
discontinue retries for certain error conditions.  I highly recommend
you enable that for ENOSPC, otherwise you _will_ see the kernel block
indefinitely in XFS if thinp runs out of space underneath XFS.

>    Thin provisioning is so cool. It would be a shame to not use it!
>    Thank you so much!

Enjoy.

Mike

Re: Is thin provisioning still experimental?

[Mike Snitzer]

On Mon, Jul 23 2018 at 10:00am -0400,
Mike Snitzer <snitzer@redhat.com> wrote:

> On Mon, Jul 23 2018 at  1:06am -0400,
> Drew Hastings <dhastings@crucialwebhost.com> wrote:
> 
> >    I love all of the work you guys do @dm-devel . Thanks for taking the time
> >    to read this.
> >    I would like to use thin provisioning targets in production, but it's hard
> >    to ignore the warning in the documentation. It seems like, with an
> >    understanding of how thin provisioning works, it should be safe to use.
> 
> It is stale.  I just committed this update that'll go upstream for the
> 4.19 merge window, see:
> https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-4.19&id=f88a3f746ff0047c92e8312646247b08264daf35
> 
> >    If the metadata and data device for the thin pool have enough space and
> >    are both error free, the kernel has plenty of free RAM, block sizes are
> >    set large enough to never run into performance issues (64 MiB), all of the
> >    underlying hardware is redundant on high performance NVME (no worries of
> >    fragmentation of data volume)... is it still unsafe for production? If so,
> >    can you shed some light on why that is?
> 
> It is safe.  You do just want to make sure to not run out of space.  We
> now handle that event favorably but it is best to tempt fate.

I meant: "... but it is best to _not_ tempt fate."

Re: Is thin provisioning still experimental?

[Drew Hastings]

[-- Attachment #1.1: Type: text/plain, Size: 2593 bytes --]

Thank you for that insight, I appreciate it.

Can you elaborate on your concern related to running out of space?

Assuming the metadata device always has space, is it unsafe to let the the
data device for the thin pool run out of space if queue_if_no_space is set?
Would it not be safe on the block IO level?

> No idea what, if any, filesystem you intend to run but XFS has the
ability to discontinue retries for certain error conditions.  I highly
recommend you enable that for ENOSPC, otherwise you _will_ see the kernel
block indefinitely in XFS if thinp runs out of space underneath XFS.

My use case involves doing block level IO for virtual machines with KVM...
so there's virtualization in between the thin device and whatever the VM is
doing. In that scenario, I would expect the IO requests to just hang until
the thin pool is provided more space for the data device and queued IO from
the VM would just wait for that. Maybe what you're describing with XFS
wouldn't be an issue on that setup, since XFS would be running within the
VM.

On Mon, Jul 23, 2018 at 7:07 AM, Mike Snitzer <snitzer@redhat.com> wrote:

> On Mon, Jul 23 2018 at 10:00am -0400,
> Mike Snitzer <snitzer@redhat.com> wrote:
>
> > On Mon, Jul 23 2018 at  1:06am -0400,
> > Drew Hastings <dhastings@crucialwebhost.com> wrote:
> >
> > >    I love all of the work you guys do @dm-devel . Thanks for taking
> the time
> > >    to read this.
> > >    I would like to use thin provisioning targets in production, but
> it's hard
> > >    to ignore the warning in the documentation. It seems like, with an
> > >    understanding of how thin provisioning works, it should be safe to
> use.
> >
> > It is stale.  I just committed this update that'll go upstream for the
> > 4.19 merge window, see:
> > https://git.kernel.org/pub/scm/linux/kernel/git/device-
> mapper/linux-dm.git/commit/?h=dm-4.19&id=f88a3f746ff0047c92e8312646247b
> 08264daf35
> >
> > >    If the metadata and data device for the thin pool have enough space
> and
> > >    are both error free, the kernel has plenty of free RAM, block sizes
> are
> > >    set large enough to never run into performance issues (64 MiB), all
> of the
> > >    underlying hardware is redundant on high performance NVME (no
> worries of
> > >    fragmentation of data volume)... is it still unsafe for production?
> If so,
> > >    can you shed some light on why that is?
> >
> > It is safe.  You do just want to make sure to not run out of space.  We
> > now handle that event favorably but it is best to tempt fate.
>
> I meant: "... but it is best to _not_ tempt fate."
>

[-- Attachment #1.2: Type: text/html, Size: 4240 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Is thin provisioning still experimental?

[Mike Snitzer]

[Please don't top-post.. it really compromises the ability to provide
proper context for inlined replies, etc]

On Fri, Aug 03 2018 at 11:13pm -0400,
Drew Hastings <dhastings@crucialwebhost.com> wrote:

>    Thank you for that insight, I appreciate it.
>    Can you elaborate on your concern related to running out of space?
>
>    Assuming the metadata device always has space, is it unsafe to let the the
>    data device for the thin pool run out of space if queue_if_no_space is
>    set? Would it not be safe on the block IO level?

You definitely want to avoid running out of metadata space.  That is one
area where we seem to have some lingering issues (that we're actively
working to find).

But running out of data space isn't a huge problem, especially if you'd
like to queue the IO indefinitely because you're closely administering
the space on your host with the hypervisor that will consume the thinp
devices.

You can configure a timeout for the queuing (after timeout expires the
thin-pool will transition to error_if_no_space mode).  It defaults to 60
seconds.  But you can also disable the timeout be setting it to 0.
 
>    > No idea what, if any, filesystem you intend to run but XFS has the
>    ability to discontinue retries for certain error conditions.  I highly
>    recommend you enable that for ENOSPC, otherwise you _will_ see the kernel
>    block indefinitely in XFS if thinp runs out of space underneath XFS.
> 
>    My use case involves doing block level IO for virtual machines with KVM...
>    so there's virtualization in between the thin device and whatever the VM
>    is doing. In that scenario, I would expect the IO requests to just hang
>    until the thin pool is provided more space for the data device and queued
>    IO from the VM would just wait for that. Maybe what you're describing with
>    XFS wouldn't be an issue on that setup, since XFS would be running within
>    the VM.

Yes, you can have it hang indefinitely (waiting for thin-pool resize) by
setting the 'no_space_timeout' module paramter for dm-thin-pool to 0,
e.g.:

modprobe dm-thin-pool no_space_timeout=0
or
echo 0 > /sys/module/dm_thin_pool/parameters/no_space_timeout

That just puts more importance on the admin (I assume you) feeding the
thin-pool extra free space as needed.

lvm2 has some pretty robust support for resizing both the thin-pool's
data and metadata devices.  Please have a look at the 'lvmthin' manpage
that is provided with a more recent lvm2 package.

Thanks,
Mike