[XFS SUMMIT] Deprecating V4 on-disk format

[XFS SUMMIT] Deprecating V4 on-disk format

[Dave Chinner]

Topic: Deprecating V4 On-disk Format

Scope:
	Long term life cycle planning
	Supporting old filesystems with new kernels.
	Unfixable integrity issues in v4 format.
	Reducing feature matrix for testing

Proposal:

The CRC-enabled V5 format has been the upstream default format now
since commit 566ebd5ae5fa ("mkfs: default to CRC enabled
filesystems") dated May 11 2015 (5 years ago!) and released in
xfsprogs v3.2.3. It is the default in all major distros, and has
been for some time.

We know that the v4 format has unfixable integrity issues apart from
the obvious lack of CRCs and self-describing metadata structures; it
has race conditions in log recovery realted to inode creation and
other such issues that could only be solved with an on-disk format
change of some kind. We are not adding new features to v4 formats,
so anyone wanting to use new XFS features must use v5 format
filesystems.

We also know that the number of v4 filesysetms in production is
slowly decreasing as systems are replaced as part of the natural
life cycle of production systems.

All this adds up to the realisation that existing v4 filesystems are
effectively in the "Maintenance Mode" era of the software life
cycle. The next stage in the life cycle is "Phasing Out" before we
drop support for it altogether, also know around here as
"deprecated" which is a sign that support will "soon" cease.

I'd like to move the v4 format to the "deprecated" state as a signal
to users that it should really not be considered viable for new
systems. New systems running modern kernels and userspace should
all be using the v5 format, so this mostly only affects existing
filesystems.

Note: I am not proposing that we drop support for the v4 format any
time soon. What I am proposing is an "end of lifecycle" tag similar
to the way we use EXPERIMENTAL to indicate that the functionality is
available but we don't recommend it for production systems yet.

Hence what I am proposing is that we introduce a DEPRECATED alert at
mount time to inform users that the functionality exists, but it
will not be maintained indefinitely into the future. For distros
with a ten year support life, this means that a near-future release
will pick up the DEPRECATED tag but still support the filesystem for
the support life of that release. A "future +1" release may not
support the v4 format at all.

Discussion points:

- How practical is this?
- What should we have mkfs do when directed to create a v4 format
  filesystem?
- How long before we decide to remove v4 support from the upstream
  kernel and tools? 5 years after deprecation? 10 years?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [XFS SUMMIT] Deprecating V4 on-disk format

[Darrick J. Wong]

On Wed, May 13, 2020 at 12:36:18PM +1000, Dave Chinner wrote:
> 
> Topic: Deprecating V4 On-disk Format
> 
> Scope:
> 	Long term life cycle planning
> 	Supporting old filesystems with new kernels.
> 	Unfixable integrity issues in v4 format.
> 	Reducing feature matrix for testing
> 
> Proposal:
> 
> The CRC-enabled V5 format has been the upstream default format now
> since commit 566ebd5ae5fa ("mkfs: default to CRC enabled
> filesystems") dated May 11 2015 (5 years ago!) and released in
> xfsprogs v3.2.3. It is the default in all major distros, and has
> been for some time.
> 
> We know that the v4 format has unfixable integrity issues apart from
> the obvious lack of CRCs and self-describing metadata structures; it
> has race conditions in log recovery realted to inode creation and
> other such issues that could only be solved with an on-disk format
> change of some kind. We are not adding new features to v4 formats,
> so anyone wanting to use new XFS features must use v5 format
> filesystems.
> 
> We also know that the number of v4 filesysetms in production is
> slowly decreasing as systems are replaced as part of the natural
> life cycle of production systems.
> 
> All this adds up to the realisation that existing v4 filesystems are
> effectively in the "Maintenance Mode" era of the software life
> cycle. The next stage in the life cycle is "Phasing Out" before we
> drop support for it altogether, also know around here as
> "deprecated" which is a sign that support will "soon" cease.
> 
> I'd like to move the v4 format to the "deprecated" state as a signal
> to users that it should really not be considered viable for new
> systems. New systems running modern kernels and userspace should
> all be using the v5 format, so this mostly only affects existing
> filesystems.
> 
> Note: I am not proposing that we drop support for the v4 format any
> time soon. What I am proposing is an "end of lifecycle" tag similar
> to the way we use EXPERIMENTAL to indicate that the functionality is
> available but we don't recommend it for production systems yet.
> 
> Hence what I am proposing is that we introduce a DEPRECATED alert at
> mount time to inform users that the functionality exists, but it
> will not be maintained indefinitely into the future. For distros
> with a ten year support life, this means that a near-future release
> will pick up the DEPRECATED tag but still support the filesystem for
> the support life of that release. A "future +1" release may not
> support the v4 format at all.

/me regrets that he is frequently failing to clear enough space out of
his schedule to respond to all of these adequately.  But here goes,
random thoughts at 23:23. :/

> Discussion points:
> 
> - How practical is this?

Well, we've killed off old features before... v1 inodes, v2 directories,
etc.  So clearly this can be done, given enough preparation time.

And we probably ought to do it, before we start to resemble the ext4
quota nightmare.

> - What should we have mkfs do when directed to create a v4 format
>   filesystem?

It probably ought to print a warning...

That said, way back when we were arguing with the syzbot people, one of
us suggested that we hide V4 behind a CONFIG_XFS_DEPRECATED=y option, so
that people who want to harden their kernel against the unfixable
structural problems in the V4 format could effectively lose V4 support
early.  Maybe we should add that for a few years?

> - How long before we decide to remove v4 support from the upstream
>   kernel and tools? 5 years after deprecation? 10 years?

That probably depends a lot on how much our respective employers want to
keep those old XFSes going.  Some of our customers are about ready to
certify that they can support their distro defaults changing from ext3
to XFS v4, but those folks have support contracts that won't terminate
until whenever the so^Hun goes out.

--D

> 
> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david@fromorbit.com

Re: [XFS SUMMIT] Deprecating V4 on-disk format

[Dave Chinner]

On Mon, May 18, 2020 at 11:23:38PM -0700, Darrick J. Wong wrote:
> On Wed, May 13, 2020 at 12:36:18PM +1000, Dave Chinner wrote:
> > 
> > Topic: Deprecating V4 On-disk Format
> > 
> > Scope:
> > 	Long term life cycle planning
> > 	Supporting old filesystems with new kernels.
> > 	Unfixable integrity issues in v4 format.
> > 	Reducing feature matrix for testing
> > 
> > Proposal:
> > 
> > The CRC-enabled V5 format has been the upstream default format now
> > since commit 566ebd5ae5fa ("mkfs: default to CRC enabled
> > filesystems") dated May 11 2015 (5 years ago!) and released in
> > xfsprogs v3.2.3. It is the default in all major distros, and has
> > been for some time.
> > 
> > We know that the v4 format has unfixable integrity issues apart from
> > the obvious lack of CRCs and self-describing metadata structures; it
> > has race conditions in log recovery realted to inode creation and
> > other such issues that could only be solved with an on-disk format
> > change of some kind. We are not adding new features to v4 formats,
> > so anyone wanting to use new XFS features must use v5 format
> > filesystems.
> > 
> > We also know that the number of v4 filesysetms in production is
> > slowly decreasing as systems are replaced as part of the natural
> > life cycle of production systems.
> > 
> > All this adds up to the realisation that existing v4 filesystems are
> > effectively in the "Maintenance Mode" era of the software life
> > cycle. The next stage in the life cycle is "Phasing Out" before we
> > drop support for it altogether, also know around here as
> > "deprecated" which is a sign that support will "soon" cease.
> > 
> > I'd like to move the v4 format to the "deprecated" state as a signal
> > to users that it should really not be considered viable for new
> > systems. New systems running modern kernels and userspace should
> > all be using the v5 format, so this mostly only affects existing
> > filesystems.
> > 
> > Note: I am not proposing that we drop support for the v4 format any
> > time soon. What I am proposing is an "end of lifecycle" tag similar
> > to the way we use EXPERIMENTAL to indicate that the functionality is
> > available but we don't recommend it for production systems yet.
> > 
> > Hence what I am proposing is that we introduce a DEPRECATED alert at
> > mount time to inform users that the functionality exists, but it
> > will not be maintained indefinitely into the future. For distros
> > with a ten year support life, this means that a near-future release
> > will pick up the DEPRECATED tag but still support the filesystem for
> > the support life of that release. A "future +1" release may not
> > support the v4 format at all.
> 
> /me regrets that he is frequently failing to clear enough space out of
> his schedule to respond to all of these adequately.  But here goes,
> random thoughts at 23:23. :/
> 
> > Discussion points:
> > 
> > - How practical is this?
> 
> Well, we've killed off old features before... v1 inodes, v2 directories,
> etc.  So clearly this can be done, given enough preparation time.
> 
> And we probably ought to do it, before we start to resemble the ext4
> quota nightmare.

*nod*

> 
> > - What should we have mkfs do when directed to create a v4 format
> >   filesystem?
> 
> It probably ought to print a warning...
> 
> That said, way back when we were arguing with the syzbot people, one of
> us suggested that we hide V4 behind a CONFIG_XFS_DEPRECATED=y option, so
> that people who want to harden their kernel against the unfixable
> structural problems in the V4 format could effectively lose V4 support
> early.  Maybe we should add that for a few years?

That sounds like a good idea - being able to config it out provides
a mechanism to isolate v4 only code paths over time....

> > - How long before we decide to remove v4 support from the upstream
> >   kernel and tools? 5 years after deprecation? 10 years?
> 
> That probably depends a lot on how much our respective employers want to
> keep those old XFSes going.  Some of our customers are about ready to
> certify that they can support their distro defaults changing from ext3
> to XFS v4, but those folks have support contracts that won't terminate
> until whenever the so^Hun goes out.

Well, there's a difference between what a distro that heavily
patches the upstream kernel is willing to support and what upstream
supports. And, realistically, v4 is going to be around for at least
one more major distro release, which means the distro support time
window is still going to be in the order of 15 years.

However, with the way typical enterprise feature policies work, a
feature needs to be deprecated for a major release before it can be
removed.

So, realistically, the deprecation decision is not for the near term
- it's for dropping support in the "current + 2" major release that
nobody is even thinking about yet. i.e. if we don't deprecate it
in the next couple of years, then we'll still need to be
maintaining the v4 filesystem format in upstream for the next 15
years. Which, IMO, is about 10 years too long...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [XFS SUMMIT] Deprecating V4 on-disk format

[Emmanuel Florac]

[-- Attachment #1: Type: text/plain, Size: 1262 bytes --]

Le Wed, 20 May 2020 11:14:30 +1000
Dave Chinner <david@fromorbit.com> écrivait:

> Well, there's a difference between what a distro that heavily
> patches the upstream kernel is willing to support and what upstream
> supports. And, realistically, v4 is going to be around for at least
> one more major distro release, which means the distro support time
> window is still going to be in the order of 15 years.

IIRC, RedHat/CentOS v.7.x shipped with a v5-capable mkfs.xfs, but
defaulted to v4. That means that unless you were extremely cautious
(like I am :) 99% of RH/COs v7 will be running v4 volumes for the
coming years. How many years, would you ask?

As for the lifecycle of a filesystem, I just ended support on a 40 TB
archival server I set up back in 2007. I still have a number of
supported systems from the years 2008-2010, and about a hundred from
2010-2013. That's how reliable XFS is, unfortunately :)

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

[-- Attachment #2: Signature digitale OpenPGP --]
[-- Type: application/pgp-signature, Size: 163 bytes --]

Re: [XFS SUMMIT] Deprecating V4 on-disk format

[Mike Fleetwood]

On Wed, 20 May 2020 at 14:25, Emmanuel Florac <eflorac@intellique.com> wrote:
>
> Le Wed, 20 May 2020 11:14:30 +1000
> Dave Chinner <david@fromorbit.com> écrivait:
>
> > Well, there's a difference between what a distro that heavily
> > patches the upstream kernel is willing to support and what upstream
> > supports. And, realistically, v4 is going to be around for at least
> > one more major distro release, which means the distro support time
> > window is still going to be in the order of 15 years.
>
> IIRC, RedHat/CentOS v.7.x shipped with a v5-capable mkfs.xfs, but
> defaulted to v4. That means that unless you were extremely cautious
> (like I am :) 99% of RH/COs v7 will be running v4 volumes for the
> coming years. How many years, would you ask?

[Trying again hopefully without HTML]

So initial RHEL/CentOS 7 releases did create XFS v4 file systems.
However from RHEL/CentOS 7.3 [1] (circa Nov 2016) they are creating XFS
v5 file systems by default.


[1] RHEL 7.3 Release Notes > Chapter 9. File Systems
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.3_release_notes/new_features_file_systems


# cat /etc/centos-release
CentOS Linux release 7.8.2003 (Core)
# mkfs.xfs -V
mkfs.xfs version 4.5.0
# mkfs.xfs /dev/sdb13
...
# xfs_db -c version -r /dev/sdb13
versionnum [0xb4a5+0x18a] =
V5,NLINK,DIRV2,ALIGN,LOGV2,EXTFLG,MOREBITS,ATTR2,LAZYSBCOUNT,PROJID32BIT,CRC,FTYPE

Re: [XFS SUMMIT] Deprecating V4 on-disk format

[Eric Sandeen]

On 5/21/20 3:29 AM, Mike Fleetwood wrote:
> On Wed, 20 May 2020 at 14:25, Emmanuel Florac <eflorac@intellique.com> wrote:
>>
>> Le Wed, 20 May 2020 11:14:30 +1000
>> Dave Chinner <david@fromorbit.com> écrivait:
>>
>>> Well, there's a difference between what a distro that heavily
>>> patches the upstream kernel is willing to support and what upstream
>>> supports. And, realistically, v4 is going to be around for at least
>>> one more major distro release, which means the distro support time
>>> window is still going to be in the order of 15 years.
>>
>> IIRC, RedHat/CentOS v.7.x shipped with a v5-capable mkfs.xfs, but
>> defaulted to v4. That means that unless you were extremely cautious
>> (like I am :) 99% of RH/COs v7 will be running v4 volumes for the
>> coming years. How many years, would you ask?
> 
> [Trying again hopefully without HTML]
> 
> So initial RHEL/CentOS 7 releases did create XFS v4 file systems.
> However from RHEL/CentOS 7.3 [1] (circa Nov 2016) they are creating XFS
> v5 file systems by default.

That's correct.  Still a little shocked that we switched midstream but it
worked out fine.  :)

But as Dave said downthread, distro defaults and support is unrelated to
upstream; distros already signed up to support their shipped features
for the lifetime of the OS (or whatever their support policy says)

In fact, deprecating old stuff upstream will help future distro releases
manage their support by getting rid of antiquated features and behaviors...

-Eric

> 
> [1] RHEL 7.3 Release Notes > Chapter 9. File Systems
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.3_release_notes/new_features_file_systems
> 
> 
> # cat /etc/centos-release
> CentOS Linux release 7.8.2003 (Core)
> # mkfs.xfs -V
> mkfs.xfs version 4.5.0
> # mkfs.xfs /dev/sdb13
> ...
> # xfs_db -c version -r /dev/sdb13
> versionnum [0xb4a5+0x18a] =
> V5,NLINK,DIRV2,ALIGN,LOGV2,EXTFLG,MOREBITS,ATTR2,LAZYSBCOUNT,PROJID32BIT,CRC,FTYPE
> 

Re: [XFS SUMMIT] Deprecating V4 on-disk format

[Dave Chinner]

On Wed, May 20, 2020 at 03:15:10PM +0200, Emmanuel Florac wrote:
> Le Wed, 20 May 2020 11:14:30 +1000
> Dave Chinner <david@fromorbit.com> écrivait:
> 
> > Well, there's a difference between what a distro that heavily
> > patches the upstream kernel is willing to support and what upstream
> > supports. And, realistically, v4 is going to be around for at least
> > one more major distro release, which means the distro support time
> > window is still going to be in the order of 15 years.
> 
> IIRC, RedHat/CentOS v.7.x shipped with a v5-capable mkfs.xfs, but
> defaulted to v4. That means that unless you were extremely cautious
> (like I am :) 99% of RH/COs v7 will be running v4 volumes for the
> coming years. How many years, would you ask?

Largely irrelevant to the question at hand, as support is dependent
on the distro lifecycle here. Essentially whatever is in RHEL7 is
supported by RH until the end of it's life.

In RHEL8, we default to v5 filesystems, but fully support v4. That
will be the case for the rest of it's life. Unless the user
specifically asks for it, no new v4 filesystems are being created on
current RHEL releases.

If we were to deprecate v4 now, then it will be marked as deprecated
in the next major RHEL release. That means it's still fully
supported in that release for it's entire life, but it will be
removed in the next major release after that. So we are still
talking about at least 15+ years of enterprise distro support for
the format, even if upstream drops it sooner...

> As for the lifecycle of a filesystem, I just ended support on a 40 TB
> archival server I set up back in 2007. I still have a number of
> supported systems from the years 2008-2010, and about a hundred from
> 2010-2013. That's how reliable XFS is, unfortunately :)

Yup, 10-15 years is pretty much the expected max life of storage
systems before the hardware really needs to be retired. We made v5
the default 5 years ago, so give it another 10 years (the sort of
timeframe we are talking about here) and just about
everything will be running v5 and that's when v4 can likely be
dropped.

The other thing to consider is that we need to drop v4 before we get
to y2038 support issues as the format will never support dates
beyond that. Essentially, we need to have the deprecation discussion
and take action in the near future so that people have stopped using
it before y2038 comes along and v4 filesystems break everything.

Not enough people think long term when it comes to computers - it
should be more obvious now why I brought this up for discussion...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [XFS SUMMIT] Deprecating V4 on-disk format

[Darrick J. Wong]

On Mon, May 25, 2020 at 01:23:54PM +1000, Dave Chinner wrote:
> On Wed, May 20, 2020 at 03:15:10PM +0200, Emmanuel Florac wrote:
> > Le Wed, 20 May 2020 11:14:30 +1000
> > Dave Chinner <david@fromorbit.com> écrivait:
> > 
> > > Well, there's a difference between what a distro that heavily
> > > patches the upstream kernel is willing to support and what upstream
> > > supports. And, realistically, v4 is going to be around for at least
> > > one more major distro release, which means the distro support time
> > > window is still going to be in the order of 15 years.
> > 
> > IIRC, RedHat/CentOS v.7.x shipped with a v5-capable mkfs.xfs, but
> > defaulted to v4. That means that unless you were extremely cautious
> > (like I am :) 99% of RH/COs v7 will be running v4 volumes for the
> > coming years. How many years, would you ask?
> 
> Largely irrelevant to the question at hand, as support is dependent
> on the distro lifecycle here. Essentially whatever is in RHEL7 is
> supported by RH until the end of it's life.
> 
> In RHEL8, we default to v5 filesystems, but fully support v4. That
> will be the case for the rest of it's life. Unless the user
> specifically asks for it, no new v4 filesystems are being created on
> current RHEL releases.
> 
> If we were to deprecate v4 now, then it will be marked as deprecated
> in the next major RHEL release. That means it's still fully
> supported in that release for it's entire life, but it will be
> removed in the next major release after that. So we are still
> talking about at least 15+ years of enterprise distro support for
> the format, even if upstream drops it sooner...

We probably ought to do it sooner than later though, particularly if we
think/care about 5.9 turning into an LTS.

> > As for the lifecycle of a filesystem, I just ended support on a 40 TB
> > archival server I set up back in 2007. I still have a number of
> > supported systems from the years 2008-2010, and about a hundred from
> > 2010-2013. That's how reliable XFS is, unfortunately :)
> 
> Yup, 10-15 years is pretty much the expected max life of storage
> systems before the hardware really needs to be retired. We made v5
> the default 5 years ago, so give it another 10 years (the sort of
> timeframe we are talking about here) and just about
> everything will be running v5 and that's when v4 can likely be
> dropped.
> 
> The other thing to consider is that we need to drop v4 before we get
> to y2038 support issues as the format will never support dates
> beyond that. Essentially, we need to have the deprecation discussion
> and take action in the near future so that people have stopped using
> it before y2038 comes along and v4 filesystems break everything.
> 
> Not enough people think long term when it comes to computers - it
> should be more obvious now why I brought this up for discussion...

Ok then, who would like to help me get the y2038 timestamp patches
reviewed for ~5.9? :D

(Anyone; not necessarily Dave)

--D

> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david@fromorbit.com

Re: [XFS SUMMIT] Deprecating V4 on-disk format

[Amir Goldstein]

On Mon, May 25, 2020 at 9:12 AM Darrick J. Wong <darrick.wong@oracle.com> wrote:
>
> On Mon, May 25, 2020 at 01:23:54PM +1000, Dave Chinner wrote:
> > On Wed, May 20, 2020 at 03:15:10PM +0200, Emmanuel Florac wrote:
> > > Le Wed, 20 May 2020 11:14:30 +1000
> > > Dave Chinner <david@fromorbit.com> écrivait:
> > >
> > > > Well, there's a difference between what a distro that heavily
> > > > patches the upstream kernel is willing to support and what upstream
> > > > supports. And, realistically, v4 is going to be around for at least
> > > > one more major distro release, which means the distro support time
> > > > window is still going to be in the order of 15 years.
> > >
> > > IIRC, RedHat/CentOS v.7.x shipped with a v5-capable mkfs.xfs, but
> > > defaulted to v4. That means that unless you were extremely cautious
> > > (like I am :) 99% of RH/COs v7 will be running v4 volumes for the
> > > coming years. How many years, would you ask?
> >
> > Largely irrelevant to the question at hand, as support is dependent
> > on the distro lifecycle here. Essentially whatever is in RHEL7 is
> > supported by RH until the end of it's life.
> >
> > In RHEL8, we default to v5 filesystems, but fully support v4. That
> > will be the case for the rest of it's life. Unless the user
> > specifically asks for it, no new v4 filesystems are being created on
> > current RHEL releases.
> >
> > If we were to deprecate v4 now, then it will be marked as deprecated
> > in the next major RHEL release. That means it's still fully
> > supported in that release for it's entire life, but it will be
> > removed in the next major release after that. So we are still
> > talking about at least 15+ years of enterprise distro support for
> > the format, even if upstream drops it sooner...
>
> We probably ought to do it sooner than later though, particularly if we
> think/care about 5.9 turning into an LTS.
>
> > > As for the lifecycle of a filesystem, I just ended support on a 40 TB
> > > archival server I set up back in 2007. I still have a number of
> > > supported systems from the years 2008-2010, and about a hundred from
> > > 2010-2013. That's how reliable XFS is, unfortunately :)
> >
> > Yup, 10-15 years is pretty much the expected max life of storage
> > systems before the hardware really needs to be retired. We made v5
> > the default 5 years ago, so give it another 10 years (the sort of
> > timeframe we are talking about here) and just about
> > everything will be running v5 and that's when v4 can likely be
> > dropped.
> >
> > The other thing to consider is that we need to drop v4 before we get
> > to y2038 support issues as the format will never support dates
> > beyond that. Essentially, we need to have the deprecation discussion
> > and take action in the near future so that people have stopped using
> > it before y2038 comes along and v4 filesystems break everything.
> >
> > Not enough people think long term when it comes to computers - it
> > should be more obvious now why I brought this up for discussion...
>
> Ok then, who would like to help me get the y2038 timestamp patches
> reviewed for ~5.9? :D
>

I can help with review. Already looked at your branch.

Thanks,
Amir.

Re: [XFS SUMMIT] Deprecating V4 on-disk format

[Emmanuel Florac]

[-- Attachment #1: Type: text/plain, Size: 659 bytes --]

Le Mon, 25 May 2020 13:23:54 +1000
Dave Chinner <david@fromorbit.com> écrivait:

> Not enough people think long term when it comes to computers - it
> should be more obvious now why I brought this up for discussion...

I'm thinking long term: I also run an Indy and an Octane and don't
plan to retire them any time soon :)

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

[-- Attachment #2: Signature digitale OpenPGP --]
[-- Type: application/pgp-signature, Size: 163 bytes --]