[Cluster-devel] How can be metadata(e.g., inode) in the GFS2 file system shared between client nodes?

[Cluster-devel] How can be metadata(e.g., inode) in the GFS2 file system shared between client nodes?

[한대규]

An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20190809/fa835e44/attachment.htm>

[Cluster-devel] How can be metadata(e.g., inode) in the GFS2 file system shared between client nodes?

[Andrew Price]

Hi Daegyu,

On 09/08/2019 09:10, ??? wrote:
> Hi, I'm Daegyu from Sungkyunkwan University.
> 
> I'm curious how GFS2's filesystem metadata is shared between nodes.

The key thing to know about gfs2 is that it is a shared storage 
filesystem where each node mounts the same storage device. It is 
different from a distributed filesystem where each node has storage 
devices that only it accesses.

> In detail, I wonder how the metadata in the memory of the node mounting GFS2
> looks the consistent filesystem to other nodes.

gfs2 uses dlm for locking of filesystem metadata among the nodes. The 
transfer of locks between nodes allows gfs2 to decide when its in-memory 
caches are invalid and require re-reading from the storage.

> In addition, what role does corosync play in gfs2?

gfs2 doesn't communicate with corosync directly but it operates on top 
of a high-availability cluster. corosync provides synchronization and 
coherency for the cluster. If a node stops responding, corosync will 
notice and trigger actions (fencing) to make sure that node is put back 
into a safe and consistent state. This is important in gfs2 to prevent 
"misbehaving" nodes from corrupting the filesystem.

Hope this helps.

Cheers,
Andy

[Cluster-devel] How can be metadata(e.g., inode) in the GFS2 file system shared between client nodes?

[Daegyu Han]

Thank you for your reply.

If what I understand is correct,
In a gfs2 file system shared by clients A and B, if A creates /foo/a.txt,
does B re-read the filesystem metadata area on storage to keep the data
consistent?

After all, what makes gfs2 different from local filesystems like ext4,
because of lock_dlm?

In general, if we mount an ext4 file system on two different clients and
update the file system on each client, we know that the file system state
is not reflected in each other.

Thank you,
Daegyu
?

2019? 8? 9? (?) ?? 7:50, Andrew Price <anprice@redhat.com>?? ??:

> Hi Daegyu,
>
> On 09/08/2019 09:10, ??? wrote:
> > Hi, I'm Daegyu from Sungkyunkwan University.
> >
> > I'm curious how GFS2's filesystem metadata is shared between nodes.
>
> The key thing to know about gfs2 is that it is a shared storage
> filesystem where each node mounts the same storage device. It is
> different from a distributed filesystem where each node has storage
> devices that only it accesses.
>
> > In detail, I wonder how the metadata in the memory of the node mounting
> GFS2
> > looks the consistent filesystem to other nodes.
>
> gfs2 uses dlm for locking of filesystem metadata among the nodes. The
> transfer of locks between nodes allows gfs2 to decide when its in-memory
> caches are invalid and require re-reading from the storage.
>
> > In addition, what role does corosync play in gfs2?
>
> gfs2 doesn't communicate with corosync directly but it operates on top
> of a high-availability cluster. corosync provides synchronization and
> coherency for the cluster. If a node stops responding, corosync will
> notice and trigger actions (fencing) to make sure that node is put back
> into a safe and consistent state. This is important in gfs2 to prevent
> "misbehaving" nodes from corrupting the filesystem.
>
> Hope this helps.
>
> Cheers,
> Andy
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20190809/be2e1c6c/attachment.htm>

[Cluster-devel] How can be metadata(e.g., inode) in the GFS2 file system shared between client nodes?

[Andrew Price]

On 09/08/2019 12:01, Daegyu Han wrote:
> Thank you for your reply.
> 
> If what I understand is correct,
> In a gfs2 file system shared by clients A and B, if A creates /foo/a.txt,
> does B re-read the filesystem metadata area on storage to keep the data
> consistent?

Yes, that's correct, although 'clients' is inaccurate as there is no 
'server'. Through the locking mechanism, B would know to re-read block 
allocation states and the contents of the /foo directory, so a path 
lookup on B would then find a.txt.

> After all, what makes gfs2 different from local filesystems like ext4,
> because of lock_dlm?

Exactly.

> In general, if we mount an ext4 file system on two different clients and
> update the file system on each client, we know that the file system state
> is not reflected in each other.

Yes.

Cheers,
Andy

> Thank you,
> Daegyu
> ?
> 
> 2019? 8? 9? (?) ?? 7:50, Andrew Price <anprice@redhat.com>?? ??:
> 
>> Hi Daegyu,
>>
>> On 09/08/2019 09:10, ??? wrote:
>>> Hi, I'm Daegyu from Sungkyunkwan University.
>>>
>>> I'm curious how GFS2's filesystem metadata is shared between nodes.
>>
>> The key thing to know about gfs2 is that it is a shared storage
>> filesystem where each node mounts the same storage device. It is
>> different from a distributed filesystem where each node has storage
>> devices that only it accesses.
>>
>>> In detail, I wonder how the metadata in the memory of the node mounting
>> GFS2
>>> looks the consistent filesystem to other nodes.
>>
>> gfs2 uses dlm for locking of filesystem metadata among the nodes. The
>> transfer of locks between nodes allows gfs2 to decide when its in-memory
>> caches are invalid and require re-reading from the storage.
>>
>>> In addition, what role does corosync play in gfs2?
>>
>> gfs2 doesn't communicate with corosync directly but it operates on top
>> of a high-availability cluster. corosync provides synchronization and
>> coherency for the cluster. If a node stops responding, corosync will
>> notice and trigger actions (fencing) to make sure that node is put back
>> into a safe and consistent state. This is important in gfs2 to prevent
>> "misbehaving" nodes from corrupting the filesystem.
>>
>> Hope this helps.
>>
>> Cheers,
>> Andy
>>
>>
>>
> 

[Cluster-devel] How can be metadata(e.g., inode) in the GFS2 file system shared between client nodes?

[Daegyu Han]

Thank you for the clarification.

I have one more question.

I've seen some web page by redhat and it says that gfs2 has a poor
filesystem performance (i.e. throughput) compared to xfs or ext4.
[image: image.png]

In a high performance hardware environment (nvme over fabric, infiniband
(56G)), I ran a FIO benchmark, expecting GFS2 to be comparable to local
filesystems (ext4, xfs).

Unexpectedly, however, GFS2 showed 25% lower IOPS or throughput than ext4,
as the web page results.

Does GFS2 perform worse than EXT4 or XFS even on high-performance network +
storage?

Thank you,
Daegyu
?

2019? 8? 9? (?) ?? 8:26, Andrew Price <anprice@redhat.com>?? ??:

> On 09/08/2019 12:01, Daegyu Han wrote:
> > Thank you for your reply.
> >
> > If what I understand is correct,
> > In a gfs2 file system shared by clients A and B, if A creates /foo/a.txt,
> > does B re-read the filesystem metadata area on storage to keep the data
> > consistent?
>
> Yes, that's correct, although 'clients' is inaccurate as there is no
> 'server'. Through the locking mechanism, B would know to re-read block
> allocation states and the contents of the /foo directory, so a path
> lookup on B would then find a.txt.
>
> > After all, what makes gfs2 different from local filesystems like ext4,
> > because of lock_dlm?
>
> Exactly.
>
> > In general, if we mount an ext4 file system on two different clients and
> > update the file system on each client, we know that the file system state
> > is not reflected in each other.
>
> Yes.
>
> Cheers,
> Andy
>
> > Thank you,
> > Daegyu
> > ?
> >
> > 2019? 8? 9? (?) ?? 7:50, Andrew Price <anprice@redhat.com>?? ??:
> >
> >> Hi Daegyu,
> >>
> >> On 09/08/2019 09:10, ??? wrote:
> >>> Hi, I'm Daegyu from Sungkyunkwan University.
> >>>
> >>> I'm curious how GFS2's filesystem metadata is shared between nodes.
> >>
> >> The key thing to know about gfs2 is that it is a shared storage
> >> filesystem where each node mounts the same storage device. It is
> >> different from a distributed filesystem where each node has storage
> >> devices that only it accesses.
> >>
> >>> In detail, I wonder how the metadata in the memory of the node mounting
> >> GFS2
> >>> looks the consistent filesystem to other nodes.
> >>
> >> gfs2 uses dlm for locking of filesystem metadata among the nodes. The
> >> transfer of locks between nodes allows gfs2 to decide when its in-memory
> >> caches are invalid and require re-reading from the storage.
> >>
> >>> In addition, what role does corosync play in gfs2?
> >>
> >> gfs2 doesn't communicate with corosync directly but it operates on top
> >> of a high-availability cluster. corosync provides synchronization and
> >> coherency for the cluster. If a node stops responding, corosync will
> >> notice and trigger actions (fencing) to make sure that node is put back
> >> into a safe and consistent state. This is important in gfs2 to prevent
> >> "misbehaving" nodes from corrupting the filesystem.
> >>
> >> Hope this helps.
> >>
> >> Cheers,
> >> Andy
> >>
> >>
> >>
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20190809/eceadd76/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 78518 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20190809/eceadd76/attachment.png>

[Cluster-devel] How can be metadata(e.g., inode) in the GFS2 file system shared between client nodes?

[Andrew Price]

On 09/08/2019 12:46, Daegyu Han wrote:
> Thank you for the clarification.
> 
> I have one more question.
> 
> I've seen some web page by redhat and it says that gfs2 has a poor
> filesystem performance (i.e. throughput) compared to xfs or ext4.
> [image: image.png]
> 
> In a high performance hardware environment (nvme over fabric, infiniband
> (56G)), I ran a FIO benchmark, expecting GFS2 to be comparable to local
> filesystems (ext4, xfs).
> 
> Unexpectedly, however, GFS2 showed 25% lower IOPS or throughput than ext4,
> as the web page results.
> 
> Does GFS2 perform worse than EXT4 or XFS even on high-performance network +
> storage?

gfs2 has performance overheads that ext4 and xfs don't encounter due to 
the extra work it has to do to keep the fs consistent across the 
cluster, such as the extra cache invalidation we've discussed, journal 
flushing and updates to structures relating to quotas and statfs. Even 
in a single-node configuration, extra codepaths are still active (but 
gfs2 isn't meant to be used as a single-node fs, so fio is not a good 
demonstration of its strengths). It's also worth noting that gfs2 is not 
extent-based so you may see performance differences relating to that. We 
are continually working to minimise the overheads, of course.

The size of the performance difference is highly dependent on the 
workload and access pattern. (Clustered) applications looking to get the 
best performance out of gfs2 will have each node processing its own 
working set - preferably in its own subdirectory - which will minimise 
the overheads.

Cheers,
Andy

> Thank you,
> Daegyu
> ?
> 
> 2019? 8? 9? (?) ?? 8:26, Andrew Price <anprice@redhat.com>?? ??:
> 
>> On 09/08/2019 12:01, Daegyu Han wrote:
>>> Thank you for your reply.
>>>
>>> If what I understand is correct,
>>> In a gfs2 file system shared by clients A and B, if A creates /foo/a.txt,
>>> does B re-read the filesystem metadata area on storage to keep the data
>>> consistent?
>>
>> Yes, that's correct, although 'clients' is inaccurate as there is no
>> 'server'. Through the locking mechanism, B would know to re-read block
>> allocation states and the contents of the /foo directory, so a path
>> lookup on B would then find a.txt.
>>
>>> After all, what makes gfs2 different from local filesystems like ext4,
>>> because of lock_dlm?
>>
>> Exactly.
>>
>>> In general, if we mount an ext4 file system on two different clients and
>>> update the file system on each client, we know that the file system state
>>> is not reflected in each other.
>>
>> Yes.
>>
>> Cheers,
>> Andy
>>
>>> Thank you,
>>> Daegyu
>>> ?
>>>
>>> 2019? 8? 9? (?) ?? 7:50, Andrew Price <anprice@redhat.com>?? ??:
>>>
>>>> Hi Daegyu,
>>>>
>>>> On 09/08/2019 09:10, ??? wrote:
>>>>> Hi, I'm Daegyu from Sungkyunkwan University.
>>>>>
>>>>> I'm curious how GFS2's filesystem metadata is shared between nodes.
>>>>
>>>> The key thing to know about gfs2 is that it is a shared storage
>>>> filesystem where each node mounts the same storage device. It is
>>>> different from a distributed filesystem where each node has storage
>>>> devices that only it accesses.
>>>>
>>>>> In detail, I wonder how the metadata in the memory of the node mounting
>>>> GFS2
>>>>> looks the consistent filesystem to other nodes.
>>>>
>>>> gfs2 uses dlm for locking of filesystem metadata among the nodes. The
>>>> transfer of locks between nodes allows gfs2 to decide when its in-memory
>>>> caches are invalid and require re-reading from the storage.
>>>>
>>>>> In addition, what role does corosync play in gfs2?
>>>>
>>>> gfs2 doesn't communicate with corosync directly but it operates on top
>>>> of a high-availability cluster. corosync provides synchronization and
>>>> coherency for the cluster. If a node stops responding, corosync will
>>>> notice and trigger actions (fencing) to make sure that node is put back
>>>> into a safe and consistent state. This is important in gfs2 to prevent
>>>> "misbehaving" nodes from corrupting the filesystem.
>>>>
>>>> Hope this helps.
>>>>
>>>> Cheers,
>>>> Andy
>>>>
>>>>
>>>>
>>>
>>
> 

[Cluster-devel] How can be metadata(e.g., inode) in the GFS2 file system shared between client nodes?

[Daegyu Han]

Thank you for your explanation.

?

2019? 8? 9? (?) ?? 9:26, Andrew Price <anprice@redhat.com>?? ??:

> On 09/08/2019 12:46, Daegyu Han wrote:
> > Thank you for the clarification.
> >
> > I have one more question.
> >
> > I've seen some web page by redhat and it says that gfs2 has a poor
> > filesystem performance (i.e. throughput) compared to xfs or ext4.
> > [image: image.png]
> >
> > In a high performance hardware environment (nvme over fabric, infiniband
> > (56G)), I ran a FIO benchmark, expecting GFS2 to be comparable to local
> > filesystems (ext4, xfs).
> >
> > Unexpectedly, however, GFS2 showed 25% lower IOPS or throughput than
> ext4,
> > as the web page results.
> >
> > Does GFS2 perform worse than EXT4 or XFS even on high-performance
> network +
> > storage?
>
> gfs2 has performance overheads that ext4 and xfs don't encounter due to
> the extra work it has to do to keep the fs consistent across the
> cluster, such as the extra cache invalidation we've discussed, journal
> flushing and updates to structures relating to quotas and statfs. Even
> in a single-node configuration, extra codepaths are still active (but
> gfs2 isn't meant to be used as a single-node fs, so fio is not a good
> demonstration of its strengths). It's also worth noting that gfs2 is not
> extent-based so you may see performance differences relating to that. We
> are continually working to minimise the overheads, of course.
>
> The size of the performance difference is highly dependent on the
> workload and access pattern. (Clustered) applications looking to get the
> best performance out of gfs2 will have each node processing its own
> working set - preferably in its own subdirectory - which will minimise
> the overheads.
>
> Cheers,
> Andy
>
> > Thank you,
> > Daegyu
> > ?
> >
> > 2019? 8? 9? (?) ?? 8:26, Andrew Price <anprice@redhat.com>?? ??:
> >
> >> On 09/08/2019 12:01, Daegyu Han wrote:
> >>> Thank you for your reply.
> >>>
> >>> If what I understand is correct,
> >>> In a gfs2 file system shared by clients A and B, if A creates
> /foo/a.txt,
> >>> does B re-read the filesystem metadata area on storage to keep the data
> >>> consistent?
> >>
> >> Yes, that's correct, although 'clients' is inaccurate as there is no
> >> 'server'. Through the locking mechanism, B would know to re-read block
> >> allocation states and the contents of the /foo directory, so a path
> >> lookup on B would then find a.txt.
> >>
> >>> After all, what makes gfs2 different from local filesystems like ext4,
> >>> because of lock_dlm?
> >>
> >> Exactly.
> >>
> >>> In general, if we mount an ext4 file system on two different clients
> and
> >>> update the file system on each client, we know that the file system
> state
> >>> is not reflected in each other.
> >>
> >> Yes.
> >>
> >> Cheers,
> >> Andy
> >>
> >>> Thank you,
> >>> Daegyu
> >>> ?
> >>>
> >>> 2019? 8? 9? (?) ?? 7:50, Andrew Price <anprice@redhat.com>?? ??:
> >>>
> >>>> Hi Daegyu,
> >>>>
> >>>> On 09/08/2019 09:10, ??? wrote:
> >>>>> Hi, I'm Daegyu from Sungkyunkwan University.
> >>>>>
> >>>>> I'm curious how GFS2's filesystem metadata is shared between nodes.
> >>>>
> >>>> The key thing to know about gfs2 is that it is a shared storage
> >>>> filesystem where each node mounts the same storage device. It is
> >>>> different from a distributed filesystem where each node has storage
> >>>> devices that only it accesses.
> >>>>
> >>>>> In detail, I wonder how the metadata in the memory of the node
> mounting
> >>>> GFS2
> >>>>> looks the consistent filesystem to other nodes.
> >>>>
> >>>> gfs2 uses dlm for locking of filesystem metadata among the nodes. The
> >>>> transfer of locks between nodes allows gfs2 to decide when its
> in-memory
> >>>> caches are invalid and require re-reading from the storage.
> >>>>
> >>>>> In addition, what role does corosync play in gfs2?
> >>>>
> >>>> gfs2 doesn't communicate with corosync directly but it operates on top
> >>>> of a high-availability cluster. corosync provides synchronization and
> >>>> coherency for the cluster. If a node stops responding, corosync will
> >>>> notice and trigger actions (fencing) to make sure that node is put
> back
> >>>> into a safe and consistent state. This is important in gfs2 to prevent
> >>>> "misbehaving" nodes from corrupting the filesystem.
> >>>>
> >>>> Hope this helps.
> >>>>
> >>>> Cheers,
> >>>> Andy
> >>>>
> >>>>
> >>>>
> >>>
> >>
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20190809/4e83a043/attachment.htm>