[Question] Btrfs on iSCSI device

[Question] Btrfs on iSCSI device

[Zhe Zhang]

Hi,

I setup 2 Linux servers to share the same device through iSCSI. Then I
created a btrfs on the device. Then I saw the problem that the 2 Linux
servers do not see a consistent file system image.

Details:
-- Server 1 running kernel 2.6.32, server 2 running 3.2.1
-- Both running btrfs v0.20-rc1
-- Server 2 has device /dev/vdc, exposed as iSCSI target
 -- Server 1 mounts the device as /dev/sda
-- Server 1 'mount /dev/sda /mnt/btrfs'; server 2 'mount /dev/vdc /mnt/btrfs',
 -- When server 1 'touch /mnt/btrfs/foo', server 2 doesn't see any
file under /mnt/btrfs
-- I created /mnt/btrfs/foo on server 2 as well; then I added some
content from both server 1 and server 2 to /mnt/btrfs/foo
-- After that each server sees the content it adds, but not the
content from the other server
-- Both server 'umount /mnt/btrfs', and mount it again
-- Then both servers see /mnt/btrfs/foo with the content added from
server 2 (I guess it's because server 2 created the foo file later
than server 1).

I did a similar test on ext4 and both servers see a consistent image
of the file system. When server 1 creates a foo file server 2
immediately sees it.

Is this how btrfs is supposed to work?

Thanks,

Zhe

Re: [Question] Btrfs on iSCSI device

[Goffredo Baroncelli]

Hi,
On 06/27/2014 05:44 PM, Zhe Zhang wrote:
> Hi,
> 
> I setup 2 Linux servers to share the same device through iSCSI. Then I
> created a btrfs on the device. Then I saw the problem that the 2 Linux
> servers do not see a consistent file system image.
> 
> Details:
> -- Server 1 running kernel 2.6.32, server 2 running 3.2.1
> -- Both running btrfs v0.20-rc1
> -- Server 2 has device /dev/vdc, exposed as iSCSI target
>  -- Server 1 mounts the device as /dev/sda
> -- Server 1 'mount /dev/sda /mnt/btrfs'; server 2 'mount /dev/vdc /mnt/btrfs',
>  -- When server 1 'touch /mnt/btrfs/foo', server 2 doesn't see any
> file under /mnt/btrfs
> -- I created /mnt/btrfs/foo on server 2 as well; then I added some
> content from both server 1 and server 2 to /mnt/btrfs/foo
> -- After that each server sees the content it adds, but not the
> content from the other server
> -- Both server 'umount /mnt/btrfs', and mount it again
> -- Then both servers see /mnt/btrfs/foo with the content added from
> server 2 (I guess it's because server 2 created the foo file later
> than server 1).
> 
> I did a similar test on ext4 and both servers see a consistent image
> of the file system. When server 1 creates a foo file server 2
> immediately sees it.
> 
> Is this how btrfs is supposed to work?

I don't think that it is possible to mount the _same device_ at the _same time_ on two different machines. And this doesn't depend by the filesystem.

The fact that you see it working, I suspect that is is casual.

When I tried this (same scsi HD connected to two machines), I had to ensure that the two machines never accessed to the HD at the same time.

> 
> Thanks,
> 
> Zhe
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli (kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: [Question] Btrfs on iSCSI device

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 2383 bytes --]

On 2014-06-27 12:34, Goffredo Baroncelli wrote:
> Hi,
> On 06/27/2014 05:44 PM, Zhe Zhang wrote:
>> Hi,
>>
>> I setup 2 Linux servers to share the same device through iSCSI. Then I
>> created a btrfs on the device. Then I saw the problem that the 2 Linux
>> servers do not see a consistent file system image.
>>
>> Details:
>> -- Server 1 running kernel 2.6.32, server 2 running 3.2.1
>> -- Both running btrfs v0.20-rc1
>> -- Server 2 has device /dev/vdc, exposed as iSCSI target
>>  -- Server 1 mounts the device as /dev/sda
>> -- Server 1 'mount /dev/sda /mnt/btrfs'; server 2 'mount /dev/vdc /mnt/btrfs',
>>  -- When server 1 'touch /mnt/btrfs/foo', server 2 doesn't see any
>> file under /mnt/btrfs
>> -- I created /mnt/btrfs/foo on server 2 as well; then I added some
>> content from both server 1 and server 2 to /mnt/btrfs/foo
>> -- After that each server sees the content it adds, but not the
>> content from the other server
>> -- Both server 'umount /mnt/btrfs', and mount it again
>> -- Then both servers see /mnt/btrfs/foo with the content added from
>> server 2 (I guess it's because server 2 created the foo file later
>> than server 1).
>>
>> I did a similar test on ext4 and both servers see a consistent image
>> of the file system. When server 1 creates a foo file server 2
>> immediately sees it.
>>
>> Is this how btrfs is supposed to work?
> 
> I don't think that it is possible to mount the _same device_ at the _same time_ on two different machines. And this doesn't depend by the filesystem.
> 
> The fact that you see it working, I suspect that is is casual.
> 
> When I tried this (same scsi HD connected to two machines), I had to ensure that the two machines never accessed to the HD at the same time.
> 
>>
>> Thanks,
>>
>> Zhe
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> 
> 
If you need shared storage like that, you need to use a real cluster
filesystem like GFS2 or OCFS2, BTRFS isn't designed for any kind of
concurrent access to shared storage from separate systems.
The reason it appears to work when using iSCSI and not with directly
connected parallel SCSI or SAS is that iSCSI doesn't provide low level
hardware access.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2967 bytes --]

Re: [Question] Btrfs on iSCSI device

[Zhe Zhang]

On Fri, Jun 27, 2014 at 1:15 PM, Austin S Hemmelgarn
<ahferroin7@gmail.com> wrote:
> On 2014-06-27 12:34, Goffredo Baroncelli wrote:
>> Hi,
>> On 06/27/2014 05:44 PM, Zhe Zhang wrote:
>>> Hi,
>>>
>>> I setup 2 Linux servers to share the same device through iSCSI. Then I
>>> created a btrfs on the device. Then I saw the problem that the 2 Linux
>>> servers do not see a consistent file system image.
>>>
>>> Details:
>>> -- Server 1 running kernel 2.6.32, server 2 running 3.2.1
>>> -- Both running btrfs v0.20-rc1
>>> -- Server 2 has device /dev/vdc, exposed as iSCSI target
>>>  -- Server 1 mounts the device as /dev/sda
>>> -- Server 1 'mount /dev/sda /mnt/btrfs'; server 2 'mount /dev/vdc /mnt/btrfs',
>>>  -- When server 1 'touch /mnt/btrfs/foo', server 2 doesn't see any
>>> file under /mnt/btrfs
>>> -- I created /mnt/btrfs/foo on server 2 as well; then I added some
>>> content from both server 1 and server 2 to /mnt/btrfs/foo
>>> -- After that each server sees the content it adds, but not the
>>> content from the other server
>>> -- Both server 'umount /mnt/btrfs', and mount it again
>>> -- Then both servers see /mnt/btrfs/foo with the content added from
>>> server 2 (I guess it's because server 2 created the foo file later
>>> than server 1).
>>>
>>> I did a similar test on ext4 and both servers see a consistent image
>>> of the file system. When server 1 creates a foo file server 2
>>> immediately sees it.
>>>
>>> Is this how btrfs is supposed to work?
>>
>> I don't think that it is possible to mount the _same device_ at the _same time_ on two different machines. And this doesn't depend by the filesystem.
>>
>> The fact that you see it working, I suspect that is is casual.
>>
>> When I tried this (same scsi HD connected to two machines), I had to ensure that the two machines never accessed to the HD at the same time.
>>
>>>
>>> Thanks,
>>>
>>> Zhe
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>>
> If you need shared storage like that, you need to use a real cluster
> filesystem like GFS2 or OCFS2, BTRFS isn't designed for any kind of
> concurrent access to shared storage from separate systems.
> The reason it appears to work when using iSCSI and not with directly
> connected parallel SCSI or SAS is that iSCSI doesn't provide low level
> hardware access.
>

I did more testing with ext4 and it supports what Goffredo and Austin
said above. Error message is "cannot access xxx: Input/output error".

It seems to me that both servers hold some file system data structures
in memory and eventually conflict with each other (like writing inode
info to the same blocks).

Re: [Question] Btrfs on iSCSI device

[ronnie sahlberg]

You can't do that, it doesn't work.

The only way you can access the same block device from multiple hosts
at the same time is if you use a cluster filesystem.
Using a non-cluster filesystem like this is highly unsafe and will
very likely quickly lead to total dataloss.

(
I am a little bit surprised that you managed to mount ext4 from both
hosts at the same time.
ext4 SHOULD have put a SCSI [PERSISTENT] RESERVATION on the disk on
the first mount
in order to prevent a second host from accessing the device.

Does btrfs put a reservation on the underlying SCSI device when it
mounts the filesystem?
If not, it should.
)

If you want to share data read/write across multiple hosts you should
either use a file server and a network file system such as NFS or
CIFS.
Or if you want to use block/iSCSI, then you must use a proper cluster
filesystem of which there are several available for linux.


 regards
ronnie sahlberg


On Fri, Jun 27, 2014 at 8:44 AM, Zhe Zhang <zhe.zhang.research@gmail.com> wrote:
> Hi,
>
> I setup 2 Linux servers to share the same device through iSCSI. Then I
> created a btrfs on the device. Then I saw the problem that the 2 Linux
> servers do not see a consistent file system image.
>
> Details:
> -- Server 1 running kernel 2.6.32, server 2 running 3.2.1
> -- Both running btrfs v0.20-rc1
> -- Server 2 has device /dev/vdc, exposed as iSCSI target
>  -- Server 1 mounts the device as /dev/sda
> -- Server 1 'mount /dev/sda /mnt/btrfs'; server 2 'mount /dev/vdc /mnt/btrfs',
>  -- When server 1 'touch /mnt/btrfs/foo', server 2 doesn't see any
> file under /mnt/btrfs
> -- I created /mnt/btrfs/foo on server 2 as well; then I added some
> content from both server 1 and server 2 to /mnt/btrfs/foo
> -- After that each server sees the content it adds, but not the
> content from the other server
> -- Both server 'umount /mnt/btrfs', and mount it again
> -- Then both servers see /mnt/btrfs/foo with the content added from
> server 2 (I guess it's because server 2 created the foo file later
> than server 1).
>
> I did a similar test on ext4 and both servers see a consistent image
> of the file system. When server 1 creates a foo file server 2
> immediately sees it.
>
> Is this how btrfs is supposed to work?
>
> Thanks,
>
> Zhe
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Question] Btrfs on iSCSI device

[Russell Coker]

On Fri, 27 Jun 2014 18:34:34 Goffredo Baroncelli wrote:
> I don't think that it is possible to mount the _same device_ at the _same
> time_ on two different machines. And this doesn't depend by the filesystem.

If you use a clustered filesystem then you can safely mount it on multiple 
machines.

If you use a non-clustered filesystem it can still mount and even appear to 
work for a while.  It's surprising how many writes you can make to a dual-
mounted filesystem that's not designed for such things before you get a 
totally broken filesystem.

On Fri, 27 Jun 2014 13:15:16 Austin S Hemmelgarn wrote:
> The reason it appears to work when using iSCSI and not with directly
> connected parallel SCSI or SAS is that iSCSI doesn't provide low level
> hardware access.

I've tried this with dual-attached FC and had no problems mounting.  In what 
way is directly connected SCSI different from FC?

-- 
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/

Re: [Question] Btrfs on iSCSI device

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1881 bytes --]

On 06/27/2014 07:40 PM, Russell Coker wrote:
> On Fri, 27 Jun 2014 18:34:34 Goffredo Baroncelli wrote:
>> I don't think that it is possible to mount the _same device_ at the _same
>> time_ on two different machines. And this doesn't depend by the filesystem.
> 
> If you use a clustered filesystem then you can safely mount it on multiple 
> machines.
> 
> If you use a non-clustered filesystem it can still mount and even appear to 
> work for a while.  It's surprising how many writes you can make to a dual-
> mounted filesystem that's not designed for such things before you get a 
> totally broken filesystem.
> 
> On Fri, 27 Jun 2014 13:15:16 Austin S Hemmelgarn wrote:
>> The reason it appears to work when using iSCSI and not with directly
>> connected parallel SCSI or SAS is that iSCSI doesn't provide low level
>> hardware access.
> 
> I've tried this with dual-attached FC and had no problems mounting.  In what 
> way is directly connected SCSI different from FC?
> 
FC is actually it's own networking stack (and you can even run (in
theory) other protocols like IP and ATM on top of it), whereas parallel
SCSI is just a multi-drop bus, and SAS is just a tree-structured bus
with point-to-point communications emulated on top of it.  In other
words, parallel SCSI has topological constraints like RS-485, SAS has
topology constraints like USB, and FC has topology constraints like
Ethernet.

Secondarily, most filesystems on Linux will let you mount them multiple
times on separate hosts (ext4 has features to prevent this, but they are
expensive and therefore turned off by default, I think XFS might have
similar features, but I'm not sure).  BTRFS should in theory be more
resilient than most because of the COW nature (as long as it's only a
few commit cycles, you should still be able to recover most of the data
just fine).


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2967 bytes --]