* mkcephfs failing on v0.48 "argonaut"
@ 2012-07-04 5:07 Paul Pettigrew
2012-07-05 22:17 ` Sage Weil
0 siblings, 1 reply; 9+ messages in thread
From: Paul Pettigrew @ 2012-07-04 5:07 UTC (permalink / raw)
To: ceph-devel
Firstly, well done guys on achieving this version milestone. I successfully upgraded to the 0.48 format uneventfully on a live (test) system.
The same system was then going through "rebuild" testing, to confirm that also worked fine.
Unfortunately, the mkcephfs command is failing:
root@dsanb1-coy:~# mkcephfs -c /etc/ceph/ceph.conf --allhosts --mkbtrfs -k /etc/ceph/keyring --crushmapsrc crushfile.txt -v
temp dir is /tmp/mkcephfs.GaRCZ9i06a
preparing monmap in /tmp/mkcephfs.GaRCZ9i06a/monmap
/usr/bin/monmaptool --create --clobber --add alpha 10.32.0.10:6789 --add bravo 10.32.0.25:6789 --add charlie 10.32.0.11:6789 --print /tmp/mkcephfs.GaRCZ9i06a/monmap
/usr/bin/monmaptool: monmap file /tmp/mkcephfs.GaRCZ9i06a/monmap
/usr/bin/monmaptool: generated fsid c7202495-468c-4678-b678-115c3ee33402
epoch 0
fsid c7202495-468c-4678-b678-115c3ee33402
last_changed 2012-07-04 15:02:31.732275
created 2012-07-04 15:02:31.732275
0: 10.32.0.10:6789/0 mon.alpha
1: 10.32.0.11:6789/0 mon.charlie
2: 10.32.0.25:6789/0 mon.bravo
/usr/bin/monmaptool: writing epoch 0 to /tmp/mkcephfs.GaRCZ9i06a/monmap (3 monitors)
/usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.0 "user"
=== osd.0 ===
--- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.GaRCZ9i06a --prepare-osdfs osd.0
umount: /srv/osd.0: not mounted
umount: /dev/disk/by-wwn/wwn-0x50014ee601246234: not mounted
WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using
fs created label (null) on /dev/disk/by-wwn/wwn-0x50014ee601246234
nodesize 4096 leafsize 4096 sectorsize 4096 size 1.82TB
Btrfs Btrfs v0.19
Scanning for Btrfs filesystems
mount: wrong fs type, bad option, bad superblock on /dev/sdc,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
failed: '/sbin/mkcephfs -d /tmp/mkcephfs.GaRCZ9i06a --prepare-osdfs osd.0'
dmesg/syslog is spitting out at the time of this failure:
Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.751945] device fsid 7de0d192-b710-4629-a201-849df1d9db17 devid 1 transid 27109 /dev/sdp
Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.751987] device fsid 08fc3479-2fa2-4388-8b61-83e2a742a13e devid 1 transid 28699 /dev/sdo
Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.752023] device fsid 8b4a7c43-1a05-4dcb-bbed-de2a5c933996 devid 1 transid 24346 /dev/sdn
Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.752068] device fsid ba5fb1ca-c642-49b1-8a41-7f56f8e59fbd devid 1 transid 27274 /dev/sdm
Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761453] device fsid 7fe8c5cf-bf8c-4276-90f2-c3f57f5275fb devid 1 transid 28724 /dev/sdi
Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761518] device fsid 93fa3631-1202-4d42-8908-e5ef4d3e600d devid 1 transid 25201 /dev/sdh
Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761579] device fsid b9a1b5e4-3e5e-4381-a29a-33470f4b870f devid 1 transid 23375 /dev/sdg
Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761635] device fsid 280ea990-23f8-4c43-9e56-140c82340fdc devid 1 transid 25559 /dev/sdf
Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761693] device fsid 2f724cde-6de5-4262-b195-1ba3eea2256e devid 1 transid 176 /dev/sde
Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761732] device fsid a66f890f-8b08-4393-aab0-f222637ca5a4 devid 1 transid 7 /dev/sdd
Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761769] device fsid 6c181a94-697c-4e0c-af0d-05eb04d3626c devid 1 transid 7 /dev/sdc
Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.775931] device fsid 6c181a94-697c-4e0c-af0d-05eb04d3626c devid 1 transid 7 /dev/sdc
Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.779716] btrfs bad fsid on block 20971520
Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.791594] btrfs bad fsid on block 20971520
Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.803608] btrfs bad fsid on block 20971520
Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.815541] btrfs bad fsid on block 20971520
Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.815878] btrfs bad fsid on block 20971520
Jul 4 15:02:32 dsanb1-coy kernel: [ 2306.823554] btrfs bad fsid on block 20971520
Jul 4 15:02:32 dsanb1-coy kernel: [ 2306.823797] btrfs bad fsid on block 20971520
Jul 4 15:02:32 dsanb1-coy kernel: [ 2306.823887] btrfs: failed to read chunk root on sdc
Jul 4 15:02:32 dsanb1-coy kernel: [ 2306.825622] btrfs: open_ctree failed
Also fails if not forcing to use btrfs, eg:
root@dsanb1-coy:~# mkcephfs -c /etc/ceph/ceph.conf --allhosts -k /etc/ceph/keyring --crushmapsrc crushfile.txt -v
temp dir is /tmp/mkcephfs.ZOh6tBPAH0
preparing monmap in /tmp/mkcephfs.ZOh6tBPAH0/monmap
/usr/bin/monmaptool --create --clobber --add alpha 10.32.0.10:6789 --add bravo 10.32.0.25:6789 --add charlie 10.32.0.11:6789 --print /tmp/mkcephfs.ZOh6tBPAH0/monmap
/usr/bin/monmaptool: monmap file /tmp/mkcephfs.ZOh6tBPAH0/monmap
/usr/bin/monmaptool: generated fsid adb8d65c-a823-4dc2-9415-22b0d7252699
epoch 0
fsid adb8d65c-a823-4dc2-9415-22b0d7252699
last_changed 2012-07-04 15:04:17.423368
created 2012-07-04 15:04:17.423368
0: 10.32.0.10:6789/0 mon.alpha
1: 10.32.0.11:6789/0 mon.charlie
2: 10.32.0.25:6789/0 mon.bravo
/usr/bin/monmaptool: writing epoch 0 to /tmp/mkcephfs.ZOh6tBPAH0/monmap (3 monitors)
/usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.0 "user"
=== osd.0 ===
--- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.ZOh6tBPAH0 --init-daemon osd.0
2012-07-04 15:04:17.789064 7fc7fadca780 -1 filestore(/srv/osd.0) limited size xattrs -- enable filestore_xattr_use_omap
2012-07-04 15:04:17.789120 7fc7fadca780 -1 OSD::mkfs: couldn't mount FileStore: error -95
2012-07-04 15:04:17.789161 7fc7fadca780 -1 ** ERROR: error creating empty object store in /srv/osd.0: (95) Operation not supported
failed: '/sbin/mkcephfs -d /tmp/mkcephfs.ZOh6tBPAH0 --init-daemon osd.0'
Confirming all this was working previously, and the crushmap, config file, etc are all proven to be OK (get same failure when not specifying a custom crushmap also). Also note that whilst the above is failing on osd.0 creation, I have swapped disk references and still get the same failure on different HDD's when they are hooked in as osd.0
The setup is: Ubuntu 12.04 x64, with "main/release" ceph installed per: http://ceph.com/docs/master/install/debian/
Many thanks!
Paul
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: mkcephfs failing on v0.48 "argonaut"
2012-07-04 5:07 mkcephfs failing on v0.48 "argonaut" Paul Pettigrew
@ 2012-07-05 22:17 ` Sage Weil
2012-07-06 2:39 ` Paul Pettigrew
0 siblings, 1 reply; 9+ messages in thread
From: Sage Weil @ 2012-07-05 22:17 UTC (permalink / raw)
To: Paul Pettigrew; +Cc: ceph-devel
Hi Paul,
On Wed, 4 Jul 2012, Paul Pettigrew wrote:
> Firstly, well done guys on achieving this version milestone. I
> successfully upgraded to the 0.48 format uneventfully on a live (test)
> system.
>
> The same system was then going through "rebuild" testing, to confirm
> that also worked fine.
>
>
> Unfortunately, the mkcephfs command is failing:
>
> root@dsanb1-coy:~# mkcephfs -c /etc/ceph/ceph.conf --allhosts --mkbtrfs -k /etc/ceph/keyring --crushmapsrc crushfile.txt -v
> temp dir is /tmp/mkcephfs.GaRCZ9i06a
> preparing monmap in /tmp/mkcephfs.GaRCZ9i06a/monmap
> /usr/bin/monmaptool --create --clobber --add alpha 10.32.0.10:6789 --add bravo 10.32.0.25:6789 --add charlie 10.32.0.11:6789 --print /tmp/mkcephfs.GaRCZ9i06a/monmap
> /usr/bin/monmaptool: monmap file /tmp/mkcephfs.GaRCZ9i06a/monmap
> /usr/bin/monmaptool: generated fsid c7202495-468c-4678-b678-115c3ee33402
> epoch 0
> fsid c7202495-468c-4678-b678-115c3ee33402
> last_changed 2012-07-04 15:02:31.732275
> created 2012-07-04 15:02:31.732275
> 0: 10.32.0.10:6789/0 mon.alpha
> 1: 10.32.0.11:6789/0 mon.charlie
> 2: 10.32.0.25:6789/0 mon.bravo
> /usr/bin/monmaptool: writing epoch 0 to /tmp/mkcephfs.GaRCZ9i06a/monmap (3 monitors)
> /usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.0 "user"
> === osd.0 ===
> --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.GaRCZ9i06a --prepare-osdfs osd.0
> umount: /srv/osd.0: not mounted
> umount: /dev/disk/by-wwn/wwn-0x50014ee601246234: not mounted
>
> WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL
> WARNING! - see http://btrfs.wiki.kernel.org before using
>
> fs created label (null) on /dev/disk/by-wwn/wwn-0x50014ee601246234
> nodesize 4096 leafsize 4096 sectorsize 4096 size 1.82TB
> Btrfs Btrfs v0.19
> Scanning for Btrfs filesystems
> mount: wrong fs type, bad option, bad superblock on /dev/sdc,
> missing codepage or helper program, or other error
> In some cases useful info is found in syslog - try
> dmesg | tail or so
>
> failed: '/sbin/mkcephfs -d /tmp/mkcephfs.GaRCZ9i06a --prepare-osdfs osd.0'
Hmm. Can you try running with -v? That will tell us exactly which
command it is running, and hopefully we can work backwards from there.
> dmesg/syslog is spitting out at the time of this failure:
>
> Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.751945] device fsid 7de0d192-b710-4629-a201-849df1d9db17 devid 1 transid 27109 /dev/sdp
> Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.751987] device fsid 08fc3479-2fa2-4388-8b61-83e2a742a13e devid 1 transid 28699 /dev/sdo
> Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.752023] device fsid 8b4a7c43-1a05-4dcb-bbed-de2a5c933996 devid 1 transid 24346 /dev/sdn
> Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.752068] device fsid ba5fb1ca-c642-49b1-8a41-7f56f8e59fbd devid 1 transid 27274 /dev/sdm
> Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761453] device fsid 7fe8c5cf-bf8c-4276-90f2-c3f57f5275fb devid 1 transid 28724 /dev/sdi
> Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761518] device fsid 93fa3631-1202-4d42-8908-e5ef4d3e600d devid 1 transid 25201 /dev/sdh
> Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761579] device fsid b9a1b5e4-3e5e-4381-a29a-33470f4b870f devid 1 transid 23375 /dev/sdg
> Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761635] device fsid 280ea990-23f8-4c43-9e56-140c82340fdc devid 1 transid 25559 /dev/sdf
> Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761693] device fsid 2f724cde-6de5-4262-b195-1ba3eea2256e devid 1 transid 176 /dev/sde
> Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761732] device fsid a66f890f-8b08-4393-aab0-f222637ca5a4 devid 1 transid 7 /dev/sdd
> Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761769] device fsid 6c181a94-697c-4e0c-af0d-05eb04d3626c devid 1 transid 7 /dev/sdc
> Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.775931] device fsid 6c181a94-697c-4e0c-af0d-05eb04d3626c devid 1 transid 7 /dev/sdc
> Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.779716] btrfs bad fsid on block 20971520
> Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.791594] btrfs bad fsid on block 20971520
> Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.803608] btrfs bad fsid on block 20971520
> Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.815541] btrfs bad fsid on block 20971520
> Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.815878] btrfs bad fsid on block 20971520
> Jul 4 15:02:32 dsanb1-coy kernel: [ 2306.823554] btrfs bad fsid on block 20971520
> Jul 4 15:02:32 dsanb1-coy kernel: [ 2306.823797] btrfs bad fsid on block 20971520
> Jul 4 15:02:32 dsanb1-coy kernel: [ 2306.823887] btrfs: failed to read chunk root on sdc
> Jul 4 15:02:32 dsanb1-coy kernel: [ 2306.825622] btrfs: open_ctree failed
Long shot, but is the kernel on that machine recent?
> Also fails if not forcing to use btrfs, eg:
>
> root@dsanb1-coy:~# mkcephfs -c /etc/ceph/ceph.conf --allhosts -k /etc/ceph/keyring --crushmapsrc crushfile.txt -v
> temp dir is /tmp/mkcephfs.ZOh6tBPAH0
> preparing monmap in /tmp/mkcephfs.ZOh6tBPAH0/monmap
> /usr/bin/monmaptool --create --clobber --add alpha 10.32.0.10:6789 --add bravo 10.32.0.25:6789 --add charlie 10.32.0.11:6789 --print /tmp/mkcephfs.ZOh6tBPAH0/monmap
> /usr/bin/monmaptool: monmap file /tmp/mkcephfs.ZOh6tBPAH0/monmap
> /usr/bin/monmaptool: generated fsid adb8d65c-a823-4dc2-9415-22b0d7252699
> epoch 0
> fsid adb8d65c-a823-4dc2-9415-22b0d7252699
> last_changed 2012-07-04 15:04:17.423368
> created 2012-07-04 15:04:17.423368
> 0: 10.32.0.10:6789/0 mon.alpha
> 1: 10.32.0.11:6789/0 mon.charlie
> 2: 10.32.0.25:6789/0 mon.bravo
> /usr/bin/monmaptool: writing epoch 0 to /tmp/mkcephfs.ZOh6tBPAH0/monmap (3 monitors)
> /usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.0 "user"
> === osd.0 ===
> --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.ZOh6tBPAH0 --init-daemon osd.0
> 2012-07-04 15:04:17.789064 7fc7fadca780 -1 filestore(/srv/osd.0) limited size xattrs -- enable filestore_xattr_use_omap
> 2012-07-04 15:04:17.789120 7fc7fadca780 -1 OSD::mkfs: couldn't mount FileStore: error -95
> 2012-07-04 15:04:17.789161 7fc7fadca780 -1 ** ERROR: error creating empty object store in /srv/osd.0: (95) Operation not supported
> failed: '/sbin/mkcephfs -d /tmp/mkcephfs.ZOh6tBPAH0 --init-daemon osd.0'
>
>
> Confirming all this was working previously, and the crushmap, config
> file, etc are all proven to be OK (get same failure when not specifying
> a custom crushmap also). Also note that whilst the above is failing on
> osd.0 creation, I have swapped disk references and still get the same
> failure on different HDD's when they are hooked in as osd.0
The only thing that changed from v0.47 is the below. Can you try
replacing 'btrfs device scan || btrfsctl -a' with 'btrfs device scan ;
btrfsctl -a'? Maybe the btrfs tool isn't being pendantic about return
codes...
sage
commit a414fd51c7c5ae5dbe9e3af7db6f17741a58c1a7
Author: Sage Weil <sage.weil@dreamhost.com>
Date: Sat Feb 11 13:43:23 2012 -0800
init-ceph, mkcephfs: try 'btrfs device scan' before 'btrfsctl -a'
Fixes: #2023
Reported-by: Wido den Hollander <wido@widodh.nl>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
diff --git a/src/mkcephfs.in b/src/mkcephfs.in
index 83fb932..17b6014 100644
--- a/src/mkcephfs.in
+++ b/src/mkcephfs.in
@@ -332,7 +332,7 @@ if [ -n "$prepareosdfs" ]; then
modprobe btrfs || true
mkfs.btrfs $btrfs_devs
- btrfsctl -a
+ btrfs device scan || btrfsctl -a
mount -t btrfs $btrfs_opt $first_dev $btrfs_path
chown $osd_user $btrfs_path
chmod +w $btrfs_path
^ permalink raw reply related [flat|nested] 9+ messages in thread
* RE: mkcephfs failing on v0.48 "argonaut"
2012-07-05 22:17 ` Sage Weil
@ 2012-07-06 2:39 ` Paul Pettigrew
2012-07-06 4:08 ` Sage Weil
0 siblings, 1 reply; 9+ messages in thread
From: Paul Pettigrew @ 2012-07-06 2:39 UTC (permalink / raw)
To: Sage Weil, ceph-devel
Hi Sage - thanks so much for the quick response :-)
Firstly, and it is a bit hard to see, but the command output below is run with the "-v" option. To help isolate what command line in the script is failing, I have added in some simple echo output, and the script now looks like:
### prepare-osdfs ###
if [ -n "$prepareosdfs" ]; then
<<SNIP>>
modprobe btrfs || true
echo "RUNNING: mkfs.btrfs $btrfs_devs"
mkfs.btrfs $btrfs_devs
btrfs device scan || btrfsctl -a
echo "RUNNING: mount -t btrfs $btrfs_opt $first_dev $btrfs_path"
mount -t btrfs $btrfs_opt $first_dev $btrfs_path
echo "DID I GET HERE - OR CRASH OUT WITH mount ABOVE?"
chown $osd_user $btrfs_path
chmod +w $btrfs_path
exit 0
fi
Per the modified script the above, here is the output displayed when running the script:
root@dsanb1-coy:/srv# /sbin/mkcephfs -c /etc/ceph/ceph.conf --allhosts --mkbtrfs -k /etc/ceph/keyring --crushmapsrc crushfile.txt -v
temp dir is /tmp/mkcephfs.uelzdJ82ej
preparing monmap in /tmp/mkcephfs.uelzdJ82ej/monmap
/usr/bin/monmaptool --create --clobber --add alpha 10.32.0.10:6789 --add bravo 10.32.0.25:6789 --add charlie 10.32.0.11:6789 --print /tmp/mkcephfs.uelzdJ82ej/monmap
/usr/bin/monmaptool: monmap file /tmp/mkcephfs.uelzdJ82ej/monmap
/usr/bin/monmaptool: generated fsid b254abdd-e036-4186-b6d5-e32b14e53b45
epoch 0
fsid b254abdd-e036-4186-b6d5-e32b14e53b45
last_changed 2012-07-06 12:31:38.416848
created 2012-07-06 12:31:38.416848
0: 10.32.0.10:6789/0 mon.alpha
1: 10.32.0.11:6789/0 mon.charlie
2: 10.32.0.25:6789/0 mon.bravo
/usr/bin/monmaptool: writing epoch 0 to /tmp/mkcephfs.uelzdJ82ej/monmap (3 monitors)
/usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.0 "user"
=== osd.0 ===
--- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.uelzdJ82ej --prepare-osdfs osd.0
umount: /srv/osd.0: not mounted
umount: /dev/sdc: not mounted
RUNNING: mkfs.btrfs /dev/sdc
WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using
fs created label (null) on /dev/sdc
nodesize 4096 leafsize 4096 sectorsize 4096 size 1.82TB
Btrfs Btrfs v0.19
Scanning for Btrfs filesystems
RUNNING: mount -t btrfs -o noatime /dev/sdc /srv/osd.0
mount: wrong fs type, bad option, bad superblock on /dev/sdc,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
failed: '/sbin/mkcephfs -d /tmp/mkcephfs.uelzdJ82ej --prepare-osdfs osd.0'
Which clearly isolates the issue to the "mount" command line.
The trouble is, I can run this precise line on the command line directly without error:
root@dsanb1-coy:/srv# mount -t btrfs -o noatime /dev/sdc /srv/osd.0
root@dsanb1-coy:/srv# mount | grep btrfs
/dev/sdc on /srv/osd.0 type btrfs (rw,noatime)
Therefore, what could possibly be preventing the mkcephfs running a simple mount command on the first OSD disk it gets to, that otherwise works fine from the command line?
Many thanks Sage
Paul
PS: changing the " btrfs device scan || btrfsctl -a" line as proposed had no effect, and neither did putting in a "sleep 10" immediately before the mount line.
PPS: zerofilling the /dev/sdc and then re-creating a partition and mounting manually, then writing data to it is all fine. Same errors if we substitute any of the other HDD's in the server as 1st/osd.0. Ie, cannot see any issues with the hardware.
-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
Sent: Friday, 6 July 2012 8:18 AM
To: Paul Pettigrew
Cc: ceph-devel@vger.kernel.org
Subject: Re: mkcephfs failing on v0.48 "argonaut"
Hi Paul,
On Wed, 4 Jul 2012, Paul Pettigrew wrote:
> Firstly, well done guys on achieving this version milestone. I
> successfully upgraded to the 0.48 format uneventfully on a live (test)
> system.
>
> The same system was then going through "rebuild" testing, to confirm
> that also worked fine.
>
>
> Unfortunately, the mkcephfs command is failing:
>
> root@dsanb1-coy:~# mkcephfs -c /etc/ceph/ceph.conf --allhosts
> --mkbtrfs -k /etc/ceph/keyring --crushmapsrc crushfile.txt -v temp dir
> is /tmp/mkcephfs.GaRCZ9i06a preparing monmap in
> /tmp/mkcephfs.GaRCZ9i06a/monmap /usr/bin/monmaptool --create --clobber
> --add alpha 10.32.0.10:6789 --add bravo 10.32.0.25:6789 --add charlie
> 10.32.0.11:6789 --print /tmp/mkcephfs.GaRCZ9i06a/monmap
> /usr/bin/monmaptool: monmap file /tmp/mkcephfs.GaRCZ9i06a/monmap
> /usr/bin/monmaptool: generated fsid
> c7202495-468c-4678-b678-115c3ee33402
> epoch 0
> fsid c7202495-468c-4678-b678-115c3ee33402
> last_changed 2012-07-04 15:02:31.732275 created 2012-07-04
> 15:02:31.732275
> 0: 10.32.0.10:6789/0 mon.alpha
> 1: 10.32.0.11:6789/0 mon.charlie
> 2: 10.32.0.25:6789/0 mon.bravo
> /usr/bin/monmaptool: writing epoch 0 to
> /tmp/mkcephfs.GaRCZ9i06a/monmap (3 monitors) /usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.0 "user"
> === osd.0 ===
> --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.GaRCZ9i06a
> --prepare-osdfs osd.0
> umount: /srv/osd.0: not mounted
> umount: /dev/disk/by-wwn/wwn-0x50014ee601246234: not mounted
>
> WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL WARNING! - see
> http://btrfs.wiki.kernel.org before using
>
> fs created label (null) on /dev/disk/by-wwn/wwn-0x50014ee601246234
> nodesize 4096 leafsize 4096 sectorsize 4096 size 1.82TB Btrfs
> Btrfs v0.19 Scanning for Btrfs filesystems
> mount: wrong fs type, bad option, bad superblock on /dev/sdc,
> missing codepage or helper program, or other error
> In some cases useful info is found in syslog - try
> dmesg | tail or so
>
> failed: '/sbin/mkcephfs -d /tmp/mkcephfs.GaRCZ9i06a --prepare-osdfs osd.0'
Hmm. Can you try running with -v? That will tell us exactly which command it is running, and hopefully we can work backwards from there.
> dmesg/syslog is spitting out at the time of this failure:
>
> Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.751945] device fsid
> 7de0d192-b710-4629-a201-849df1d9db17 devid 1 transid 27109 /dev/sdp
> Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.751987] device fsid
> 08fc3479-2fa2-4388-8b61-83e2a742a13e devid 1 transid 28699 /dev/sdo
> Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.752023] device fsid
> 8b4a7c43-1a05-4dcb-bbed-de2a5c933996 devid 1 transid 24346 /dev/sdn
> Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.752068] device fsid
> ba5fb1ca-c642-49b1-8a41-7f56f8e59fbd devid 1 transid 27274 /dev/sdm
> Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761453] device fsid
> 7fe8c5cf-bf8c-4276-90f2-c3f57f5275fb devid 1 transid 28724 /dev/sdi
> Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761518] device fsid
> 93fa3631-1202-4d42-8908-e5ef4d3e600d devid 1 transid 25201 /dev/sdh
> Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761579] device fsid
> b9a1b5e4-3e5e-4381-a29a-33470f4b870f devid 1 transid 23375 /dev/sdg
> Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761635] device fsid
> 280ea990-23f8-4c43-9e56-140c82340fdc devid 1 transid 25559 /dev/sdf
> Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761693] device fsid
> 2f724cde-6de5-4262-b195-1ba3eea2256e devid 1 transid 176 /dev/sde Jul
> 4 15:02:31 dsanb1-coy kernel: [ 2306.761732] device fsid
> a66f890f-8b08-4393-aab0-f222637ca5a4 devid 1 transid 7 /dev/sdd Jul 4
> 15:02:31 dsanb1-coy kernel: [ 2306.761769] device fsid
> 6c181a94-697c-4e0c-af0d-05eb04d3626c devid 1 transid 7 /dev/sdc Jul 4
> 15:02:31 dsanb1-coy kernel: [ 2306.775931] device fsid
> 6c181a94-697c-4e0c-af0d-05eb04d3626c devid 1 transid 7 /dev/sdc Jul 4
> 15:02:31 dsanb1-coy kernel: [ 2306.779716] btrfs bad fsid on block
> 20971520 Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.791594] btrfs bad
> fsid on block 20971520 Jul 4 15:02:31 dsanb1-coy kernel: [
> 2306.803608] btrfs bad fsid on block 20971520 Jul 4 15:02:31
> dsanb1-coy kernel: [ 2306.815541] btrfs bad fsid on block 20971520 Jul
> 4 15:02:31 dsanb1-coy kernel: [ 2306.815878] btrfs bad fsid on block
> 20971520 Jul 4 15:02:32 dsanb1-coy kernel: [ 2306.823554] btrfs bad
> fsid on block 20971520 Jul 4 15:02:32 dsanb1-coy kernel: [
> 2306.823797] btrfs bad fsid on block 20971520 Jul 4 15:02:32
> dsanb1-coy kernel: [ 2306.823887] btrfs: failed to read chunk root on
> sdc Jul 4 15:02:32 dsanb1-coy kernel: [ 2306.825622] btrfs:
> open_ctree failed
Long shot, but is the kernel on that machine recent?
> Also fails if not forcing to use btrfs, eg:
>
> root@dsanb1-coy:~# mkcephfs -c /etc/ceph/ceph.conf --allhosts -k
> /etc/ceph/keyring --crushmapsrc crushfile.txt -v temp dir is
> /tmp/mkcephfs.ZOh6tBPAH0 preparing monmap in
> /tmp/mkcephfs.ZOh6tBPAH0/monmap /usr/bin/monmaptool --create --clobber
> --add alpha 10.32.0.10:6789 --add bravo 10.32.0.25:6789 --add charlie
> 10.32.0.11:6789 --print /tmp/mkcephfs.ZOh6tBPAH0/monmap
> /usr/bin/monmaptool: monmap file /tmp/mkcephfs.ZOh6tBPAH0/monmap
> /usr/bin/monmaptool: generated fsid
> adb8d65c-a823-4dc2-9415-22b0d7252699
> epoch 0
> fsid adb8d65c-a823-4dc2-9415-22b0d7252699
> last_changed 2012-07-04 15:04:17.423368 created 2012-07-04
> 15:04:17.423368
> 0: 10.32.0.10:6789/0 mon.alpha
> 1: 10.32.0.11:6789/0 mon.charlie
> 2: 10.32.0.25:6789/0 mon.bravo
> /usr/bin/monmaptool: writing epoch 0 to
> /tmp/mkcephfs.ZOh6tBPAH0/monmap (3 monitors) /usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.0 "user"
> === osd.0 ===
> --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.ZOh6tBPAH0
> --init-daemon osd.0
> 2012-07-04 15:04:17.789064 7fc7fadca780 -1 filestore(/srv/osd.0)
> limited size xattrs -- enable filestore_xattr_use_omap
> 2012-07-04 15:04:17.789120 7fc7fadca780 -1 OSD::mkfs: couldn't mount
> FileStore: error -95
> 2012-07-04 15:04:17.789161 7fc7fadca780 -1 ** ERROR: error creating
> empty object store in /srv/osd.0: (95) Operation not supported
> failed: '/sbin/mkcephfs -d /tmp/mkcephfs.ZOh6tBPAH0 --init-daemon osd.0'
>
>
> Confirming all this was working previously, and the crushmap, config
> file, etc are all proven to be OK (get same failure when not
> specifying a custom crushmap also). Also note that whilst the above is
> failing on
> osd.0 creation, I have swapped disk references and still get the same
> failure on different HDD's when they are hooked in as osd.0
The only thing that changed from v0.47 is the below. Can you try replacing 'btrfs device scan || btrfsctl -a' with 'btrfs device scan ; btrfsctl -a'? Maybe the btrfs tool isn't being pendantic about return codes...
sage
commit a414fd51c7c5ae5dbe9e3af7db6f17741a58c1a7
Author: Sage Weil <sage.weil@dreamhost.com>
Date: Sat Feb 11 13:43:23 2012 -0800
init-ceph, mkcephfs: try 'btrfs device scan' before 'btrfsctl -a'
Fixes: #2023
Reported-by: Wido den Hollander <wido@widodh.nl>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
diff --git a/src/mkcephfs.in b/src/mkcephfs.in index 83fb932..17b6014 100644
--- a/src/mkcephfs.in
+++ b/src/mkcephfs.in
@@ -332,7 +332,7 @@ if [ -n "$prepareosdfs" ]; then
modprobe btrfs || true
mkfs.btrfs $btrfs_devs
- btrfsctl -a
+ btrfs device scan || btrfsctl -a
mount -t btrfs $btrfs_opt $first_dev $btrfs_path
chown $osd_user $btrfs_path
chmod +w $btrfs_path
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: mkcephfs failing on v0.48 "argonaut"
2012-07-06 2:39 ` Paul Pettigrew
@ 2012-07-06 4:08 ` Sage Weil
2012-07-07 3:33 ` Paul Pettigrew
0 siblings, 1 reply; 9+ messages in thread
From: Sage Weil @ 2012-07-06 4:08 UTC (permalink / raw)
To: Paul Pettigrew; +Cc: ceph-devel
On Fri, 6 Jul 2012, Paul Pettigrew wrote:
> Hi Sage - thanks so much for the quick response :-)
>
> Firstly, and it is a bit hard to see, but the command output below is run with the "-v" option. To help isolate what command line in the script is failing, I have added in some simple echo output, and the script now looks like:
>
>
> ### prepare-osdfs ###
>
> if [ -n "$prepareosdfs" ]; then
> <<SNIP>>
> modprobe btrfs || true
> echo "RUNNING: mkfs.btrfs $btrfs_devs"
> mkfs.btrfs $btrfs_devs
> btrfs device scan || btrfsctl -a
> echo "RUNNING: mount -t btrfs $btrfs_opt $first_dev $btrfs_path"
> mount -t btrfs $btrfs_opt $first_dev $btrfs_path
> echo "DID I GET HERE - OR CRASH OUT WITH mount ABOVE?"
> chown $osd_user $btrfs_path
> chmod +w $btrfs_path
>
> exit 0
> fi
>
> Per the modified script the above, here is the output displayed when running the script:
>
> root@dsanb1-coy:/srv# /sbin/mkcephfs -c /etc/ceph/ceph.conf --allhosts --mkbtrfs -k /etc/ceph/keyring --crushmapsrc crushfile.txt -v
> temp dir is /tmp/mkcephfs.uelzdJ82ej
> preparing monmap in /tmp/mkcephfs.uelzdJ82ej/monmap
> /usr/bin/monmaptool --create --clobber --add alpha 10.32.0.10:6789 --add bravo 10.32.0.25:6789 --add charlie 10.32.0.11:6789 --print /tmp/mkcephfs.uelzdJ82ej/monmap
> /usr/bin/monmaptool: monmap file /tmp/mkcephfs.uelzdJ82ej/monmap
> /usr/bin/monmaptool: generated fsid b254abdd-e036-4186-b6d5-e32b14e53b45
> epoch 0
> fsid b254abdd-e036-4186-b6d5-e32b14e53b45
> last_changed 2012-07-06 12:31:38.416848
> created 2012-07-06 12:31:38.416848
> 0: 10.32.0.10:6789/0 mon.alpha
> 1: 10.32.0.11:6789/0 mon.charlie
> 2: 10.32.0.25:6789/0 mon.bravo
> /usr/bin/monmaptool: writing epoch 0 to /tmp/mkcephfs.uelzdJ82ej/monmap (3 monitors)
> /usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.0 "user"
> === osd.0 ===
> --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.uelzdJ82ej --prepare-osdfs osd.0
> umount: /srv/osd.0: not mounted
> umount: /dev/sdc: not mounted
> RUNNING: mkfs.btrfs /dev/sdc
>
> WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL
> WARNING! - see http://btrfs.wiki.kernel.org before using
>
> fs created label (null) on /dev/sdc
> nodesize 4096 leafsize 4096 sectorsize 4096 size 1.82TB
> Btrfs Btrfs v0.19
> Scanning for Btrfs filesystems
> RUNNING: mount -t btrfs -o noatime /dev/sdc /srv/osd.0
> mount: wrong fs type, bad option, bad superblock on /dev/sdc,
> missing codepage or helper program, or other error
> In some cases useful info is found in syslog - try
> dmesg | tail or so
>
> failed: '/sbin/mkcephfs -d /tmp/mkcephfs.uelzdJ82ej --prepare-osdfs osd.0'
>
>
> Which clearly isolates the issue to the "mount" command line.
>
> The trouble is, I can run this precise line on the command line directly without error:
>
> root@dsanb1-coy:/srv# mount -t btrfs -o noatime /dev/sdc /srv/osd.0
> root@dsanb1-coy:/srv# mount | grep btrfs
> /dev/sdc on /srv/osd.0 type btrfs (rw,noatime)
What if you run the exact sequence of commands that mkcephfs is doing?
(mkfs.btrfs, btrfs ..., mount ...). If that doesn't work, put `which
mkfs.btfs` etc in the script to make sure you're running the exact version
the script is...
sage
>
>
> Therefore, what could possibly be preventing the mkcephfs running a simple mount command on the first OSD disk it gets to, that otherwise works fine from the command line?
>
> Many thanks Sage
>
> Paul
>
> PS: changing the " btrfs device scan || btrfsctl -a" line as proposed had no effect, and neither did putting in a "sleep 10" immediately before the mount line.
> PPS: zerofilling the /dev/sdc and then re-creating a partition and mounting manually, then writing data to it is all fine. Same errors if we substitute any of the other HDD's in the server as 1st/osd.0. Ie, cannot see any issues with the hardware.
>
>
>
>
>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
> Sent: Friday, 6 July 2012 8:18 AM
> To: Paul Pettigrew
> Cc: ceph-devel@vger.kernel.org
> Subject: Re: mkcephfs failing on v0.48 "argonaut"
>
> Hi Paul,
>
> On Wed, 4 Jul 2012, Paul Pettigrew wrote:
> > Firstly, well done guys on achieving this version milestone. I
> > successfully upgraded to the 0.48 format uneventfully on a live (test)
> > system.
> >
> > The same system was then going through "rebuild" testing, to confirm
> > that also worked fine.
> >
> >
> > Unfortunately, the mkcephfs command is failing:
> >
> > root@dsanb1-coy:~# mkcephfs -c /etc/ceph/ceph.conf --allhosts
> > --mkbtrfs -k /etc/ceph/keyring --crushmapsrc crushfile.txt -v temp dir
> > is /tmp/mkcephfs.GaRCZ9i06a preparing monmap in
> > /tmp/mkcephfs.GaRCZ9i06a/monmap /usr/bin/monmaptool --create --clobber
> > --add alpha 10.32.0.10:6789 --add bravo 10.32.0.25:6789 --add charlie
> > 10.32.0.11:6789 --print /tmp/mkcephfs.GaRCZ9i06a/monmap
> > /usr/bin/monmaptool: monmap file /tmp/mkcephfs.GaRCZ9i06a/monmap
> > /usr/bin/monmaptool: generated fsid
> > c7202495-468c-4678-b678-115c3ee33402
> > epoch 0
> > fsid c7202495-468c-4678-b678-115c3ee33402
> > last_changed 2012-07-04 15:02:31.732275 created 2012-07-04
> > 15:02:31.732275
> > 0: 10.32.0.10:6789/0 mon.alpha
> > 1: 10.32.0.11:6789/0 mon.charlie
> > 2: 10.32.0.25:6789/0 mon.bravo
> > /usr/bin/monmaptool: writing epoch 0 to
> > /tmp/mkcephfs.GaRCZ9i06a/monmap (3 monitors) /usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.0 "user"
> > === osd.0 ===
> > --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.GaRCZ9i06a
> > --prepare-osdfs osd.0
> > umount: /srv/osd.0: not mounted
> > umount: /dev/disk/by-wwn/wwn-0x50014ee601246234: not mounted
> >
> > WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL WARNING! - see
> > http://btrfs.wiki.kernel.org before using
> >
> > fs created label (null) on /dev/disk/by-wwn/wwn-0x50014ee601246234
> > nodesize 4096 leafsize 4096 sectorsize 4096 size 1.82TB Btrfs
> > Btrfs v0.19 Scanning for Btrfs filesystems
> > mount: wrong fs type, bad option, bad superblock on /dev/sdc,
> > missing codepage or helper program, or other error
> > In some cases useful info is found in syslog - try
> > dmesg | tail or so
> >
> > failed: '/sbin/mkcephfs -d /tmp/mkcephfs.GaRCZ9i06a --prepare-osdfs osd.0'
>
> Hmm. Can you try running with -v? That will tell us exactly which command it is running, and hopefully we can work backwards from there.
>
> > dmesg/syslog is spitting out at the time of this failure:
> >
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.751945] device fsid
> > 7de0d192-b710-4629-a201-849df1d9db17 devid 1 transid 27109 /dev/sdp
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.751987] device fsid
> > 08fc3479-2fa2-4388-8b61-83e2a742a13e devid 1 transid 28699 /dev/sdo
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.752023] device fsid
> > 8b4a7c43-1a05-4dcb-bbed-de2a5c933996 devid 1 transid 24346 /dev/sdn
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.752068] device fsid
> > ba5fb1ca-c642-49b1-8a41-7f56f8e59fbd devid 1 transid 27274 /dev/sdm
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761453] device fsid
> > 7fe8c5cf-bf8c-4276-90f2-c3f57f5275fb devid 1 transid 28724 /dev/sdi
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761518] device fsid
> > 93fa3631-1202-4d42-8908-e5ef4d3e600d devid 1 transid 25201 /dev/sdh
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761579] device fsid
> > b9a1b5e4-3e5e-4381-a29a-33470f4b870f devid 1 transid 23375 /dev/sdg
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761635] device fsid
> > 280ea990-23f8-4c43-9e56-140c82340fdc devid 1 transid 25559 /dev/sdf
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761693] device fsid
> > 2f724cde-6de5-4262-b195-1ba3eea2256e devid 1 transid 176 /dev/sde Jul
> > 4 15:02:31 dsanb1-coy kernel: [ 2306.761732] device fsid
> > a66f890f-8b08-4393-aab0-f222637ca5a4 devid 1 transid 7 /dev/sdd Jul 4
> > 15:02:31 dsanb1-coy kernel: [ 2306.761769] device fsid
> > 6c181a94-697c-4e0c-af0d-05eb04d3626c devid 1 transid 7 /dev/sdc Jul 4
> > 15:02:31 dsanb1-coy kernel: [ 2306.775931] device fsid
> > 6c181a94-697c-4e0c-af0d-05eb04d3626c devid 1 transid 7 /dev/sdc Jul 4
> > 15:02:31 dsanb1-coy kernel: [ 2306.779716] btrfs bad fsid on block
> > 20971520 Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.791594] btrfs bad
> > fsid on block 20971520 Jul 4 15:02:31 dsanb1-coy kernel: [
> > 2306.803608] btrfs bad fsid on block 20971520 Jul 4 15:02:31
> > dsanb1-coy kernel: [ 2306.815541] btrfs bad fsid on block 20971520 Jul
> > 4 15:02:31 dsanb1-coy kernel: [ 2306.815878] btrfs bad fsid on block
> > 20971520 Jul 4 15:02:32 dsanb1-coy kernel: [ 2306.823554] btrfs bad
> > fsid on block 20971520 Jul 4 15:02:32 dsanb1-coy kernel: [
> > 2306.823797] btrfs bad fsid on block 20971520 Jul 4 15:02:32
> > dsanb1-coy kernel: [ 2306.823887] btrfs: failed to read chunk root on
> > sdc Jul 4 15:02:32 dsanb1-coy kernel: [ 2306.825622] btrfs:
> > open_ctree failed
>
> Long shot, but is the kernel on that machine recent?
>
> > Also fails if not forcing to use btrfs, eg:
> >
> > root@dsanb1-coy:~# mkcephfs -c /etc/ceph/ceph.conf --allhosts -k
> > /etc/ceph/keyring --crushmapsrc crushfile.txt -v temp dir is
> > /tmp/mkcephfs.ZOh6tBPAH0 preparing monmap in
> > /tmp/mkcephfs.ZOh6tBPAH0/monmap /usr/bin/monmaptool --create --clobber
> > --add alpha 10.32.0.10:6789 --add bravo 10.32.0.25:6789 --add charlie
> > 10.32.0.11:6789 --print /tmp/mkcephfs.ZOh6tBPAH0/monmap
> > /usr/bin/monmaptool: monmap file /tmp/mkcephfs.ZOh6tBPAH0/monmap
> > /usr/bin/monmaptool: generated fsid
> > adb8d65c-a823-4dc2-9415-22b0d7252699
> > epoch 0
> > fsid adb8d65c-a823-4dc2-9415-22b0d7252699
> > last_changed 2012-07-04 15:04:17.423368 created 2012-07-04
> > 15:04:17.423368
> > 0: 10.32.0.10:6789/0 mon.alpha
> > 1: 10.32.0.11:6789/0 mon.charlie
> > 2: 10.32.0.25:6789/0 mon.bravo
> > /usr/bin/monmaptool: writing epoch 0 to
> > /tmp/mkcephfs.ZOh6tBPAH0/monmap (3 monitors) /usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.0 "user"
> > === osd.0 ===
> > --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.ZOh6tBPAH0
> > --init-daemon osd.0
> > 2012-07-04 15:04:17.789064 7fc7fadca780 -1 filestore(/srv/osd.0)
> > limited size xattrs -- enable filestore_xattr_use_omap
> > 2012-07-04 15:04:17.789120 7fc7fadca780 -1 OSD::mkfs: couldn't mount
> > FileStore: error -95
> > 2012-07-04 15:04:17.789161 7fc7fadca780 -1 ** ERROR: error creating
> > empty object store in /srv/osd.0: (95) Operation not supported
> > failed: '/sbin/mkcephfs -d /tmp/mkcephfs.ZOh6tBPAH0 --init-daemon osd.0'
> >
> >
> > Confirming all this was working previously, and the crushmap, config
> > file, etc are all proven to be OK (get same failure when not
> > specifying a custom crushmap also). Also note that whilst the above is
> > failing on
> > osd.0 creation, I have swapped disk references and still get the same
> > failure on different HDD's when they are hooked in as osd.0
>
> The only thing that changed from v0.47 is the below. Can you try replacing 'btrfs device scan || btrfsctl -a' with 'btrfs device scan ; btrfsctl -a'? Maybe the btrfs tool isn't being pendantic about return codes...
>
> sage
>
>
> commit a414fd51c7c5ae5dbe9e3af7db6f17741a58c1a7
> Author: Sage Weil <sage.weil@dreamhost.com>
> Date: Sat Feb 11 13:43:23 2012 -0800
>
> init-ceph, mkcephfs: try 'btrfs device scan' before 'btrfsctl -a'
>
> Fixes: #2023
> Reported-by: Wido den Hollander <wido@widodh.nl>
> Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
>
> diff --git a/src/mkcephfs.in b/src/mkcephfs.in index 83fb932..17b6014 100644
> --- a/src/mkcephfs.in
> +++ b/src/mkcephfs.in
> @@ -332,7 +332,7 @@ if [ -n "$prepareosdfs" ]; then
>
> modprobe btrfs || true
> mkfs.btrfs $btrfs_devs
> - btrfsctl -a
> + btrfs device scan || btrfsctl -a
> mount -t btrfs $btrfs_opt $first_dev $btrfs_path
> chown $osd_user $btrfs_path
> chmod +w $btrfs_path
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: mkcephfs failing on v0.48 "argonaut"
2012-07-06 4:08 ` Sage Weil
@ 2012-07-07 3:33 ` Paul Pettigrew
2012-07-07 3:45 ` Paul Pettigrew
2012-07-07 4:20 ` Sage Weil
0 siblings, 2 replies; 9+ messages in thread
From: Paul Pettigrew @ 2012-07-07 3:33 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-devel
Hi again Sage
This is very perplexing. Confirming this system is a stock Ubuntu 12.04 x64, with no custom kernel or anything else, fully apt-get dist-upgrade'd up to date.
root@dsanb1-coy:~# uname -r
3.2.0-26-generic
I have added in the suggestions you made to the script, we now have:
modprobe btrfs || true
which mkfs.btrfs
echo "RUNNING: mkfs.btrfs $btrfs_devs"
mkfs.btrfs $btrfs_devs
btrfs device scan || btrfsctl -a
echo "RUNNING: mount -t btrfs $btrfs_opt $first_dev $btrfs_path"
which mount
mount -t btrfs $btrfs_opt $first_dev $btrfs_path
echo "DID I GET HERE - OR CRASH OUT WITH mount ABOVE?"
chown $osd_user $btrfs_path
See below that the same command within the mkcephfs that is failing, is working fine on a standard command line:
=== osd.0 ===
--- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.xgk025tjkQ --prepare-osdfs osd.0
umount: /srv/osd.0: not mounted
umount: /dev/sdc: not mounted
/sbin/mkfs.btrfs
RUNNING: mkfs.btrfs /dev/sdc
WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using
fs created label (null) on /dev/sdc
nodesize 4096 leafsize 4096 sectorsize 4096 size 1.82TB
Btrfs Btrfs v0.19
Scanning for Btrfs filesystems
RUNNING: mount -t btrfs -o noatime /dev/sdc /srv/osd.0
/bin/mount
mount: wrong fs type, bad option, bad superblock on /dev/sdc,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
failed: '/sbin/mkcephfs -d /tmp/mkcephfs.xgk025tjkQ --prepare-osdfs osd.0'
root@dsanb1-coy:~# /bin/mount -t btrfs -o noatime /dev/sdc /srv/osd.0
root@dsanb1-coy:~# mount | grep btrfs
/dev/sdc on /srv/osd.0 type btrfs (rw,noatime)
Remember, this is not isolated to btrfs, as per my original post it fails when not specifying to use btrfs.
I can only conclude that /bin/sh &/or /bin/bash and the way they interact with the mkcephfs script, which does call itself etc, is somehow now become fuddled up? Must be something wiggy, when the script output confirms it is calling the same command ( /bin/mount ) but somehow finds a way for that to not work and therefore cause the mkcephfs script terminate.
Many thanks - will be a relief to sort this out, as all our Ceph project works are on hold til we can sort this one out.
Cheers
Paul
-----Original Message-----
From: Sage Weil [mailto:sage@inktank.com]
Sent: Friday, 6 July 2012 2:09 PM
To: Paul Pettigrew
Cc: ceph-devel@vger.kernel.org
Subject: RE: mkcephfs failing on v0.48 "argonaut"
On Fri, 6 Jul 2012, Paul Pettigrew wrote:
> Hi Sage - thanks so much for the quick response :-)
>
> Firstly, and it is a bit hard to see, but the command output below is run with the "-v" option. To help isolate what command line in the script is failing, I have added in some simple echo output, and the script now looks like:
>
>
> ### prepare-osdfs ###
>
> if [ -n "$prepareosdfs" ]; then
> <<SNIP>>
> modprobe btrfs || true
> echo "RUNNING: mkfs.btrfs $btrfs_devs"
> mkfs.btrfs $btrfs_devs
> btrfs device scan || btrfsctl -a
> echo "RUNNING: mount -t btrfs $btrfs_opt $first_dev $btrfs_path"
> mount -t btrfs $btrfs_opt $first_dev $btrfs_path echo "DID I GET
> HERE - OR CRASH OUT WITH mount ABOVE?"
> chown $osd_user $btrfs_path
> chmod +w $btrfs_path
>
> exit 0
> fi
>
> Per the modified script the above, here is the output displayed when running the script:
>
> root@dsanb1-coy:/srv# /sbin/mkcephfs -c /etc/ceph/ceph.conf --allhosts
> --mkbtrfs -k /etc/ceph/keyring --crushmapsrc crushfile.txt -v temp dir
> is /tmp/mkcephfs.uelzdJ82ej preparing monmap in
> /tmp/mkcephfs.uelzdJ82ej/monmap /usr/bin/monmaptool --create --clobber
> --add alpha 10.32.0.10:6789 --add bravo 10.32.0.25:6789 --add charlie
> 10.32.0.11:6789 --print /tmp/mkcephfs.uelzdJ82ej/monmap
> /usr/bin/monmaptool: monmap file /tmp/mkcephfs.uelzdJ82ej/monmap
> /usr/bin/monmaptool: generated fsid
> b254abdd-e036-4186-b6d5-e32b14e53b45
> epoch 0
> fsid b254abdd-e036-4186-b6d5-e32b14e53b45
> last_changed 2012-07-06 12:31:38.416848 created 2012-07-06
> 12:31:38.416848
> 0: 10.32.0.10:6789/0 mon.alpha
> 1: 10.32.0.11:6789/0 mon.charlie
> 2: 10.32.0.25:6789/0 mon.bravo
> /usr/bin/monmaptool: writing epoch 0 to
> /tmp/mkcephfs.uelzdJ82ej/monmap (3 monitors) /usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.0 "user"
> === osd.0 ===
> --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.uelzdJ82ej
> --prepare-osdfs osd.0
> umount: /srv/osd.0: not mounted
> umount: /dev/sdc: not mounted
> RUNNING: mkfs.btrfs /dev/sdc
>
> WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL WARNING! - see
> http://btrfs.wiki.kernel.org before using
>
> fs created label (null) on /dev/sdc
> nodesize 4096 leafsize 4096 sectorsize 4096 size 1.82TB Btrfs
> Btrfs v0.19 Scanning for Btrfs filesystems
> RUNNING: mount -t btrfs -o noatime /dev/sdc /srv/osd.0
> mount: wrong fs type, bad option, bad superblock on /dev/sdc,
> missing codepage or helper program, or other error
> In some cases useful info is found in syslog - try
> dmesg | tail or so
>
> failed: '/sbin/mkcephfs -d /tmp/mkcephfs.uelzdJ82ej --prepare-osdfs osd.0'
>
>
> Which clearly isolates the issue to the "mount" command line.
>
> The trouble is, I can run this precise line on the command line directly without error:
>
> root@dsanb1-coy:/srv# mount -t btrfs -o noatime /dev/sdc /srv/osd.0
> root@dsanb1-coy:/srv# mount | grep btrfs /dev/sdc on /srv/osd.0 type
> btrfs (rw,noatime)
What if you run the exact sequence of commands that mkcephfs is doing?
(mkfs.btrfs, btrfs ..., mount ...). If that doesn't work, put `which mkfs.btfs` etc in the script to make sure you're running the exact version the script is...
sage
>
>
> Therefore, what could possibly be preventing the mkcephfs running a simple mount command on the first OSD disk it gets to, that otherwise works fine from the command line?
>
> Many thanks Sage
>
> Paul
>
> PS: changing the " btrfs device scan || btrfsctl -a" line as proposed had no effect, and neither did putting in a "sleep 10" immediately before the mount line.
> PPS: zerofilling the /dev/sdc and then re-creating a partition and mounting manually, then writing data to it is all fine. Same errors if we substitute any of the other HDD's in the server as 1st/osd.0. Ie, cannot see any issues with the hardware.
>
>
>
>
>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org
> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
> Sent: Friday, 6 July 2012 8:18 AM
> To: Paul Pettigrew
> Cc: ceph-devel@vger.kernel.org
> Subject: Re: mkcephfs failing on v0.48 "argonaut"
>
> Hi Paul,
>
> On Wed, 4 Jul 2012, Paul Pettigrew wrote:
> > Firstly, well done guys on achieving this version milestone. I
> > successfully upgraded to the 0.48 format uneventfully on a live
> > (test) system.
> >
> > The same system was then going through "rebuild" testing, to confirm
> > that also worked fine.
> >
> >
> > Unfortunately, the mkcephfs command is failing:
> >
> > root@dsanb1-coy:~# mkcephfs -c /etc/ceph/ceph.conf --allhosts
> > --mkbtrfs -k /etc/ceph/keyring --crushmapsrc crushfile.txt -v temp
> > dir is /tmp/mkcephfs.GaRCZ9i06a preparing monmap in
> > /tmp/mkcephfs.GaRCZ9i06a/monmap /usr/bin/monmaptool --create
> > --clobber --add alpha 10.32.0.10:6789 --add bravo 10.32.0.25:6789
> > --add charlie
> > 10.32.0.11:6789 --print /tmp/mkcephfs.GaRCZ9i06a/monmap
> > /usr/bin/monmaptool: monmap file /tmp/mkcephfs.GaRCZ9i06a/monmap
> > /usr/bin/monmaptool: generated fsid
> > c7202495-468c-4678-b678-115c3ee33402
> > epoch 0
> > fsid c7202495-468c-4678-b678-115c3ee33402
> > last_changed 2012-07-04 15:02:31.732275 created 2012-07-04
> > 15:02:31.732275
> > 0: 10.32.0.10:6789/0 mon.alpha
> > 1: 10.32.0.11:6789/0 mon.charlie
> > 2: 10.32.0.25:6789/0 mon.bravo
> > /usr/bin/monmaptool: writing epoch 0 to
> > /tmp/mkcephfs.GaRCZ9i06a/monmap (3 monitors) /usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.0 "user"
> > === osd.0 ===
> > --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.GaRCZ9i06a
> > --prepare-osdfs osd.0
> > umount: /srv/osd.0: not mounted
> > umount: /dev/disk/by-wwn/wwn-0x50014ee601246234: not mounted
> >
> > WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL WARNING! - see
> > http://btrfs.wiki.kernel.org before using
> >
> > fs created label (null) on /dev/disk/by-wwn/wwn-0x50014ee601246234
> > nodesize 4096 leafsize 4096 sectorsize 4096 size 1.82TB
> > Btrfs Btrfs v0.19 Scanning for Btrfs filesystems
> > mount: wrong fs type, bad option, bad superblock on /dev/sdc,
> > missing codepage or helper program, or other error
> > In some cases useful info is found in syslog - try
> > dmesg | tail or so
> >
> > failed: '/sbin/mkcephfs -d /tmp/mkcephfs.GaRCZ9i06a --prepare-osdfs osd.0'
>
> Hmm. Can you try running with -v? That will tell us exactly which command it is running, and hopefully we can work backwards from there.
>
> > dmesg/syslog is spitting out at the time of this failure:
> >
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.751945] device fsid
> > 7de0d192-b710-4629-a201-849df1d9db17 devid 1 transid 27109 /dev/sdp
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.751987] device fsid
> > 08fc3479-2fa2-4388-8b61-83e2a742a13e devid 1 transid 28699 /dev/sdo
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.752023] device fsid
> > 8b4a7c43-1a05-4dcb-bbed-de2a5c933996 devid 1 transid 24346 /dev/sdn
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.752068] device fsid
> > ba5fb1ca-c642-49b1-8a41-7f56f8e59fbd devid 1 transid 27274 /dev/sdm
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761453] device fsid
> > 7fe8c5cf-bf8c-4276-90f2-c3f57f5275fb devid 1 transid 28724 /dev/sdi
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761518] device fsid
> > 93fa3631-1202-4d42-8908-e5ef4d3e600d devid 1 transid 25201 /dev/sdh
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761579] device fsid
> > b9a1b5e4-3e5e-4381-a29a-33470f4b870f devid 1 transid 23375 /dev/sdg
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761635] device fsid
> > 280ea990-23f8-4c43-9e56-140c82340fdc devid 1 transid 25559 /dev/sdf
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761693] device fsid
> > 2f724cde-6de5-4262-b195-1ba3eea2256e devid 1 transid 176 /dev/sde
> > Jul
> > 4 15:02:31 dsanb1-coy kernel: [ 2306.761732] device fsid
> > a66f890f-8b08-4393-aab0-f222637ca5a4 devid 1 transid 7 /dev/sdd Jul
> > 4
> > 15:02:31 dsanb1-coy kernel: [ 2306.761769] device fsid
> > 6c181a94-697c-4e0c-af0d-05eb04d3626c devid 1 transid 7 /dev/sdc Jul
> > 4
> > 15:02:31 dsanb1-coy kernel: [ 2306.775931] device fsid
> > 6c181a94-697c-4e0c-af0d-05eb04d3626c devid 1 transid 7 /dev/sdc Jul
> > 4
> > 15:02:31 dsanb1-coy kernel: [ 2306.779716] btrfs bad fsid on block
> > 20971520 Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.791594] btrfs bad
> > fsid on block 20971520 Jul 4 15:02:31 dsanb1-coy kernel: [
> > 2306.803608] btrfs bad fsid on block 20971520 Jul 4 15:02:31
> > dsanb1-coy kernel: [ 2306.815541] btrfs bad fsid on block 20971520
> > Jul
> > 4 15:02:31 dsanb1-coy kernel: [ 2306.815878] btrfs bad fsid on block
> > 20971520 Jul 4 15:02:32 dsanb1-coy kernel: [ 2306.823554] btrfs bad
> > fsid on block 20971520 Jul 4 15:02:32 dsanb1-coy kernel: [
> > 2306.823797] btrfs bad fsid on block 20971520 Jul 4 15:02:32
> > dsanb1-coy kernel: [ 2306.823887] btrfs: failed to read chunk root
> > on sdc Jul 4 15:02:32 dsanb1-coy kernel: [ 2306.825622] btrfs:
> > open_ctree failed
>
> Long shot, but is the kernel on that machine recent?
>
> > Also fails if not forcing to use btrfs, eg:
> >
> > root@dsanb1-coy:~# mkcephfs -c /etc/ceph/ceph.conf --allhosts -k
> > /etc/ceph/keyring --crushmapsrc crushfile.txt -v temp dir is
> > /tmp/mkcephfs.ZOh6tBPAH0 preparing monmap in
> > /tmp/mkcephfs.ZOh6tBPAH0/monmap /usr/bin/monmaptool --create
> > --clobber --add alpha 10.32.0.10:6789 --add bravo 10.32.0.25:6789
> > --add charlie
> > 10.32.0.11:6789 --print /tmp/mkcephfs.ZOh6tBPAH0/monmap
> > /usr/bin/monmaptool: monmap file /tmp/mkcephfs.ZOh6tBPAH0/monmap
> > /usr/bin/monmaptool: generated fsid
> > adb8d65c-a823-4dc2-9415-22b0d7252699
> > epoch 0
> > fsid adb8d65c-a823-4dc2-9415-22b0d7252699
> > last_changed 2012-07-04 15:04:17.423368 created 2012-07-04
> > 15:04:17.423368
> > 0: 10.32.0.10:6789/0 mon.alpha
> > 1: 10.32.0.11:6789/0 mon.charlie
> > 2: 10.32.0.25:6789/0 mon.bravo
> > /usr/bin/monmaptool: writing epoch 0 to
> > /tmp/mkcephfs.ZOh6tBPAH0/monmap (3 monitors) /usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.0 "user"
> > === osd.0 ===
> > --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.ZOh6tBPAH0
> > --init-daemon osd.0
> > 2012-07-04 15:04:17.789064 7fc7fadca780 -1 filestore(/srv/osd.0)
> > limited size xattrs -- enable filestore_xattr_use_omap
> > 2012-07-04 15:04:17.789120 7fc7fadca780 -1 OSD::mkfs: couldn't mount
> > FileStore: error -95
> > 2012-07-04 15:04:17.789161 7fc7fadca780 -1 ** ERROR: error creating
> > empty object store in /srv/osd.0: (95) Operation not supported
> > failed: '/sbin/mkcephfs -d /tmp/mkcephfs.ZOh6tBPAH0 --init-daemon osd.0'
> >
> >
> > Confirming all this was working previously, and the crushmap, config
> > file, etc are all proven to be OK (get same failure when not
> > specifying a custom crushmap also). Also note that whilst the above
> > is failing on
> > osd.0 creation, I have swapped disk references and still get the
> > same failure on different HDD's when they are hooked in as osd.0
>
> The only thing that changed from v0.47 is the below. Can you try replacing 'btrfs device scan || btrfsctl -a' with 'btrfs device scan ; btrfsctl -a'? Maybe the btrfs tool isn't being pendantic about return codes...
>
> sage
>
>
> commit a414fd51c7c5ae5dbe9e3af7db6f17741a58c1a7
> Author: Sage Weil <sage.weil@dreamhost.com>
> Date: Sat Feb 11 13:43:23 2012 -0800
>
> init-ceph, mkcephfs: try 'btrfs device scan' before 'btrfsctl -a'
>
> Fixes: #2023
> Reported-by: Wido den Hollander <wido@widodh.nl>
> Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
>
> diff --git a/src/mkcephfs.in b/src/mkcephfs.in index 83fb932..17b6014
> 100644
> --- a/src/mkcephfs.in
> +++ b/src/mkcephfs.in
> @@ -332,7 +332,7 @@ if [ -n "$prepareosdfs" ]; then
>
> modprobe btrfs || true
> mkfs.btrfs $btrfs_devs
> - btrfsctl -a
> + btrfs device scan || btrfsctl -a
> mount -t btrfs $btrfs_opt $first_dev $btrfs_path
> chown $osd_user $btrfs_path
> chmod +w $btrfs_path
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo
> info at http://vger.kernel.org/majordomo-info.html
>
>
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: mkcephfs failing on v0.48 "argonaut"
2012-07-07 3:33 ` Paul Pettigrew
@ 2012-07-07 3:45 ` Paul Pettigrew
2012-07-07 4:20 ` Sage Weil
1 sibling, 0 replies; 9+ messages in thread
From: Paul Pettigrew @ 2012-07-07 3:45 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-devel
UPDATED code now within the below (paste snafu, sorry - ignore most recent post), my comments/findings the same however.......
Paul
-----Original Message-----
Hi again Sage
This is very perplexing. Confirming this system is a stock Ubuntu 12.04 x64, with no custom kernel or anything else, fully apt-get dist-upgrade'd up to date.
root@dsanb1-coy:~# uname -r
3.2.0-26-generic
I have added in the suggestions you made to the script, we now have:
modprobe btrfs || true
which mkfs.btrfs
echo "RUNNING: mkfs.btrfs $btrfs_devs"
mkfs.btrfs $btrfs_devs
btrfs device scan || btrfsctl -a
which mount
echo "RUNNING: mount -t btrfs $btrfs_opt $first_dev $btrfs_path"
mount -t btrfs $btrfs_opt $first_dev $btrfs_path
echo "DID I GET HERE - OR CRASH OUT WITH mount ABOVE?"
chown $osd_user $btrfs_path
chmod +w $btrfs_path
See below that the same command within the mkcephfs that is failing, is working fine on a standard command line:
=== osd.0 ===
--- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.ruZy4Apo23 --prepare-osdfs osd.0
umount: /dev/sdc: not mounted
/sbin/mkfs.btrfs
RUNNING: mkfs.btrfs /dev/sdc
WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using
fs created label (null) on /dev/sdc
nodesize 4096 leafsize 4096 sectorsize 4096 size 1.82TB
Btrfs Btrfs v0.19
Scanning for Btrfs filesystems
/bin/mount
RUNNING: mount -t btrfs -o noatime /dev/sdc /srv/osd.0
mount: wrong fs type, bad option, bad superblock on /dev/sdc,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
failed: '/sbin/mkcephfs -d /tmp/mkcephfs.ruZy4Apo23 --prepare-osdfs osd.0'
root@dsanb1-coy:~# /bin/mount -t btrfs -o noatime /dev/sdc /srv/osd.0
root@dsanb1-coy:~# mount | grep btrfs /dev/sdc on /srv/osd.0 type btrfs (rw,noatime)
Remember, this is not isolated to btrfs, as per my original post it fails when not specifying to use btrfs.
I can only conclude that /bin/sh &/or /bin/bash and the way they interact with the mkcephfs script, which does call itself etc, is somehow now become fuddled up? Must be something wiggy, when the script output confirms it is calling the same command ( /bin/mount ) but somehow finds a way for that to not work and therefore cause the mkcephfs script terminate.
Many thanks - will be a relief to sort this out, as all our Ceph project works are on hold til we can sort this one out.
Cheers
Paul
-----Original Message-----
From: Sage Weil [mailto:sage@inktank.com]
Sent: Friday, 6 July 2012 2:09 PM
To: Paul Pettigrew
Cc: ceph-devel@vger.kernel.org
Subject: RE: mkcephfs failing on v0.48 "argonaut"
On Fri, 6 Jul 2012, Paul Pettigrew wrote:
> Hi Sage - thanks so much for the quick response :-)
>
> Firstly, and it is a bit hard to see, but the command output below is run with the "-v" option. To help isolate what command line in the script is failing, I have added in some simple echo output, and the script now looks like:
>
>
> ### prepare-osdfs ###
>
> if [ -n "$prepareosdfs" ]; then
> <<SNIP>>
> modprobe btrfs || true
> echo "RUNNING: mkfs.btrfs $btrfs_devs"
> mkfs.btrfs $btrfs_devs
> btrfs device scan || btrfsctl -a
> echo "RUNNING: mount -t btrfs $btrfs_opt $first_dev $btrfs_path"
> mount -t btrfs $btrfs_opt $first_dev $btrfs_path echo "DID I GET
> HERE - OR CRASH OUT WITH mount ABOVE?"
> chown $osd_user $btrfs_path
> chmod +w $btrfs_path
>
> exit 0
> fi
>
> Per the modified script the above, here is the output displayed when running the script:
>
> root@dsanb1-coy:/srv# /sbin/mkcephfs -c /etc/ceph/ceph.conf --allhosts
> --mkbtrfs -k /etc/ceph/keyring --crushmapsrc crushfile.txt -v temp dir
> is /tmp/mkcephfs.uelzdJ82ej preparing monmap in
> /tmp/mkcephfs.uelzdJ82ej/monmap /usr/bin/monmaptool --create --clobber
> --add alpha 10.32.0.10:6789 --add bravo 10.32.0.25:6789 --add charlie
> 10.32.0.11:6789 --print /tmp/mkcephfs.uelzdJ82ej/monmap
> /usr/bin/monmaptool: monmap file /tmp/mkcephfs.uelzdJ82ej/monmap
> /usr/bin/monmaptool: generated fsid
> b254abdd-e036-4186-b6d5-e32b14e53b45
> epoch 0
> fsid b254abdd-e036-4186-b6d5-e32b14e53b45
> last_changed 2012-07-06 12:31:38.416848 created 2012-07-06
> 12:31:38.416848
> 0: 10.32.0.10:6789/0 mon.alpha
> 1: 10.32.0.11:6789/0 mon.charlie
> 2: 10.32.0.25:6789/0 mon.bravo
> /usr/bin/monmaptool: writing epoch 0 to
> /tmp/mkcephfs.uelzdJ82ej/monmap (3 monitors) /usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.0 "user"
> === osd.0 ===
> --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.uelzdJ82ej
> --prepare-osdfs osd.0
> umount: /srv/osd.0: not mounted
> umount: /dev/sdc: not mounted
> RUNNING: mkfs.btrfs /dev/sdc
>
> WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL WARNING! - see
> http://btrfs.wiki.kernel.org before using
>
> fs created label (null) on /dev/sdc
> nodesize 4096 leafsize 4096 sectorsize 4096 size 1.82TB Btrfs
> Btrfs v0.19 Scanning for Btrfs filesystems
> RUNNING: mount -t btrfs -o noatime /dev/sdc /srv/osd.0
> mount: wrong fs type, bad option, bad superblock on /dev/sdc,
> missing codepage or helper program, or other error
> In some cases useful info is found in syslog - try
> dmesg | tail or so
>
> failed: '/sbin/mkcephfs -d /tmp/mkcephfs.uelzdJ82ej --prepare-osdfs osd.0'
>
>
> Which clearly isolates the issue to the "mount" command line.
>
> The trouble is, I can run this precise line on the command line directly without error:
>
> root@dsanb1-coy:/srv# mount -t btrfs -o noatime /dev/sdc /srv/osd.0
> root@dsanb1-coy:/srv# mount | grep btrfs /dev/sdc on /srv/osd.0 type
> btrfs (rw,noatime)
What if you run the exact sequence of commands that mkcephfs is doing?
(mkfs.btrfs, btrfs ..., mount ...). If that doesn't work, put `which mkfs.btfs` etc in the script to make sure you're running the exact version the script is...
sage
>
>
> Therefore, what could possibly be preventing the mkcephfs running a simple mount command on the first OSD disk it gets to, that otherwise works fine from the command line?
>
> Many thanks Sage
>
> Paul
>
> PS: changing the " btrfs device scan || btrfsctl -a" line as proposed had no effect, and neither did putting in a "sleep 10" immediately before the mount line.
> PPS: zerofilling the /dev/sdc and then re-creating a partition and mounting manually, then writing data to it is all fine. Same errors if we substitute any of the other HDD's in the server as 1st/osd.0. Ie, cannot see any issues with the hardware.
>
>
>
>
>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org
> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
> Sent: Friday, 6 July 2012 8:18 AM
> To: Paul Pettigrew
> Cc: ceph-devel@vger.kernel.org
> Subject: Re: mkcephfs failing on v0.48 "argonaut"
>
> Hi Paul,
>
> On Wed, 4 Jul 2012, Paul Pettigrew wrote:
> > Firstly, well done guys on achieving this version milestone. I
> > successfully upgraded to the 0.48 format uneventfully on a live
> > (test) system.
> >
> > The same system was then going through "rebuild" testing, to confirm
> > that also worked fine.
> >
> >
> > Unfortunately, the mkcephfs command is failing:
> >
> > root@dsanb1-coy:~# mkcephfs -c /etc/ceph/ceph.conf --allhosts
> > --mkbtrfs -k /etc/ceph/keyring --crushmapsrc crushfile.txt -v temp
> > dir is /tmp/mkcephfs.GaRCZ9i06a preparing monmap in
> > /tmp/mkcephfs.GaRCZ9i06a/monmap /usr/bin/monmaptool --create
> > --clobber --add alpha 10.32.0.10:6789 --add bravo 10.32.0.25:6789
> > --add charlie
> > 10.32.0.11:6789 --print /tmp/mkcephfs.GaRCZ9i06a/monmap
> > /usr/bin/monmaptool: monmap file /tmp/mkcephfs.GaRCZ9i06a/monmap
> > /usr/bin/monmaptool: generated fsid
> > c7202495-468c-4678-b678-115c3ee33402
> > epoch 0
> > fsid c7202495-468c-4678-b678-115c3ee33402
> > last_changed 2012-07-04 15:02:31.732275 created 2012-07-04
> > 15:02:31.732275
> > 0: 10.32.0.10:6789/0 mon.alpha
> > 1: 10.32.0.11:6789/0 mon.charlie
> > 2: 10.32.0.25:6789/0 mon.bravo
> > /usr/bin/monmaptool: writing epoch 0 to
> > /tmp/mkcephfs.GaRCZ9i06a/monmap (3 monitors) /usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.0 "user"
> > === osd.0 ===
> > --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.GaRCZ9i06a
> > --prepare-osdfs osd.0
> > umount: /srv/osd.0: not mounted
> > umount: /dev/disk/by-wwn/wwn-0x50014ee601246234: not mounted
> >
> > WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL WARNING! - see
> > http://btrfs.wiki.kernel.org before using
> >
> > fs created label (null) on /dev/disk/by-wwn/wwn-0x50014ee601246234
> > nodesize 4096 leafsize 4096 sectorsize 4096 size 1.82TB
> > Btrfs Btrfs v0.19 Scanning for Btrfs filesystems
> > mount: wrong fs type, bad option, bad superblock on /dev/sdc,
> > missing codepage or helper program, or other error
> > In some cases useful info is found in syslog - try
> > dmesg | tail or so
> >
> > failed: '/sbin/mkcephfs -d /tmp/mkcephfs.GaRCZ9i06a --prepare-osdfs osd.0'
>
> Hmm. Can you try running with -v? That will tell us exactly which command it is running, and hopefully we can work backwards from there.
>
> > dmesg/syslog is spitting out at the time of this failure:
> >
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.751945] device fsid
> > 7de0d192-b710-4629-a201-849df1d9db17 devid 1 transid 27109 /dev/sdp
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.751987] device fsid
> > 08fc3479-2fa2-4388-8b61-83e2a742a13e devid 1 transid 28699 /dev/sdo
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.752023] device fsid
> > 8b4a7c43-1a05-4dcb-bbed-de2a5c933996 devid 1 transid 24346 /dev/sdn
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.752068] device fsid
> > ba5fb1ca-c642-49b1-8a41-7f56f8e59fbd devid 1 transid 27274 /dev/sdm
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761453] device fsid
> > 7fe8c5cf-bf8c-4276-90f2-c3f57f5275fb devid 1 transid 28724 /dev/sdi
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761518] device fsid
> > 93fa3631-1202-4d42-8908-e5ef4d3e600d devid 1 transid 25201 /dev/sdh
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761579] device fsid
> > b9a1b5e4-3e5e-4381-a29a-33470f4b870f devid 1 transid 23375 /dev/sdg
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761635] device fsid
> > 280ea990-23f8-4c43-9e56-140c82340fdc devid 1 transid 25559 /dev/sdf
> > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761693] device fsid
> > 2f724cde-6de5-4262-b195-1ba3eea2256e devid 1 transid 176 /dev/sde
> > Jul
> > 4 15:02:31 dsanb1-coy kernel: [ 2306.761732] device fsid
> > a66f890f-8b08-4393-aab0-f222637ca5a4 devid 1 transid 7 /dev/sdd Jul
> > 4
> > 15:02:31 dsanb1-coy kernel: [ 2306.761769] device fsid
> > 6c181a94-697c-4e0c-af0d-05eb04d3626c devid 1 transid 7 /dev/sdc Jul
> > 4
> > 15:02:31 dsanb1-coy kernel: [ 2306.775931] device fsid
> > 6c181a94-697c-4e0c-af0d-05eb04d3626c devid 1 transid 7 /dev/sdc Jul
> > 4
> > 15:02:31 dsanb1-coy kernel: [ 2306.779716] btrfs bad fsid on block
> > 20971520 Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.791594] btrfs bad
> > fsid on block 20971520 Jul 4 15:02:31 dsanb1-coy kernel: [
> > 2306.803608] btrfs bad fsid on block 20971520 Jul 4 15:02:31
> > dsanb1-coy kernel: [ 2306.815541] btrfs bad fsid on block 20971520
> > Jul
> > 4 15:02:31 dsanb1-coy kernel: [ 2306.815878] btrfs bad fsid on block
> > 20971520 Jul 4 15:02:32 dsanb1-coy kernel: [ 2306.823554] btrfs bad
> > fsid on block 20971520 Jul 4 15:02:32 dsanb1-coy kernel: [
> > 2306.823797] btrfs bad fsid on block 20971520 Jul 4 15:02:32
> > dsanb1-coy kernel: [ 2306.823887] btrfs: failed to read chunk root
> > on sdc Jul 4 15:02:32 dsanb1-coy kernel: [ 2306.825622] btrfs:
> > open_ctree failed
>
> Long shot, but is the kernel on that machine recent?
>
> > Also fails if not forcing to use btrfs, eg:
> >
> > root@dsanb1-coy:~# mkcephfs -c /etc/ceph/ceph.conf --allhosts -k
> > /etc/ceph/keyring --crushmapsrc crushfile.txt -v temp dir is
> > /tmp/mkcephfs.ZOh6tBPAH0 preparing monmap in
> > /tmp/mkcephfs.ZOh6tBPAH0/monmap /usr/bin/monmaptool --create
> > --clobber --add alpha 10.32.0.10:6789 --add bravo 10.32.0.25:6789
> > --add charlie
> > 10.32.0.11:6789 --print /tmp/mkcephfs.ZOh6tBPAH0/monmap
> > /usr/bin/monmaptool: monmap file /tmp/mkcephfs.ZOh6tBPAH0/monmap
> > /usr/bin/monmaptool: generated fsid
> > adb8d65c-a823-4dc2-9415-22b0d7252699
> > epoch 0
> > fsid adb8d65c-a823-4dc2-9415-22b0d7252699
> > last_changed 2012-07-04 15:04:17.423368 created 2012-07-04
> > 15:04:17.423368
> > 0: 10.32.0.10:6789/0 mon.alpha
> > 1: 10.32.0.11:6789/0 mon.charlie
> > 2: 10.32.0.25:6789/0 mon.bravo
> > /usr/bin/monmaptool: writing epoch 0 to
> > /tmp/mkcephfs.ZOh6tBPAH0/monmap (3 monitors) /usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.0 "user"
> > === osd.0 ===
> > --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.ZOh6tBPAH0
> > --init-daemon osd.0
> > 2012-07-04 15:04:17.789064 7fc7fadca780 -1 filestore(/srv/osd.0)
> > limited size xattrs -- enable filestore_xattr_use_omap
> > 2012-07-04 15:04:17.789120 7fc7fadca780 -1 OSD::mkfs: couldn't mount
> > FileStore: error -95
> > 2012-07-04 15:04:17.789161 7fc7fadca780 -1 ** ERROR: error creating
> > empty object store in /srv/osd.0: (95) Operation not supported
> > failed: '/sbin/mkcephfs -d /tmp/mkcephfs.ZOh6tBPAH0 --init-daemon osd.0'
> >
> >
> > Confirming all this was working previously, and the crushmap, config
> > file, etc are all proven to be OK (get same failure when not
> > specifying a custom crushmap also). Also note that whilst the above
> > is failing on
> > osd.0 creation, I have swapped disk references and still get the
> > same failure on different HDD's when they are hooked in as osd.0
>
> The only thing that changed from v0.47 is the below. Can you try replacing 'btrfs device scan || btrfsctl -a' with 'btrfs device scan ; btrfsctl -a'? Maybe the btrfs tool isn't being pendantic about return codes...
>
> sage
>
>
> commit a414fd51c7c5ae5dbe9e3af7db6f17741a58c1a7
> Author: Sage Weil <sage.weil@dreamhost.com>
> Date: Sat Feb 11 13:43:23 2012 -0800
>
> init-ceph, mkcephfs: try 'btrfs device scan' before 'btrfsctl -a'
>
> Fixes: #2023
> Reported-by: Wido den Hollander <wido@widodh.nl>
> Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
>
> diff --git a/src/mkcephfs.in b/src/mkcephfs.in index 83fb932..17b6014
> 100644
> --- a/src/mkcephfs.in
> +++ b/src/mkcephfs.in
> @@ -332,7 +332,7 @@ if [ -n "$prepareosdfs" ]; then
>
> modprobe btrfs || true
> mkfs.btrfs $btrfs_devs
> - btrfsctl -a
> + btrfs device scan || btrfsctl -a
> mount -t btrfs $btrfs_opt $first_dev $btrfs_path
> chown $osd_user $btrfs_path
> chmod +w $btrfs_path
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo
> info at http://vger.kernel.org/majordomo-info.html
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: mkcephfs failing on v0.48 "argonaut"
2012-07-07 3:33 ` Paul Pettigrew
2012-07-07 3:45 ` Paul Pettigrew
@ 2012-07-07 4:20 ` Sage Weil
2012-07-07 21:15 ` Paul Pettigrew
1 sibling, 1 reply; 9+ messages in thread
From: Sage Weil @ 2012-07-07 4:20 UTC (permalink / raw)
To: Paul Pettigrew; +Cc: ceph-devel
On Sat, 7 Jul 2012, Paul Pettigrew wrote:
> Hi again Sage
>
> This is very perplexing. Confirming this system is a stock Ubuntu 12.04 x64, with no custom kernel or anything else, fully apt-get dist-upgrade'd up to date.
> root@dsanb1-coy:~# uname -r
> 3.2.0-26-generic
>
> I have added in the suggestions you made to the script, we now have:
>
> modprobe btrfs || true
> which mkfs.btrfs
> echo "RUNNING: mkfs.btrfs $btrfs_devs"
> mkfs.btrfs $btrfs_devs
> btrfs device scan || btrfsctl -a
> echo "RUNNING: mount -t btrfs $btrfs_opt $first_dev $btrfs_path"
> which mount
> mount -t btrfs $btrfs_opt $first_dev $btrfs_path
> echo "DID I GET HERE - OR CRASH OUT WITH mount ABOVE?"
> chown $osd_user $btrfs_path
>
>
> See below that the same command within the mkcephfs that is failing, is
> working fine on a standard command line:
Weirdness!
>
> === osd.0 ===
> --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.xgk025tjkQ --prepare-osdfs osd.0
Can you run this command with -x to see exactly what bash is doing?
sh -x /sbin/mkcephfs -d /tmp/mkcephfs.xgk025tjkQ --prepare-osdfs osd.0
In particular, I'm curious if you do
mkfs.btrfs /dev/sdc
btrfs device scan
mount /dev/sdc /srv/osd.0
(or whatever the exact sequence that mkcephfs does is) from the command
line, does it give you the same error?
sage
> umount: /srv/osd.0: not mounted
> umount: /dev/sdc: not mounted
> /sbin/mkfs.btrfs
> RUNNING: mkfs.btrfs /dev/sdc
>
> WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL
> WARNING! - see http://btrfs.wiki.kernel.org before using
>
> fs created label (null) on /dev/sdc
> nodesize 4096 leafsize 4096 sectorsize 4096 size 1.82TB
> Btrfs Btrfs v0.19
> Scanning for Btrfs filesystems
> RUNNING: mount -t btrfs -o noatime /dev/sdc /srv/osd.0
> /bin/mount
> mount: wrong fs type, bad option, bad superblock on /dev/sdc,
> missing codepage or helper program, or other error
> In some cases useful info is found in syslog - try
> dmesg | tail or so
>
> failed: '/sbin/mkcephfs -d /tmp/mkcephfs.xgk025tjkQ --prepare-osdfs osd.0'
> root@dsanb1-coy:~# /bin/mount -t btrfs -o noatime /dev/sdc /srv/osd.0
> root@dsanb1-coy:~# mount | grep btrfs
> /dev/sdc on /srv/osd.0 type btrfs (rw,noatime)
>
>
> Remember, this is not isolated to btrfs, as per my original post it fails when not specifying to use btrfs.
>
> I can only conclude that /bin/sh &/or /bin/bash and the way they interact with the mkcephfs script, which does call itself etc, is somehow now become fuddled up? Must be something wiggy, when the script output confirms it is calling the same command ( /bin/mount ) but somehow finds a way for that to not work and therefore cause the mkcephfs script termin5Date.
>
> Many thanks - will be a relief to sort this out, as all our Ceph project works are on hold til we can sort this one out.
>
> Cheers
>
> Paul
>
>
>
> -----Original Message-----
> From: Sage Weil [mailto:sage@inktank.com]
> Sent: Friday, 6 July 2012 2:09 PM
> To: Paul Pettigrew
> Cc: ceph-devel@vger.kernel.org
> Subject: RE: mkcephfs failing on v0.48 "argonaut"
>
> On Fri, 6 Jul 2012, Paul Pettigrew wrote:
> > Hi Sage - thanks so much for the quick response :-)
> >
> > Firstly, and it is a bit hard to see, but the command output below is run with the "-v" option. To help isolate what command line in the script is failing, I have added in some simple echo output, and the script now looks like:
> >
> >
> > ### prepare-osdfs ###
> >
> > if [ -n "$prepareosdfs" ]; then
> > <<SNIP>>
> > modprobe btrfs || true
> > echo "RUNNING: mkfs.btrfs $btrfs_devs"
> > mkfs.btrfs $btrfs_devs
> > btrfs device scan || btrfsctl -a
> > echo "RUNNING: mount -t btrfs $btrfs_opt $first_dev $btrfs_path"
> > mount -t btrfs $btrfs_opt $first_dev $btrfs_path echo "DID I GET
> > HERE - OR CRASH OUT WITH mount ABOVE?"
> > chown $osd_user $btrfs_path
> > chmod +w $btrfs_path
> >
> > exit 0
> > fi
> >
> > Per the modified script the above, here is the output displayed when running the script:
> >
> > root@dsanb1-coy:/srv# /sbin/mkcephfs -c /etc/ceph/ceph.conf --allhosts
> > --mkbtrfs -k /etc/ceph/keyring --crushmapsrc crushfile.txt -v temp dir
> > is /tmp/mkcephfs.uelzdJ82ej preparing monmap in
> > /tmp/mkcephfs.uelzdJ82ej/monmap /usr/bin/monmaptool --create --clobber
> > --add alpha 10.32.0.10:6789 --add bravo 10.32.0.25:6789 --add charlie
> > 10.32.0.11:6789 --print /tmp/mkcephfs.uelzdJ82ej/monmap
> > /usr/bin/monmaptool: monmap file /tmp/mkcephfs.uelzdJ82ej/monmap
> > /usr/bin/monmaptool: generated fsid
> > b254abdd-e036-4186-b6d5-e32b14e53b45
> > epoch 0
> > fsid b254abdd-e036-4186-b6d5-e32b14e53b45
> > last_changed 2012-07-06 12:31:38.416848 created 2012-07-06
> > 12:31:38.416848
> > 0: 10.32.0.10:6789/0 mon.alpha
> > 1: 10.32.0.11:6789/0 mon.charlie
> > 2: 10.32.0.25:6789/0 mon.bravo
> > /usr/bin/monmaptool: writing epoch 0 to
> > /tmp/mkcephfs.uelzdJ82ej/monmap (3 monitors) /usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.0 "user"
> > === osd.0 ===
> > --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.uelzdJ82ej
> > --prepare-osdfs osd.0
> > umount: /srv/osd.0: not mounted
> > umount: /dev/sdc: not mounted
> > RUNNING: mkfs.btrfs /dev/sdc
> >
> > WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL WARNING! - see
> > http://btrfs.wiki.kernel.org before using
> >
> > fs created label (null) on /dev/sdc
> > nodesize 4096 leafsize 4096 sectorsize 4096 size 1.82TB Btrfs
> > Btrfs v0.19 Scanning for Btrfs filesystems
> > RUNNING: mount -t btrfs -o noatime /dev/sdc /srv/osd.0
> > mount: wrong fs type, bad option, bad superblock on /dev/sdc,
> > missing codepage or helper program, or other error
> > In some cases useful info is found in syslog - try
> > dmesg | tail or so
> >
> > failed: '/sbin/mkcephfs -d /tmp/mkcephfs.uelzdJ82ej --prepare-osdfs osd.0'
> >
> >
> > Which clearly isolates the issue to the "mount" command line.
> >
> > The trouble is, I can run this precise line on the command line directly without error:
> >
> > root@dsanb1-coy:/srv# mount -t btrfs -o noatime /dev/sdc /srv/osd.0
> > root@dsanb1-coy:/srv# mount | grep btrfs /dev/sdc on /srv/osd.0 type
> > btrfs (rw,noatime)
>
> What if you run the exact sequence of commands that mkcephfs is doing?
> (mkfs.btrfs, btrfs ..., mount ...). If that doesn't work, put `which mkfs.btfs` etc in the script to make sure you're running the exact version the script is...
>
> sage
>
>
>
> >
> >
> > Therefore, what could possibly be preventing the mkcephfs running a simple mount command on the first OSD disk it gets to, that otherwise works fine from the command line?
> >
> > Many thanks Sage
> >
> > Paul
> >
> > PS: changing the " btrfs device scan || btrfsctl -a" line as proposed had no effect, and neither did putting in a "sleep 10" immediately before the mount line.
> > PPS: zerofilling the /dev/sdc and then re-creating a partition and mounting manually, then writing data to it is all fine. Same errors if we substitute any of the other HDD's in the server as 1st/osd.0. Ie, cannot see any issues with the hardware.
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: ceph-devel-owner@vger.kernel.org
> > [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
> > Sent: Friday, 6 July 2012 8:18 AM
> > To: Paul Pettigrew
> > Cc: ceph-devel@vger.kernel.org
> > Subject: Re: mkcephfs failing on v0.48 "argonaut"
> >
> > Hi Paul,
> >
> > On Wed, 4 Jul 2012, Paul Pettigrew wrote:
> > > Firstly, well done guys on achieving this version milestone. I
> > > successfully upgraded to the 0.48 format uneventfully on a live
> > > (test) system.
> > >
> > > The same system was then going through "rebuild" testing, to confirm
> > > that also worked fine.
> > >
> > >
> > > Unfortunately, the mkcephfs command is failing:
> > >
> > > root@dsanb1-coy:~# mkcephfs -c /etc/ceph/ceph.conf --allhosts
> > > --mkbtrfs -k /etc/ceph/keyring --crushmapsrc crushfile.txt -v temp
> > > dir is /tmp/mkcephfs.GaRCZ9i06a preparing monmap in
> > > /tmp/mkcephfs.GaRCZ9i06a/monmap /usr/bin/monmaptool --create
> > > --clobber --add alpha 10.32.0.10:6789 --add bravo 10.32.0.25:6789
> > > --add charlie
> > > 10.32.0.11:6789 --print /tmp/mkcephfs.GaRCZ9i06a/monmap
> > > /usr/bin/monmaptool: monmap file /tmp/mkcephfs.GaRCZ9i06a/monmap
> > > /usr/bin/monmaptool: generated fsid
> > > c7202495-468c-4678-b678-115c3ee33402
> > > epoch 0
> > > fsid c7202495-468c-4678-b678-115c3ee33402
> > > last_changed 2012-07-04 15:02:31.732275 created 2012-07-04
> > > 15:02:31.732275
> > > 0: 10.32.0.10:6789/0 mon.alpha
> > > 1: 10.32.0.11:6789/0 mon.charlie
> > > 2: 10.32.0.25:6789/0 mon.bravo
> > > /usr/bin/monmaptool: writing epoch 0 to
> > > /tmp/mkcephfs.GaRCZ9i06a/monmap (3 monitors) /usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.0 "user"
> > > === osd.0 ===
> > > --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.GaRCZ9i06a
> > > --prepare-osdfs osd.0
> > > umount: /srv/osd.0: not mounted
> > > umount: /dev/disk/by-wwn/wwn-0x50014ee601246234: not mounted
> > >
> > > WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL WARNING! - see
> > > http://btrfs.wiki.kernel.org before using
> > >
> > > fs created label (null) on /dev/disk/by-wwn/wwn-0x50014ee601246234
> > > nodesize 4096 leafsize 4096 sectorsize 4096 size 1.82TB
> > > Btrfs Btrfs v0.19 Scanning for Btrfs filesystems
> > > mount: wrong fs type, bad option, bad superblock on /dev/sdc,
> > > missing codepage or helper program, or other error
> > > In some cases useful info is found in syslog - try
> > > dmesg | tail or so
> > >
> > > failed: '/sbin/mkcephfs -d /tmp/mkcephfs.GaRCZ9i06a --prepare-osdfs osd.0'
> >
> > Hmm. Can you try running with -v? That will tell us exactly which command it is running, and hopefully we can work backwards from there.
> >
> > > dmesg/syslog is spitting out at the time of this failure:
> > >
> > > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.751945] device fsid
> > > 7de0d192-b710-4629-a201-849df1d9db17 devid 1 transid 27109 /dev/sdp
> > > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.751987] device fsid
> > > 08fc3479-2fa2-4388-8b61-83e2a742a13e devid 1 transid 28699 /dev/sdo
> > > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.752023] device fsid
> > > 8b4a7c43-1a05-4dcb-bbed-de2a5c933996 devid 1 transid 24346 /dev/sdn
> > > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.752068] device fsid
> > > ba5fb1ca-c642-49b1-8a41-7f56f8e59fbd devid 1 transid 27274 /dev/sdm
> > > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761453] device fsid
> > > 7fe8c5cf-bf8c-4276-90f2-c3f57f5275fb devid 1 transid 28724 /dev/sdi
> > > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761518] device fsid
> > > 93fa3631-1202-4d42-8908-e5ef4d3e600d devid 1 transid 25201 /dev/sdh
> > > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761579] device fsid
> > > b9a1b5e4-3e5e-4381-a29a-33470f4b870f devid 1 transid 23375 /dev/sdg
> > > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761635] device fsid
> > > 280ea990-23f8-4c43-9e56-140c82340fdc devid 1 transid 25559 /dev/sdf
> > > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761693] device fsid
> > > 2f724cde-6de5-4262-b195-1ba3eea2256e devid 1 transid 176 /dev/sde
> > > Jul
> > > 4 15:02:31 dsanb1-coy kernel: [ 2306.761732] device fsid
> > > a66f890f-8b08-4393-aab0-f222637ca5a4 devid 1 transid 7 /dev/sdd Jul
> > > 4
> > > 15:02:31 dsanb1-coy kernel: [ 2306.761769] device fsid
> > > 6c181a94-697c-4e0c-af0d-05eb04d3626c devid 1 transid 7 /dev/sdc Jul
> > > 4
> > > 15:02:31 dsanb1-coy kernel: [ 2306.775931] device fsid
> > > 6c181a94-697c-4e0c-af0d-05eb04d3626c devid 1 transid 7 /dev/sdc Jul
> > > 4
> > > 15:02:31 dsanb1-coy kernel: [ 2306.779716] btrfs bad fsid on block
> > > 20971520 Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.791594] btrfs bad
> > > fsid on block 20971520 Jul 4 15:02:31 dsanb1-coy kernel: [
> > > 2306.803608] btrfs bad fsid on block 20971520 Jul 4 15:02:31
> > > dsanb1-coy kernel: [ 2306.815541] btrfs bad fsid on block 20971520
> > > Jul
> > > 4 15:02:31 dsanb1-coy kernel: [ 2306.815878] btrfs bad fsid on block
> > > 20971520 Jul 4 15:02:32 dsanb1-coy kernel: [ 2306.823554] btrfs bad
> > > fsid on block 20971520 Jul 4 15:02:32 dsanb1-coy kernel: [
> > > 2306.823797] btrfs bad fsid on block 20971520 Jul 4 15:02:32
> > > dsanb1-coy kernel: [ 2306.823887] btrfs: failed to read chunk root
> > > on sdc Jul 4 15:02:32 dsanb1-coy kernel: [ 2306.825622] btrfs:
> > > open_ctree failed
> >
> > Long shot, but is the kernel on that machine recent?
> >
> > > Also fails if not forcing to use btrfs, eg:
> > >
> > > root@dsanb1-coy:~# mkcephfs -c /etc/ceph/ceph.conf --allhosts -k
> > > /etc/ceph/keyring --crushmapsrc crushfile.txt -v temp dir is
> > > /tmp/mkcephfs.ZOh6tBPAH0 preparing monmap in
> > > /tmp/mkcephfs.ZOh6tBPAH0/monmap /usr/bin/monmaptool --create
> > > --clobber --add alpha 10.32.0.10:6789 --add bravo 10.32.0.25:6789
> > > --add charlie
> > > 10.32.0.11:6789 --print /tmp/mkcephfs.ZOh6tBPAH0/monmap
> > > /usr/bin/monmaptool: monmap file /tmp/mkcephfs.ZOh6tBPAH0/monmap
> > > /usr/bin/monmaptool: generated fsid
> > > adb8d65c-a823-4dc2-9415-22b0d7252699
> > > epoch 0
> > > fsid adb8d65c-a823-4dc2-9415-22b0d7252699
> > > last_changed 2012-07-04 15:04:17.423368 created 2012-07-04
> > > 15:04:17.423368
> > > 0: 10.32.0.10:6789/0 mon.alpha
> > > 1: 10.32.0.11:6789/0 mon.charlie
> > > 2: 10.32.0.25:6789/0 mon.bravo
> > > /usr/bin/monmaptool: writing epoch 0 to
> > > /tmp/mkcephfs.ZOh6tBPAH0/monmap (3 monitors) /usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.0 "user"
> > > === osd.0 ===
> > > --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.ZOh6tBPAH0
> > > --init-daemon osd.0
> > > 2012-07-04 15:04:17.789064 7fc7fadca780 -1 filestore(/srv/osd.0)
> > > limited size xattrs -- enable filestore_xattr_use_omap
> > > 2012-07-04 15:04:17.789120 7fc7fadca780 -1 OSD::mkfs: couldn't mount
> > > FileStore: error -95
> > > 2012-07-04 15:04:17.789161 7fc7fadca780 -1 ** ERROR: error creating
> > > empty object store in /srv/osd.0: (95) Operation not supported
> > > failed: '/sbin/mkcephfs -d /tmp/mkcephfs.ZOh6tBPAH0 --init-daemon osd.0'
> > >
> > >
> > > Confirming all this was working previously, and the crushmap, config
> > > file, etc are all proven to be OK (get same failure when not
> > > specifying a custom crushmap also). Also note that whilst the above
> > > is failing on
> > > osd.0 creation, I have swapped disk references and still get the
> > > same failure on different HDD's when they are hooked in as osd.0
> >
> > The only thing that changed from v0.47 is the below. Can you try replacing 'btrfs device scan || btrfsctl -a' with 'btrfs device scan ; btrfsctl -a'? Maybe the btrfs tool isn't being pendantic about return codes...
> >
> > sage
> >
> >
> > commit a414fd51c7c5ae5dbe9e3af7db6f17741a58c1a7
> > Author: Sage Weil <sage.weil@dreamhost.com>
> > Date: Sat Feb 11 13:43:23 2012 -0800
> >
> > init-ceph, mkcephfs: try 'btrfs device scan' before 'btrfsctl -a'
> >
> > Fixes: #2023
> > Reported-by: Wido den Hollander <wido@widodh.nl>
> > Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
> >
> > diff --git a/src/mkcephfs.in b/src/mkcephfs.in index 83fb932..17b6014
> > 100644
> > --- a/src/mkcephfs.in
> > +++ b/src/mkcephfs.in
> > @@ -332,7 +332,7 @@ if [ -n "$prepareosdfs" ]; then
> >
> > modprobe btrfs || true
> > mkfs.btrfs $btrfs_devs
> > - btrfsctl -a
> > + btrfs device scan || btrfsctl -a
> > mount -t btrfs $btrfs_opt $first_dev $btrfs_path
> > chown $osd_user $btrfs_path
> > chmod +w $btrfs_path
> >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > in the body of a message to majordomo@vger.kernel.org More majordomo
> > info at http://vger.kernel.org/majordomo-info.html
> >
> >
> >
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: mkcephfs failing on v0.48 "argonaut"
2012-07-07 4:20 ` Sage Weil
@ 2012-07-07 21:15 ` Paul Pettigrew
2012-07-10 23:34 ` Sage Weil
0 siblings, 1 reply; 9+ messages in thread
From: Paul Pettigrew @ 2012-07-07 21:15 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-devel
Hi Sage
Confirming running the commands from a root prompt in the same sequence as requested:
mkfs.btrfs /dev/sdc
btrfs device scan
mount /dev/sdc /srv/osd.0
root@dsanb1-coy:~# mkfs.btrfs /dev/sdc
WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using
fs created label (null) on /dev/sdc
nodesize 4096 leafsize 4096 sectorsize 4096 size 1.82TB
Btrfs Btrfs v0.19
root@dsanb1-coy:~# btrfs device scan
Scanning for Btrfs filesystems
root@dsanb1-coy:~# mount /dev/sdc /srv/osd.0
mount: wrong fs type, bad option, bad superblock on /dev/sdc,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
root@dsanb1-coy:~# mount | grep btrfs
root@dsanb1-coy:~# mount -t btrfs /dev/sdc /srv/osd.0
root@dsanb1-coy:~# mount | grep btrfs
/dev/sdc on /srv/osd.0 type btrfs (rw)
So - you can see I had to explicitly set the flag to mount "-t btrfs". It seems that when mkcepfs is running the command "mount -t btrfs -o noatime /dev/sdc /srv/osd.0" it is doing so with the effect that the "-t btrfs" was not present in the line, but it is (symptom the same at least). Crazy.
Secondly, as requested I cannot run the command "sh -x /sbin/mkcephfs -d /tmp/mkcephfs.xgk025tjkQ --prepare-osdfs osd.0" because the contents of the /tmp/ directory are deleted each time mkcephfs finishes its run. I have however called the overall mkcephfs command with "sh -x" so you can see what is occurring.
See below:
root@dsanb1-coy:~# sh -x /sbin/mkcephfs -c /etc/ceph/ceph.conf --allhosts --mkbtrfs -k /etc/ceph/keyring --crushmapsrc crushfile.txt -v
<SNIP>
=== osd.0 ===
+ return 0
+ [ -n ]
+ rdir=/tmp/mkcephfs.kJjIwsEnfZ
+ [ 0 -eq 0 ]
+ cp /tmp/mkcephfs.kJjIwsEnfZ/conf /etc/ceph/ceph.conf
+ [ 1 -eq 1 ]
+ [ osd = osd ]
+ do_root_cmd /sbin/mkcephfs -d /tmp/mkcephfs.kJjIwsEnfZ --prepare-osdfs osd.0
+ [ -z ]
+ [ 1 -eq 1 ]
+ echo --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.kJjIwsEnfZ --prepare-osdfs osd.0
--- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.kJjIwsEnfZ --prepare-osdfs osd.0
+ ulimit -c unlimited
+ whoami
+ whoami=root
+ [ root = root ]
+ bash -c /sbin/mkcephfs -d /tmp/mkcephfs.kJjIwsEnfZ --prepare-osdfs osd.0
umount: /srv/osd.0: not mounted
umount: /dev/sdc: not mounted
/sbin/mkfs.btrfs
RUNNING: mkfs.btrfs /dev/sdc
WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using
fs created label (null) on /dev/sdc
nodesize 4096 leafsize 4096 sectorsize 4096 size 1.82TB
Btrfs Btrfs v0.19
Scanning for Btrfs filesystems
/bin/mount
RUNNING: mount -t btrfs -o noatime /dev/sdc /srv/osd.0
mount: wrong fs type, bad option, bad superblock on /dev/sdc,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
+ echo failed: '/sbin/mkcephfs -d /tmp/mkcephfs.kJjIwsEnfZ --prepare-osdfs osd.0'
failed: '/sbin/mkcephfs -d /tmp/mkcephfs.kJjIwsEnfZ --prepare-osdfs osd.0'
+ exit 1
+ rm -rf /tmp/mkcephfs.kJjIwsEnfZ
+ exit
-----Original Message-----
From: Sage Weil [mailto:sage@inktank.com]
Sent: Saturday, 7 July 2012 2:20 PM
To: Paul Pettigrew
Cc: ceph-devel@vger.kernel.org
Subject: RE: mkcephfs failing on v0.48 "argonaut"
On Sat, 7 Jul 2012, Paul Pettigrew wrote:
> Hi again Sage
>
> This is very perplexing. Confirming this system is a stock Ubuntu 12.04 x64, with no custom kernel or anything else, fully apt-get dist-upgrade'd up to date.
> root@dsanb1-coy:~# uname -r
> 3.2.0-26-generic
>
> I have added in the suggestions you made to the script, we now have:
>
> modprobe btrfs || true
> which mkfs.btrfs
> echo "RUNNING: mkfs.btrfs $btrfs_devs"
> mkfs.btrfs $btrfs_devs
> btrfs device scan || btrfsctl -a
> which mount
> echo "RUNNING: mount -t btrfs $btrfs_opt $first_dev $btrfs_path"
> mount -t btrfs $btrfs_opt $first_dev $btrfs_path echo "DID I GET
> HERE - OR CRASH OUT WITH mount ABOVE?"
> chown $osd_user $btrfs_path
>
>
> See below that the same command within the mkcephfs that is failing,
> is working fine on a standard command line:
Weirdness!
>
> === osd.0 ===
> --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.xgk025tjkQ
> --prepare-osdfs osd.0
Can you run this command with -x to see exactly what bash is doing?
sh -x /sbin/mkcephfs -d /tmp/mkcephfs.xgk025tjkQ --prepare-osdfs osd.0
In particular, I'm curious if you do
mkfs.btrfs /dev/sdc
btrfs device scan
mount /dev/sdc /srv/osd.0
(or whatever the exact sequence that mkcephfs does is) from the command line, does it give you the same error?
sage
> umount: /srv/osd.0: not mounted
> umount: /dev/sdc: not mounted
> /sbin/mkfs.btrfs
> RUNNING: mkfs.btrfs /dev/sdc
>
> WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL WARNING! - see
> http://btrfs.wiki.kernel.org before using
>
> fs created label (null) on /dev/sdc
> nodesize 4096 leafsize 4096 sectorsize 4096 size 1.82TB Btrfs
> Btrfs v0.19 Scanning for Btrfs filesystems
> /bin/mount
> RUNNING: mount -t btrfs -o noatime /dev/sdc /srv/osd.0 /bin/mount
> mount: wrong fs type, bad option, bad superblock on /dev/sdc,
> missing codepage or helper program, or other error
> In some cases useful info is found in syslog - try
> dmesg | tail or so
>
> failed: '/sbin/mkcephfs -d /tmp/mkcephfs.xgk025tjkQ --prepare-osdfs osd.0'
> root@dsanb1-coy:~# /bin/mount -t btrfs -o noatime /dev/sdc /srv/osd.0
> root@dsanb1-coy:~# mount | grep btrfs /dev/sdc on /srv/osd.0 type
> btrfs (rw,noatime)
>
>
> Remember, this is not isolated to btrfs, as per my original post it fails when not specifying to use btrfs.
>
> I can only conclude that /bin/sh &/or /bin/bash and the way they interact with the mkcephfs script, which does call itself etc, is somehow now become fuddled up? Must be something wiggy, when the script output confirms it is calling the same command ( /bin/mount ) but somehow finds a way for that to not work and therefore cause the mkcephfs script terminate.
>
> Many thanks - will be a relief to sort this out, as all our Ceph project works are on hold til we can sort this one out.
>
> Cheers
>
> Paul
>
>
>
> -----Original Message-----
> From: Sage Weil [mailto:sage@inktank.com]
> Sent: Friday, 6 July 2012 2:09 PM
> To: Paul Pettigrew
> Cc: ceph-devel@vger.kernel.org
> Subject: RE: mkcephfs failing on v0.48 "argonaut"
>
> On Fri, 6 Jul 2012, Paul Pettigrew wrote:
> > Hi Sage - thanks so much for the quick response :-)
> >
> > Firstly, and it is a bit hard to see, but the command output below is run with the "-v" option. To help isolate what command line in the script is failing, I have added in some simple echo output, and the script now looks like:
> >
> >
> > ### prepare-osdfs ###
> >
> > if [ -n "$prepareosdfs" ]; then
> > <<SNIP>>
> > modprobe btrfs || true
> > echo "RUNNING: mkfs.btrfs $btrfs_devs"
> > mkfs.btrfs $btrfs_devs
> > btrfs device scan || btrfsctl -a echo "RUNNING: mount -t btrfs
> > $btrfs_opt $first_dev $btrfs_path"
> > mount -t btrfs $btrfs_opt $first_dev $btrfs_path echo "DID I GET
> > HERE - OR CRASH OUT WITH mount ABOVE?"
> > chown $osd_user $btrfs_path
> > chmod +w $btrfs_path
> >
> > exit 0
> > fi
> >
> > Per the modified script the above, here is the output displayed when running the script:
> >
> > root@dsanb1-coy:/srv# /sbin/mkcephfs -c /etc/ceph/ceph.conf
> > --allhosts --mkbtrfs -k /etc/ceph/keyring --crushmapsrc
> > crushfile.txt -v temp dir is /tmp/mkcephfs.uelzdJ82ej preparing
> > monmap in /tmp/mkcephfs.uelzdJ82ej/monmap /usr/bin/monmaptool
> > --create --clobber --add alpha 10.32.0.10:6789 --add bravo
> > 10.32.0.25:6789 --add charlie
> > 10.32.0.11:6789 --print /tmp/mkcephfs.uelzdJ82ej/monmap
> > /usr/bin/monmaptool: monmap file /tmp/mkcephfs.uelzdJ82ej/monmap
> > /usr/bin/monmaptool: generated fsid
> > b254abdd-e036-4186-b6d5-e32b14e53b45
> > epoch 0
> > fsid b254abdd-e036-4186-b6d5-e32b14e53b45
> > last_changed 2012-07-06 12:31:38.416848 created 2012-07-06
> > 12:31:38.416848
> > 0: 10.32.0.10:6789/0 mon.alpha
> > 1: 10.32.0.11:6789/0 mon.charlie
> > 2: 10.32.0.25:6789/0 mon.bravo
> > /usr/bin/monmaptool: writing epoch 0 to
> > /tmp/mkcephfs.uelzdJ82ej/monmap (3 monitors) /usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.0 "user"
> > === osd.0 ===
> > --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.uelzdJ82ej
> > --prepare-osdfs osd.0
> > umount: /srv/osd.0: not mounted
> > umount: /dev/sdc: not mounted
> > RUNNING: mkfs.btrfs /dev/sdc
> >
> > WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL WARNING! - see
> > http://btrfs.wiki.kernel.org before using
> >
> > fs created label (null) on /dev/sdc
> > nodesize 4096 leafsize 4096 sectorsize 4096 size 1.82TB
> > Btrfs Btrfs v0.19 Scanning for Btrfs filesystems
> > RUNNING: mount -t btrfs -o noatime /dev/sdc /srv/osd.0
> > mount: wrong fs type, bad option, bad superblock on /dev/sdc,
> > missing codepage or helper program, or other error
> > In some cases useful info is found in syslog - try
> > dmesg | tail or so
> >
> > failed: '/sbin/mkcephfs -d /tmp/mkcephfs.uelzdJ82ej --prepare-osdfs osd.0'
> >
> >
> > Which clearly isolates the issue to the "mount" command line.
> >
> > The trouble is, I can run this precise line on the command line directly without error:
> >
> > root@dsanb1-coy:/srv# mount -t btrfs -o noatime /dev/sdc /srv/osd.0
> > root@dsanb1-coy:/srv# mount | grep btrfs
> > /dev/sdc on /srv/osd.0 type btrfs (rw,noatime)
>
> What if you run the exact sequence of commands that mkcephfs is doing?
> (mkfs.btrfs, btrfs ..., mount ...). If that doesn't work, put `which mkfs.btfs` etc in the script to make sure you're running the exact version the script is...
>
> sage
>
>
>
> >
> >
> > Therefore, what could possibly be preventing the mkcephfs running a simple mount command on the first OSD disk it gets to, that otherwise works fine from the command line?
> >
> > Many thanks Sage
> >
> > Paul
> >
> > PS: changing the " btrfs device scan || btrfsctl -a" line as proposed had no effect, and neither did putting in a "sleep 10" immediately before the mount line.
> > PPS: zerofilling the /dev/sdc and then re-creating a partition and mounting manually, then writing data to it is all fine. Same errors if we substitute any of the other HDD's in the server as 1st/osd.0. Ie, cannot see any issues with the hardware.
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: ceph-devel-owner@vger.kernel.org
> > [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
> > Sent: Friday, 6 July 2012 8:18 AM
> > To: Paul Pettigrew
> > Cc: ceph-devel@vger.kernel.org
> > Subject: Re: mkcephfs failing on v0.48 "argonaut"
> >
> > Hi Paul,
> >
> > On Wed, 4 Jul 2012, Paul Pettigrew wrote:
> > > Firstly, well done guys on achieving this version milestone. I
> > > successfully upgraded to the 0.48 format uneventfully on a live
> > > (test) system.
> > >
> > > The same system was then going through "rebuild" testing, to
> > > confirm that also worked fine.
> > >
> > >
> > > Unfortunately, the mkcephfs command is failing:
> > >
> > > root@dsanb1-coy:~# mkcephfs -c /etc/ceph/ceph.conf --allhosts
> > > --mkbtrfs -k /etc/ceph/keyring --crushmapsrc crushfile.txt -v temp
> > > dir is /tmp/mkcephfs.GaRCZ9i06a preparing monmap in
> > > /tmp/mkcephfs.GaRCZ9i06a/monmap /usr/bin/monmaptool --create
> > > --clobber --add alpha 10.32.0.10:6789 --add bravo 10.32.0.25:6789
> > > --add charlie
> > > 10.32.0.11:6789 --print /tmp/mkcephfs.GaRCZ9i06a/monmap
> > > /usr/bin/monmaptool: monmap file /tmp/mkcephfs.GaRCZ9i06a/monmap
> > > /usr/bin/monmaptool: generated fsid
> > > c7202495-468c-4678-b678-115c3ee33402
> > > epoch 0
> > > fsid c7202495-468c-4678-b678-115c3ee33402
> > > last_changed 2012-07-04 15:02:31.732275 created 2012-07-04
> > > 15:02:31.732275
> > > 0: 10.32.0.10:6789/0 mon.alpha
> > > 1: 10.32.0.11:6789/0 mon.charlie
> > > 2: 10.32.0.25:6789/0 mon.bravo
> > > /usr/bin/monmaptool: writing epoch 0 to
> > > /tmp/mkcephfs.GaRCZ9i06a/monmap (3 monitors) /usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.0 "user"
> > > === osd.0 ===
> > > --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.GaRCZ9i06a
> > > --prepare-osdfs osd.0
> > > umount: /srv/osd.0: not mounted
> > > umount: /dev/disk/by-wwn/wwn-0x50014ee601246234: not mounted
> > >
> > > WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL WARNING! - see
> > > http://btrfs.wiki.kernel.org before using
> > >
> > > fs created label (null) on /dev/disk/by-wwn/wwn-0x50014ee601246234
> > > nodesize 4096 leafsize 4096 sectorsize 4096 size 1.82TB
> > > Btrfs Btrfs v0.19 Scanning for Btrfs filesystems
> > > mount: wrong fs type, bad option, bad superblock on /dev/sdc,
> > > missing codepage or helper program, or other error
> > > In some cases useful info is found in syslog - try
> > > dmesg | tail or so
> > >
> > > failed: '/sbin/mkcephfs -d /tmp/mkcephfs.GaRCZ9i06a --prepare-osdfs osd.0'
> >
> > Hmm. Can you try running with -v? That will tell us exactly which command it is running, and hopefully we can work backwards from there.
> >
> > > dmesg/syslog is spitting out at the time of this failure:
> > >
> > > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.751945] device fsid
> > > 7de0d192-b710-4629-a201-849df1d9db17 devid 1 transid 27109
> > > /dev/sdp Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.751987] device
> > > fsid 08fc3479-2fa2-4388-8b61-83e2a742a13e devid 1 transid 28699
> > > /dev/sdo Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.752023] device
> > > fsid
> > > 8b4a7c43-1a05-4dcb-bbed-de2a5c933996 devid 1 transid 24346
> > > /dev/sdn Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.752068] device
> > > fsid ba5fb1ca-c642-49b1-8a41-7f56f8e59fbd devid 1 transid 27274
> > > /dev/sdm Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761453] device
> > > fsid 7fe8c5cf-bf8c-4276-90f2-c3f57f5275fb devid 1 transid 28724
> > > /dev/sdi Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761518] device
> > > fsid 93fa3631-1202-4d42-8908-e5ef4d3e600d devid 1 transid 25201
> > > /dev/sdh Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761579] device
> > > fsid b9a1b5e4-3e5e-4381-a29a-33470f4b870f devid 1 transid 23375
> > > /dev/sdg Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761635] device
> > > fsid 280ea990-23f8-4c43-9e56-140c82340fdc devid 1 transid 25559
> > > /dev/sdf Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761693] device
> > > fsid 2f724cde-6de5-4262-b195-1ba3eea2256e devid 1 transid 176
> > > /dev/sde Jul
> > > 4 15:02:31 dsanb1-coy kernel: [ 2306.761732] device fsid
> > > a66f890f-8b08-4393-aab0-f222637ca5a4 devid 1 transid 7 /dev/sdd
> > > Jul
> > > 4
> > > 15:02:31 dsanb1-coy kernel: [ 2306.761769] device fsid
> > > 6c181a94-697c-4e0c-af0d-05eb04d3626c devid 1 transid 7 /dev/sdc
> > > Jul
> > > 4
> > > 15:02:31 dsanb1-coy kernel: [ 2306.775931] device fsid
> > > 6c181a94-697c-4e0c-af0d-05eb04d3626c devid 1 transid 7 /dev/sdc
> > > Jul
> > > 4
> > > 15:02:31 dsanb1-coy kernel: [ 2306.779716] btrfs bad fsid on block
> > > 20971520 Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.791594] btrfs
> > > bad fsid on block 20971520 Jul 4 15:02:31 dsanb1-coy kernel: [
> > > 2306.803608] btrfs bad fsid on block 20971520 Jul 4 15:02:31
> > > dsanb1-coy kernel: [ 2306.815541] btrfs bad fsid on block 20971520
> > > Jul
> > > 4 15:02:31 dsanb1-coy kernel: [ 2306.815878] btrfs bad fsid on
> > > block
> > > 20971520 Jul 4 15:02:32 dsanb1-coy kernel: [ 2306.823554] btrfs
> > > bad fsid on block 20971520 Jul 4 15:02:32 dsanb1-coy kernel: [
> > > 2306.823797] btrfs bad fsid on block 20971520 Jul 4 15:02:32
> > > dsanb1-coy kernel: [ 2306.823887] btrfs: failed to read chunk root
> > > on sdc Jul 4 15:02:32 dsanb1-coy kernel: [ 2306.825622] btrfs:
> > > open_ctree failed
> >
> > Long shot, but is the kernel on that machine recent?
> >
> > > Also fails if not forcing to use btrfs, eg:
> > >
> > > root@dsanb1-coy:~# mkcephfs -c /etc/ceph/ceph.conf --allhosts -k
> > > /etc/ceph/keyring --crushmapsrc crushfile.txt -v temp dir is
> > > /tmp/mkcephfs.ZOh6tBPAH0 preparing monmap in
> > > /tmp/mkcephfs.ZOh6tBPAH0/monmap /usr/bin/monmaptool --create
> > > --clobber --add alpha 10.32.0.10:6789 --add bravo 10.32.0.25:6789
> > > --add charlie
> > > 10.32.0.11:6789 --print /tmp/mkcephfs.ZOh6tBPAH0/monmap
> > > /usr/bin/monmaptool: monmap file /tmp/mkcephfs.ZOh6tBPAH0/monmap
> > > /usr/bin/monmaptool: generated fsid
> > > adb8d65c-a823-4dc2-9415-22b0d7252699
> > > epoch 0
> > > fsid adb8d65c-a823-4dc2-9415-22b0d7252699
> > > last_changed 2012-07-04 15:04:17.423368 created 2012-07-04
> > > 15:04:17.423368
> > > 0: 10.32.0.10:6789/0 mon.alpha
> > > 1: 10.32.0.11:6789/0 mon.charlie
> > > 2: 10.32.0.25:6789/0 mon.bravo
> > > /usr/bin/monmaptool: writing epoch 0 to
> > > /tmp/mkcephfs.ZOh6tBPAH0/monmap (3 monitors) /usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.0 "user"
> > > === osd.0 ===
> > > --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.ZOh6tBPAH0
> > > --init-daemon osd.0
> > > 2012-07-04 15:04:17.789064 7fc7fadca780 -1 filestore(/srv/osd.0)
> > > limited size xattrs -- enable filestore_xattr_use_omap
> > > 2012-07-04 15:04:17.789120 7fc7fadca780 -1 OSD::mkfs: couldn't
> > > mount
> > > FileStore: error -95
> > > 2012-07-04 15:04:17.789161 7fc7fadca780 -1 ** ERROR: error
> > > creating empty object store in /srv/osd.0: (95) Operation not
> > > supported
> > > failed: '/sbin/mkcephfs -d /tmp/mkcephfs.ZOh6tBPAH0 --init-daemon osd.0'
> > >
> > >
> > > Confirming all this was working previously, and the crushmap,
> > > config file, etc are all proven to be OK (get same failure when
> > > not specifying a custom crushmap also). Also note that whilst the
> > > above is failing on
> > > osd.0 creation, I have swapped disk references and still get the
> > > same failure on different HDD's when they are hooked in as osd.0
> >
> > The only thing that changed from v0.47 is the below. Can you try replacing 'btrfs device scan || btrfsctl -a' with 'btrfs device scan ; btrfsctl -a'? Maybe the btrfs tool isn't being pendantic about return codes...
> >
> > sage
> >
> >
> > commit a414fd51c7c5ae5dbe9e3af7db6f17741a58c1a7
> > Author: Sage Weil <sage.weil@dreamhost.com>
> > Date: Sat Feb 11 13:43:23 2012 -0800
> >
> > init-ceph, mkcephfs: try 'btrfs device scan' before 'btrfsctl -a'
> >
> > Fixes: #2023
> > Reported-by: Wido den Hollander <wido@widodh.nl>
> > Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
> >
> > diff --git a/src/mkcephfs.in b/src/mkcephfs.in index
> > 83fb932..17b6014
> > 100644
> > --- a/src/mkcephfs.in
> > +++ b/src/mkcephfs.in
> > @@ -332,7 +332,7 @@ if [ -n "$prepareosdfs" ]; then
> >
> > modprobe btrfs || true
> > mkfs.btrfs $btrfs_devs
> > - btrfsctl -a
> > + btrfs device scan || btrfsctl -a
> > mount -t btrfs $btrfs_opt $first_dev $btrfs_path
> > chown $osd_user $btrfs_path
> > chmod +w $btrfs_path
> >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > in the body of a message to majordomo@vger.kernel.org More majordomo
> > info at http://vger.kernel.org/majordomo-info.html
> >
> >
> >
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo
> info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: mkcephfs failing on v0.48 "argonaut"
2012-07-07 21:15 ` Paul Pettigrew
@ 2012-07-10 23:34 ` Sage Weil
0 siblings, 0 replies; 9+ messages in thread
From: Sage Weil @ 2012-07-10 23:34 UTC (permalink / raw)
To: Paul Pettigrew; +Cc: ceph-devel
Hi Paul,
Were you able to make any progress on this?
On Sun, 8 Jul 2012, Paul Pettigrew wrote:
> Hi Sage
>
> Confirming running the commands from a root prompt in the same sequence as requested:
> mkfs.btrfs /dev/sdc
> btrfs device scan
> mount /dev/sdc /srv/osd.0
>
>
> root@dsanb1-coy:~# mkfs.btrfs /dev/sdc
>
> WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL
> WARNING! - see http://btrfs.wiki.kernel.org before using
>
> fs created label (null) on /dev/sdc
> nodesize 4096 leafsize 4096 sectorsize 4096 size 1.82TB
> Btrfs Btrfs v0.19
> root@dsanb1-coy:~# btrfs device scan
> Scanning for Btrfs filesystems
> root@dsanb1-coy:~# mount /dev/sdc /srv/osd.0
> mount: wrong fs type, bad option, bad superblock on /dev/sdc,
> missing codepage or helper program, or other error
> In some cases useful info is found in syslog - try
> dmesg | tail or so
>
> root@dsanb1-coy:~# mount | grep btrfs
> root@dsanb1-coy:~# mount -t btrfs /dev/sdc /srv/osd.0
> root@dsanb1-coy:~# mount | grep btrfs
> /dev/sdc on /srv/osd.0 type btrfs (rw)
>
>
> So - you can see I had to explicitly set the flag to mount "-t btrfs". It seems that when mkcepfs is running the command "mount -t btrfs -o noatime /dev/sdc /srv/osd.0" it is doing so with the effect that the "-t btrfs" was not present in the line, but it is (symptom the same at least). Crazy.
>
> Secondly, as requested I cannot run the command "sh -x /sbin/mkcephfs -d
> /tmp/mkcephfs.xgk025tjkQ --prepare-osdfs osd.0" because the contents of
> the /tmp/ directory are deleted each time mkcephfs finishes its run. I
> have however called the overall mkcephfs command with "sh -x" so you can
> see what is occurring.
Can you modify the mkcephfs command so that when it re-runs itself, it
passes in -x?
You can also remove the 'rm -r' bit that cleans up on error so that you
can run the command manually when it fails.
>
> See below:
>
> root@dsanb1-coy:~# sh -x /sbin/mkcephfs -c /etc/ceph/ceph.conf --allhosts --mkbtrfs -k /etc/ceph/keyring --crushmapsrc crushfile.txt -v
> <SNIP>
> === osd.0 ===
> + return 0
> + [ -n ]
> + rdir=/tmp/mkcephfs.kJjIwsEnfZ
> + [ 0 -eq 0 ]
> + cp /tmp/mkcephfs.kJjIwsEnfZ/conf /etc/ceph/ceph.conf
> + [ 1 -eq 1 ]
> + [ osd = osd ]
> + do_root_cmd /sbin/mkcephfs -d /tmp/mkcephfs.kJjIwsEnfZ --prepare-osdfs osd.0
> + [ -z ]
> + [ 1 -eq 1 ]
> + echo --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.kJjIwsEnfZ --prepare-osdfs osd.0
> --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.kJjIwsEnfZ --prepare-osdfs osd.0
> + ulimit -c unlimited
> + whoami
> + whoami=root
> + [ root = root ]
> + bash -c /sbin/mkcephfs -d /tmp/mkcephfs.kJjIwsEnfZ --prepare-osdfs osd.0
i.e., 'bash -c -x ...' here
> umount: /srv/osd.0: not mounted
> umount: /dev/sdc: not mounted
> /sbin/mkfs.btrfs
> RUNNING: mkfs.btrfs /dev/sdc
>
> WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL
> WARNING! - see http://btrfs.wiki.kernel.org before using
>
> fs created label (null) on /dev/sdc
> nodesize 4096 leafsize 4096 sectorsize 4096 size 1.82TB
> Btrfs Btrfs v0.19
> Scanning for Btrfs filesystems
> /bin/mount
> RUNNING: mount -t btrfs -o noatime /dev/sdc /srv/osd.0
> mount: wrong fs type, bad option, bad superblock on /dev/sdc,
> missing codepage or helper program, or other error
> In some cases useful info is found in syslog - try
> dmesg | tail or so
>
> + echo failed: '/sbin/mkcephfs -d /tmp/mkcephfs.kJjIwsEnfZ --prepare-osdfs osd.0'
> failed: '/sbin/mkcephfs -d /tmp/mkcephfs.kJjIwsEnfZ --prepare-osdfs osd.0'
> + exit 1
> + rm -rf /tmp/mkcephfs.kJjIwsEnfZ
and comment out this command.
sage
> + exit
>
>
>
> -----Original Message-----
> From: Sage Weil [mailto:sage@inktank.com]
> Sent: Saturday, 7 July 2012 2:20 PM
> To: Paul Pettigrew
> Cc: ceph-devel@vger.kernel.org
> Subject: RE: mkcephfs failing on v0.48 "argonaut"
>
> On Sat, 7 Jul 2012, Paul Pettigrew wrote:
> > Hi again Sage
> >
> > This is very perplexing. Confirming this system is a stock Ubuntu 12.04 x64, with no custom kernel or anything else, fully apt-get dist-upgrade'd up to date.
> > root@dsanb1-coy:~# uname -r
> > 3.2.0-26-generic
> >
> > I have added in the suggestions you made to the script, we now have:
> >
> > modprobe btrfs || true
> > which mkfs.btrfs
> > echo "RUNNING: mkfs.btrfs $btrfs_devs"
> > mkfs.btrfs $btrfs_devs
> > btrfs device scan || btrfsctl -a
> > which mount
> > echo "RUNNING: mount -t btrfs $btrfs_opt $first_dev $btrfs_path"
> > mount -t btrfs $btrfs_opt $first_dev $btrfs_path echo "DID I GET
> > HERE - OR CRASH OUT WITH mount ABOVE?"
> > chown $osd_user $btrfs_path
> >
> >
> > See below that the same command within the mkcephfs that is failing,
> > is working fine on a standard command line:
>
> Weirdness!
>
> >
> > === osd.0 ===
> > --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.xgk025tjkQ
> > --prepare-osdfs osd.0
>
> Can you run this command with -x to see exactly what bash is doing?
>
> sh -x /sbin/mkcephfs -d /tmp/mkcephfs.xgk025tjkQ --prepare-osdfs osd.0
>
> In particular, I'm curious if you do
>
> mkfs.btrfs /dev/sdc
> btrfs device scan
> mount /dev/sdc /srv/osd.0
>
> (or whatever the exact sequence that mkcephfs does is) from the command line, does it give you the same error?
>
> sage
>
>
> > umount: /srv/osd.0: not mounted
> > umount: /dev/sdc: not mounted
> > /sbin/mkfs.btrfs
> > RUNNING: mkfs.btrfs /dev/sdc
> >
> > WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL WARNING! - see
> > http://btrfs.wiki.kernel.org before using
> >
> > fs created label (null) on /dev/sdc
> > nodesize 4096 leafsize 4096 sectorsize 4096 size 1.82TB Btrfs
> > Btrfs v0.19 Scanning for Btrfs filesystems
> > /bin/mount
> > RUNNING: mount -t btrfs -o noatime /dev/sdc /srv/osd.0 /bin/mount
> > mount: wrong fs type, bad option, bad superblock on /dev/sdc,
> > missing codepage or helper program, or other error
> > In some cases useful info is found in syslog - try
> > dmesg | tail or so
> >
> > failed: '/sbin/mkcephfs -d /tmp/mkcephfs.xgk025tjkQ --prepare-osdfs osd.0'
> > root@dsanb1-coy:~# /bin/mount -t btrfs -o noatime /dev/sdc /srv/osd.0
> > root@dsanb1-coy:~# mount | grep btrfs /dev/sdc on /srv/osd.0 type
> > btrfs (rw,noatime)
> >
> >
> > Remember, this is not isolated to btrfs, as per my original post it fails when not specifying to use btrfs.
> >
> > I can only conclude that /bin/sh &/or /bin/bash and the way they interact with the mkcephfs script, which does call itself etc, is somehow now become fuddled up? Must be something wiggy, when the script output confirms it is calling the same command ( /bin/mount ) but somehow finds a way for that to not work and therefore cause the mkcephfs script terminate.
> >
> > Many thanks - will be a relief to sort this out, as all our Ceph project works are on hold til we can sort this one out.
> >
> > Cheers
> >
> > Paul
> >
> >
> >
> > -----Original Message-----
> > From: Sage Weil [mailto:sage@inktank.com]
> > Sent: Friday, 6 July 2012 2:09 PM
> > To: Paul Pettigrew
> > Cc: ceph-devel@vger.kernel.org
> > Subject: RE: mkcephfs failing on v0.48 "argonaut"
> >
> > On Fri, 6 Jul 2012, Paul Pettigrew wrote:
> > > Hi Sage - thanks so much for the quick response :-)
> > >
> > > Firstly, and it is a bit hard to see, but the command output below is run with the "-v" option. To help isolate what command line in the script is failing, I have added in some simple echo output, and the script now looks like:
> > >
> > >
> > > ### prepare-osdfs ###
> > >
> > > if [ -n "$prepareosdfs" ]; then
> > > <<SNIP>>
> > > modprobe btrfs || true
> > > echo "RUNNING: mkfs.btrfs $btrfs_devs"
> > > mkfs.btrfs $btrfs_devs
> > > btrfs device scan || btrfsctl -a echo "RUNNING: mount -t btrfs
> > > $btrfs_opt $first_dev $btrfs_path"
> > > mount -t btrfs $btrfs_opt $first_dev $btrfs_path echo "DID I GET
> > > HERE - OR CRASH OUT WITH mount ABOVE?"
> > > chown $osd_user $btrfs_path
> > > chmod +w $btrfs_path
> > >
> > > exit 0
> > > fi
> > >
> > > Per the modified script the above, here is the output displayed when running the script:
> > >
> > > root@dsanb1-coy:/srv# /sbin/mkcephfs -c /etc/ceph/ceph.conf
> > > --allhosts --mkbtrfs -k /etc/ceph/keyring --crushmapsrc
> > > crushfile.txt -v temp dir is /tmp/mkcephfs.uelzdJ82ej preparing
> > > monmap in /tmp/mkcephfs.uelzdJ82ej/monmap /usr/bin/monmaptool
> > > --create --clobber --add alpha 10.32.0.10:6789 --add bravo
> > > 10.32.0.25:6789 --add charlie
> > > 10.32.0.11:6789 --print /tmp/mkcephfs.uelzdJ82ej/monmap
> > > /usr/bin/monmaptool: monmap file /tmp/mkcephfs.uelzdJ82ej/monmap
> > > /usr/bin/monmaptool: generated fsid
> > > b254abdd-e036-4186-b6d5-e32b14e53b45
> > > epoch 0
> > > fsid b254abdd-e036-4186-b6d5-e32b14e53b45
> > > last_changed 2012-07-06 12:31:38.416848 created 2012-07-06
> > > 12:31:38.416848
> > > 0: 10.32.0.10:6789/0 mon.alpha
> > > 1: 10.32.0.11:6789/0 mon.charlie
> > > 2: 10.32.0.25:6789/0 mon.bravo
> > > /usr/bin/monmaptool: writing epoch 0 to
> > > /tmp/mkcephfs.uelzdJ82ej/monmap (3 monitors) /usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.0 "user"
> > > === osd.0 ===
> > > --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.uelzdJ82ej
> > > --prepare-osdfs osd.0
> > > umount: /srv/osd.0: not mounted
> > > umount: /dev/sdc: not mounted
> > > RUNNING: mkfs.btrfs /dev/sdc
> > >
> > > WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL WARNING! - see
> > > http://btrfs.wiki.kernel.org before using
> > >
> > > fs created label (null) on /dev/sdc
> > > nodesize 4096 leafsize 4096 sectorsize 4096 size 1.82TB
> > > Btrfs Btrfs v0.19 Scanning for Btrfs filesystems
> > > RUNNING: mount -t btrfs -o noatime /dev/sdc /srv/osd.0
> > > mount: wrong fs type, bad option, bad superblock on /dev/sdc,
> > > missing codepage or helper program, or other error
> > > In some cases useful info is found in syslog - try
> > > dmesg | tail or so
> > >
> > > failed: '/sbin/mkcephfs -d /tmp/mkcephfs.uelzdJ82ej --prepare-osdfs osd.0'
> > >
> > >
> > > Which clearly isolates the issue to the "mount" command line.
> > >
> > > The trouble is, I can run this precise line on the command line directly without error:
> > >
> > > root@dsanb1-coy:/srv# mount -t btrfs -o noatime /dev/sdc /srv/osd.0
> > > root@dsanb1-coy:/srv# mount | grep btrfs
> > > /dev/sdc on /srv/osd.0 type btrfs (rw,noatime)
> >
> > What if you run the exact sequence of commands that mkcephfs is doing?
> > (mkfs.btrfs, btrfs ..., mount ...). If that doesn't work, put `which mkfs.btfs` etc in the script to make sure you're running the exact version the script is...
> >
> > sage
> >
> >
> >
> > >
> > >
> > > Therefore, what could possibly be preventing the mkcephfs running a simple mount command on the first OSD disk it gets to, that otherwise works fine from the command line?
> > >
> > > Many thanks Sage
> > >
> > > Paul
> > >
> > > PS: changing the " btrfs device scan || btrfsctl -a" line as proposed had no effect, and neither did putting in a "sleep 10" immediately before the mount line.
> > > PPS: zerofilling the /dev/sdc and then re-creating a partition and mounting manually, then writing data to it is all fine. Same errors if we substitute any of the other HDD's in the server as 1st/osd.0. Ie, cannot see any issues with the hardware.
> > >
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: ceph-devel-owner@vger.kernel.org
> > > [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
> > > Sent: Friday, 6 July 2012 8:18 AM
> > > To: Paul Pettigrew
> > > Cc: ceph-devel@vger.kernel.org
> > > Subject: Re: mkcephfs failing on v0.48 "argonaut"
> > >
> > > Hi Paul,
> > >
> > > On Wed, 4 Jul 2012, Paul Pettigrew wrote:
> > > > Firstly, well done guys on achieving this version milestone. I
> > > > successfully upgraded to the 0.48 format uneventfully on a live
> > > > (test) system.
> > > >
> > > > The same system was then going through "rebuild" testing, to
> > > > confirm that also worked fine.
> > > >
> > > >
> > > > Unfortunately, the mkcephfs command is failing:
> > > >
> > > > root@dsanb1-coy:~# mkcephfs -c /etc/ceph/ceph.conf --allhosts
> > > > --mkbtrfs -k /etc/ceph/keyring --crushmapsrc crushfile.txt -v temp
> > > > dir is /tmp/mkcephfs.GaRCZ9i06a preparing monmap in
> > > > /tmp/mkcephfs.GaRCZ9i06a/monmap /usr/bin/monmaptool --create
> > > > --clobber --add alpha 10.32.0.10:6789 --add bravo 10.32.0.25:6789
> > > > --add charlie
> > > > 10.32.0.11:6789 --print /tmp/mkcephfs.GaRCZ9i06a/monmap
> > > > /usr/bin/monmaptool: monmap file /tmp/mkcephfs.GaRCZ9i06a/monmap
> > > > /usr/bin/monmaptool: generated fsid
> > > > c7202495-468c-4678-b678-115c3ee33402
> > > > epoch 0
> > > > fsid c7202495-468c-4678-b678-115c3ee33402
> > > > last_changed 2012-07-04 15:02:31.732275 created 2012-07-04
> > > > 15:02:31.732275
> > > > 0: 10.32.0.10:6789/0 mon.alpha
> > > > 1: 10.32.0.11:6789/0 mon.charlie
> > > > 2: 10.32.0.25:6789/0 mon.bravo
> > > > /usr/bin/monmaptool: writing epoch 0 to
> > > > /tmp/mkcephfs.GaRCZ9i06a/monmap (3 monitors) /usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.0 "user"
> > > > === osd.0 ===
> > > > --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.GaRCZ9i06a
> > > > --prepare-osdfs osd.0
> > > > umount: /srv/osd.0: not mounted
> > > > umount: /dev/disk/by-wwn/wwn-0x50014ee601246234: not mounted
> > > >
> > > > WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL WARNING! - see
> > > > http://btrfs.wiki.kernel.org before using
> > > >
> > > > fs created label (null) on /dev/disk/by-wwn/wwn-0x50014ee601246234
> > > > nodesize 4096 leafsize 4096 sectorsize 4096 size 1.82TB
> > > > Btrfs Btrfs v0.19 Scanning for Btrfs filesystems
> > > > mount: wrong fs type, bad option, bad superblock on /dev/sdc,
> > > > missing codepage or helper program, or other error
> > > > In some cases useful info is found in syslog - try
> > > > dmesg | tail or so
> > > >
> > > > failed: '/sbin/mkcephfs -d /tmp/mkcephfs.GaRCZ9i06a --prepare-osdfs osd.0'
> > >
> > > Hmm. Can you try running with -v? That will tell us exactly which command it is running, and hopefully we can work backwards from there.
> > >
> > > > dmesg/syslog is spitting out at the time of this failure:
> > > >
> > > > Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.751945] device fsid
> > > > 7de0d192-b710-4629-a201-849df1d9db17 devid 1 transid 27109
> > > > /dev/sdp Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.751987] device
> > > > fsid 08fc3479-2fa2-4388-8b61-83e2a742a13e devid 1 transid 28699
> > > > /dev/sdo Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.752023] device
> > > > fsid
> > > > 8b4a7c43-1a05-4dcb-bbed-de2a5c933996 devid 1 transid 24346
> > > > /dev/sdn Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.752068] device
> > > > fsid ba5fb1ca-c642-49b1-8a41-7f56f8e59fbd devid 1 transid 27274
> > > > /dev/sdm Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761453] device
> > > > fsid 7fe8c5cf-bf8c-4276-90f2-c3f57f5275fb devid 1 transid 28724
> > > > /dev/sdi Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761518] device
> > > > fsid 93fa3631-1202-4d42-8908-e5ef4d3e600d devid 1 transid 25201
> > > > /dev/sdh Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761579] device
> > > > fsid b9a1b5e4-3e5e-4381-a29a-33470f4b870f devid 1 transid 23375
> > > > /dev/sdg Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761635] device
> > > > fsid 280ea990-23f8-4c43-9e56-140c82340fdc devid 1 transid 25559
> > > > /dev/sdf Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.761693] device
> > > > fsid 2f724cde-6de5-4262-b195-1ba3eea2256e devid 1 transid 176
> > > > /dev/sde Jul
> > > > 4 15:02:31 dsanb1-coy kernel: [ 2306.761732] device fsid
> > > > a66f890f-8b08-4393-aab0-f222637ca5a4 devid 1 transid 7 /dev/sdd
> > > > Jul
> > > > 4
> > > > 15:02:31 dsanb1-coy kernel: [ 2306.761769] device fsid
> > > > 6c181a94-697c-4e0c-af0d-05eb04d3626c devid 1 transid 7 /dev/sdc
> > > > Jul
> > > > 4
> > > > 15:02:31 dsanb1-coy kernel: [ 2306.775931] device fsid
> > > > 6c181a94-697c-4e0c-af0d-05eb04d3626c devid 1 transid 7 /dev/sdc
> > > > Jul
> > > > 4
> > > > 15:02:31 dsanb1-coy kernel: [ 2306.779716] btrfs bad fsid on block
> > > > 20971520 Jul 4 15:02:31 dsanb1-coy kernel: [ 2306.791594] btrfs
> > > > bad fsid on block 20971520 Jul 4 15:02:31 dsanb1-coy kernel: [
> > > > 2306.803608] btrfs bad fsid on block 20971520 Jul 4 15:02:31
> > > > dsanb1-coy kernel: [ 2306.815541] btrfs bad fsid on block 20971520
> > > > Jul
> > > > 4 15:02:31 dsanb1-coy kernel: [ 2306.815878] btrfs bad fsid on
> > > > block
> > > > 20971520 Jul 4 15:02:32 dsanb1-coy kernel: [ 2306.823554] btrfs
> > > > bad fsid on block 20971520 Jul 4 15:02:32 dsanb1-coy kernel: [
> > > > 2306.823797] btrfs bad fsid on block 20971520 Jul 4 15:02:32
> > > > dsanb1-coy kernel: [ 2306.823887] btrfs: failed to read chunk root
> > > > on sdc Jul 4 15:02:32 dsanb1-coy kernel: [ 2306.825622] btrfs:
> > > > open_ctree failed
> > >
> > > Long shot, but is the kernel on that machine recent?
> > >
> > > > Also fails if not forcing to use btrfs, eg:
> > > >
> > > > root@dsanb1-coy:~# mkcephfs -c /etc/ceph/ceph.conf --allhosts -k
> > > > /etc/ceph/keyring --crushmapsrc crushfile.txt -v temp dir is
> > > > /tmp/mkcephfs.ZOh6tBPAH0 preparing monmap in
> > > > /tmp/mkcephfs.ZOh6tBPAH0/monmap /usr/bin/monmaptool --create
> > > > --clobber --add alpha 10.32.0.10:6789 --add bravo 10.32.0.25:6789
> > > > --add charlie
> > > > 10.32.0.11:6789 --print /tmp/mkcephfs.ZOh6tBPAH0/monmap
> > > > /usr/bin/monmaptool: monmap file /tmp/mkcephfs.ZOh6tBPAH0/monmap
> > > > /usr/bin/monmaptool: generated fsid
> > > > adb8d65c-a823-4dc2-9415-22b0d7252699
> > > > epoch 0
> > > > fsid adb8d65c-a823-4dc2-9415-22b0d7252699
> > > > last_changed 2012-07-04 15:04:17.423368 created 2012-07-04
> > > > 15:04:17.423368
> > > > 0: 10.32.0.10:6789/0 mon.alpha
> > > > 1: 10.32.0.11:6789/0 mon.charlie
> > > > 2: 10.32.0.25:6789/0 mon.bravo
> > > > /usr/bin/monmaptool: writing epoch 0 to
> > > > /tmp/mkcephfs.ZOh6tBPAH0/monmap (3 monitors) /usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.0 "user"
> > > > === osd.0 ===
> > > > --- dsanb1-coy# /sbin/mkcephfs -d /tmp/mkcephfs.ZOh6tBPAH0
> > > > --init-daemon osd.0
> > > > 2012-07-04 15:04:17.789064 7fc7fadca780 -1 filestore(/srv/osd.0)
> > > > limited size xattrs -- enable filestore_xattr_use_omap
> > > > 2012-07-04 15:04:17.789120 7fc7fadca780 -1 OSD::mkfs: couldn't
> > > > mount
> > > > FileStore: error -95
> > > > 2012-07-04 15:04:17.789161 7fc7fadca780 -1 ** ERROR: error
> > > > creating empty object store in /srv/osd.0: (95) Operation not
> > > > supported
> > > > failed: '/sbin/mkcephfs -d /tmp/mkcephfs.ZOh6tBPAH0 --init-daemon osd.0'
> > > >
> > > >
> > > > Confirming all this was working previously, and the crushmap,
> > > > config file, etc are all proven to be OK (get same failure when
> > > > not specifying a custom crushmap also). Also note that whilst the
> > > > above is failing on
> > > > osd.0 creation, I have swapped disk references and still get the
> > > > same failure on different HDD's when they are hooked in as osd.0
> > >
> > > The only thing that changed from v0.47 is the below. Can you try replacing 'btrfs device scan || btrfsctl -a' with 'btrfs device scan ; btrfsctl -a'? Maybe the btrfs tool isn't being pendantic about return codes...
> > >
> > > sage
> > >
> > >
> > > commit a414fd51c7c5ae5dbe9e3af7db6f17741a58c1a7
> > > Author: Sage Weil <sage.weil@dreamhost.com>
> > > Date: Sat Feb 11 13:43:23 2012 -0800
> > >
> > > init-ceph, mkcephfs: try 'btrfs device scan' before 'btrfsctl -a'
> > >
> > > Fixes: #2023
> > > Reported-by: Wido den Hollander <wido@widodh.nl>
> > > Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
> > >
> > > diff --git a/src/mkcephfs.in b/src/mkcephfs.in index
> > > 83fb932..17b6014
> > > 100644
> > > --- a/src/mkcephfs.in
> > > +++ b/src/mkcephfs.in
> > > @@ -332,7 +332,7 @@ if [ -n "$prepareosdfs" ]; then
> > >
> > > modprobe btrfs || true
> > > mkfs.btrfs $btrfs_devs
> > > - btrfsctl -a
> > > + btrfs device scan || btrfsctl -a
> > > mount -t btrfs $btrfs_opt $first_dev $btrfs_path
> > > chown $osd_user $btrfs_path
> > > chmod +w $btrfs_path
> > >
> > >
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > > in the body of a message to majordomo@vger.kernel.org More majordomo
> > > info at http://vger.kernel.org/majordomo-info.html
> > >
> > >
> > >
> >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > in the body of a message to majordomo@vger.kernel.org More majordomo
> > info at http://vger.kernel.org/majordomo-info.html
> >
> >
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2012-07-10 23:34 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-07-04 5:07 mkcephfs failing on v0.48 "argonaut" Paul Pettigrew
2012-07-05 22:17 ` Sage Weil
2012-07-06 2:39 ` Paul Pettigrew
2012-07-06 4:08 ` Sage Weil
2012-07-07 3:33 ` Paul Pettigrew
2012-07-07 3:45 ` Paul Pettigrew
2012-07-07 4:20 ` Sage Weil
2012-07-07 21:15 ` Paul Pettigrew
2012-07-10 23:34 ` Sage Weil
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.