* RBD stale on VM, and RBD cache enable problem
@ 2012-06-11 19:53 Sławomir Skowron
2012-07-03 17:39 ` Gregory Farnum
0 siblings, 1 reply; 3+ messages in thread
From: Sławomir Skowron @ 2012-06-11 19:53 UTC (permalink / raw)
To: ceph-devel
[-- Attachment #1: Type: text/plain, Size: 10726 bytes --]
I have two questions. My newly created cluster with xfs on all osd,
ubuntu precise, kernel 3.2.0-23-generic. Ceph 0.47.2-1precise
pool 0 'data' rep size 3 crush_ruleset 0 object_hash rjenkins pg_num
64 pgp_num 64 last_change 1228 owner 0 crash_replay_interval 45
pool 1 'metadata' rep size 3 crush_ruleset 1 object_hash rjenkins
pg_num 64 pgp_num 64 last_change 1226 owner 0
pool 2 'rbd' rep size 3 crush_ruleset 2 object_hash rjenkins pg_num 64
pgp_num 64 last_change 1232 owner 0
pool 3 '.rgw' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 8
pgp_num 8 last_change 3878 owner 18446744073709551615
1. After i stop all daemons on 1 machine in my 3 node cluster with 3
replicas, rbd image operations on vm, staling. DD on this device in VM
freezing, and after ceph start on this machine everything goes online.
Is there any problem with my config ?? in this situation ceph should
go from another copies with reads, and writes into another osd in
replica chain, yes ??
Another test iozone on device, and it's stop after daemons stop on 1
machine, and after osd up, iozone go forward, how can i tune this to
work without freeze ??
2012-06-11 21:38:49.583133 pg v88173: 200 pgs: 60 active+clean, 1
stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
2012-06-11 21:38:50.582257 pg v88174: 200 pgs: 60 active+clean, 1
stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
.....
2012-06-11 21:39:49.991893 pg v88197: 200 pgs: 60 active+clean, 1
stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
2012-06-11 21:39:50.992755 pg v88198: 200 pgs: 60 active+clean, 1
stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
2012-06-11 21:39:51.993533 pg v88199: 200 pgs: 60 active+clean, 1
stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
2012-06-11 21:39:52.994397 pg v88200: 200 pgs: 60 active+clean, 1
stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
After boot all osd on stoped machine:
2012-06-11 21:40:37.826619 osd e4162: 72 osds: 53 up, 72 in
2012-06-11 21:40:37.825706 mon.0 10.177.66.4:6790/0 348 : [INF] osd.24
10.177.66.6:6800/21597 boot
2012-06-11 21:40:38.825297 pg v88202: 200 pgs: 54 active+clean, 7
stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
2012-06-11 21:40:38.826517 osd e4163: 72 osds: 54 up, 72 in
2012-06-11 21:40:38.825250 mon.0 10.177.66.4:6790/0 349 : [INF] osd.25
10.177.66.6:6803/21712 boot
2012-06-11 21:40:38.825655 mon.0 10.177.66.4:6790/0 350 : [INF] osd.28
10.177.66.6:6812/26210 boot
2012-06-11 21:40:38.825907 mon.0 10.177.66.4:6790/0 351 : [INF] osd.29
10.177.66.6:6815/26327 boot
2012-06-11 21:40:39.826738 pg v88203: 200 pgs: 56 active+clean, 4
stale+active+clean, 3 peering, 137 active+degraded; 783 GB data, 1928
GB used, 18111 GB / 20040 GB avail; 76921/254952 degraded (30.171%)
2012-06-11 21:40:39.830098 osd e4164: 72 osds: 59 up, 72 in
2012-06-11 21:40:39.826570 mon.0 10.177.66.4:6790/0 352 : [INF] osd.26
10.177.66.6:6806/21835 boot
2012-06-11 21:40:39.826961 mon.0 10.177.66.4:6790/0 353 : [INF] osd.27
10.177.66.6:6809/21953 boot
2012-06-11 21:40:39.828147 mon.0 10.177.66.4:6790/0 354 : [INF] osd.30
10.177.66.6:6818/26511 boot
2012-06-11 21:40:39.828418 mon.0 10.177.66.4:6790/0 355 : [INF] osd.31
10.177.66.6:6821/26583 boot
2012-06-11 21:40:39.828935 mon.0 10.177.66.4:6790/0 356 : [INF] osd.33
10.177.66.6:6827/26859 boot
2012-06-11 21:40:39.829274 mon.0 10.177.66.4:6790/0 357 : [INF] osd.34
10.177.66.6:6830/26979 boot
2012-06-11 21:40:40.827935 pg v88204: 200 pgs: 56 active+clean, 4
stale+active+clean, 3 peering, 137 active+degraded; 783 GB data, 1928
GB used, 18111 GB / 20040 GB avail; 76921/254952 degraded (30.171%)
2012-06-11 21:40:40.830059 osd e4165: 72 osds: 62 up, 72 in
2012-06-11 21:40:40.827798 mon.0 10.177.66.4:6790/0 358 : [INF] osd.32
10.177.66.6:6824/26701 boot
2012-06-11 21:40:40.829043 mon.0 10.177.66.4:6790/0 359 : [INF] osd.35
10.177.66.6:6833/27165 boot
2012-06-11 21:40:40.829316 mon.0 10.177.66.4:6790/0 360 : [INF] osd.36
10.177.66.6:6836/27280 boot
2012-06-11 21:40:40.829602 mon.0 10.177.66.4:6790/0 361 : [INF] osd.37
10.177.66.6:6839/27397 boot
2012-06-11 21:40:41.828776 pg v88205: 200 pgs: 56 active+clean, 4
stale+active+clean, 3 peering, 137 active+degraded; 783 GB data, 1928
GB used, 18111 GB / 20040 GB avail; 76921/254952 degraded (30.171%)
2012-06-11 21:40:41.831823 osd e4166: 72 osds: 68 up, 72 in
2012-06-11 21:40:41.828713 mon.0 10.177.66.4:6790/0 362 : [INF] osd.38
10.177.66.6:6842/27513 boot
2012-06-11 21:40:41.829440 mon.0 10.177.66.4:6790/0 363 : [INF] osd.39
10.177.66.6:6845/27628 boot
2012-06-11 21:40:41.830226 mon.0 10.177.66.4:6790/0 364 : [INF] osd.40
10.177.66.6:6848/27835 boot
2012-06-11 21:40:41.830531 mon.0 10.177.66.4:6790/0 365 : [INF] osd.41
10.177.66.6:6851/27950 boot
2012-06-11 21:40:41.830778 mon.0 10.177.66.4:6790/0 366 : [INF] osd.42
10.177.66.6:6854/28065 boot
2012-06-11 21:40:41.831249 mon.0 10.177.66.4:6790/0 367 : [INF] osd.43
10.177.66.6:6857/28181 boot
2012-06-11 21:40:42.830440 pg v88206: 200 pgs: 57 active+clean, 4
stale+active+clean, 7 peering, 132 active+degraded; 783 GB data, 1928
GB used, 18111 GB / 20040 GB avail; 75543/254952 degraded (29.630%)
2012-06-11 21:40:42.833294 osd e4167: 72 osds: 72 up, 72 in
2012-06-11 21:40:42.831046 mon.0 10.177.66.4:6790/0 368 : [INF] osd.44
10.177.66.6:6860/28373 boot
2012-06-11 21:40:42.832004 mon.0 10.177.66.4:6790/0 369 : [INF] osd.45
10.177.66.6:6863/28489 boot
2012-06-11 21:40:42.832314 mon.0 10.177.66.4:6790/0 370 : [INF] osd.46
10.177.66.6:6866/28607 boot
2012-06-11 21:40:42.832545 mon.0 10.177.66.4:6790/0 371 : [INF] osd.47
10.177.66.6:6869/28731 boot
2012-06-11 21:40:43.830481 pg v88207: 200 pgs: 64 active+clean, 4
stale+active+clean, 7 peering, 125 active+degraded; 783 GB data, 1928
GB used, 18111 GB / 20040 GB avail; 72874/254952 degraded (28.583%)
2012-06-11 21:40:43.831113 osd e4168: 72 osds: 72 up, 72 in
2012-06-11 21:40:44.832521 pg v88208: 200 pgs: 79 active+clean, 1
stale+active+clean, 4 peering, 113 active+degraded, 3
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail; 66185/254952 degraded (25.960%)
2012-06-11 21:40:45.834077 pg v88209: 200 pgs: 104 active+clean, 1
stale+active+clean, 4 peering, 85 active+degraded, 6
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail; 50399/254952 degraded (19.768%)
2012-06-11 21:40:46.835367 pg v88210: 200 pgs: 125 active+clean, 1
stale+active+clean, 4 peering, 59 active+degraded, 11
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail; 38563/254952 degraded (15.126%)
2012-06-11 21:40:47.836516 pg v88211: 200 pgs: 158 active+clean, 1
stale+active+clean, 26 active+degraded, 15 active+recovering; 783 GB
data, 1928 GB used, 18111 GB / 20040 GB avail; 18542/254952 degraded
(7.273%)
2012-06-11 21:40:48.853560 pg v88212: 200 pgs: 184 active+clean, 16
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail; 1/254952 degraded (0.000%)
2012-06-11 21:40:49.868514 pg v88213: 200 pgs: 184 active+clean, 16
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail; 1/254952 degraded (0.000%)
2012-06-11 21:40:50.858244 pg v88214: 200 pgs: 184 active+clean, 16
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail; 1/254952 degraded (0.000%)
2012-06-11 21:40:51.845622 pg v88215: 200 pgs: 184 active+clean, 16
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail
2012-06-11 21:40:52.857823 pg v88216: 200 pgs: 184 active+clean, 16
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail
2012-06-11 21:40:53.858281 pg v88217: 200 pgs: 184 active+clean, 16
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail
2012-06-11 21:40:54.855602 pg v88218: 200 pgs: 184 active+clean, 16
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail
2012-06-11 21:40:55.857241 pg v88219: 200 pgs: 184 active+clean, 16
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail
2012-06-11 21:40:56.857631 pg v88220: 200 pgs: 184 active+clean, 16
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail
2012-06-11 21:40:57.858987 pg v88221: 200 pgs: 185 active+clean, 15
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail
2012-06-11 21:40:58.880252 pg v88222: 200 pgs: 185 active+clean, 15
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail
2012-06-11 21:40:59.861910 pg v88223: 200 pgs: 188 active+clean, 12
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail
2012-06-11 21:41:00.902582 pg v88224: 200 pgs: 191 active+clean, 9
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail
2012-06-11 21:41:01.907767 pg v88225: 200 pgs: 196 active+clean, 4
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail
2012-06-11 21:41:02.876377 pg v88226: 200 pgs: 199 active+clean, 1
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail
2012-06-11 21:41:03.876929 pg v88227: 200 pgs: 200 active+clean;
783 GB data, 1928 GB used, 18111 GB / 20040 GB avail
<disk type="network" device="disk">
<driver name="qemu" type="raw"/>
<source protocol="rbd" name="rbd/foo4">
</source>
<target dev="vdf" bus="virtio"/>
</disk>
2. When i use rbd_cache=1, or true in my xml, for libvirt i get:
<disk type="network" device="disk">
<driver name="qemu" type="raw"/>
<source protocol="rbd" name="rbd/foo5:rbd_cache=1">
</source>
<target dev="vdf" bus="virtio"/>
</disk>
libvirtd.log
2012-06-11 18:50:36.992+0000: 1751: error :
qemuMonitorTextAddDrive:2820 : operation failed: open disk image file
failed
Libvirt version 0.9.8-2ubuntu17 with some additional patch set before
ceph 0.46 version appears. Qemu-kvm 1.0+noroms-0ubuntu13.
Do i need any other patch for libvirt ?? Without rbd_cache attaching is ok.
--
-----
Pozdrawiam
Sławek "sZiBis" Skowron
[-- Attachment #2: ceph.conf.txt --]
[-- Type: text/plain, Size: 5753 bytes --]
; global
[global]
; enable secure authentication
auth supported = cephx
keyring = /etc/ceph/$cluster.keyring
rgw_cache_enabled = false ;rgw cache enabled
rgw_cache_lru_size = 131072 ;num of entries in rgw cache
rgw_thread_pool_size = 4096
rgw print continue = false ;enable if 100-Continue works
rgw_enable_ops_log = true ;enable logging every rgw operation
rgw socket path = /var/run/radosgw.sock
debug rgw = 1
rbd cache = true
rbd cache max dirty = 0
admin_socket = /var/run/ceph/$cluster-$name.asok
; radosgw client list
[client.radosgw.obs-10-177-66-4]
host = obs-10-177-66-4
keyring = /etc/ceph/client.radosgw.obs-10-177-66-4.bin
log file = /var/log/radosgw/$name.log
debug ms = 1
[client.radosgw.obs-10-177-66-6]
host = obs-10-177-66-6
keyring = /etc/ceph/client.radosgw.obs-10-177-66-6.bin
log file = /var/log/radosgw/$name.log
debug ms = 1
[client.radosgw.obs-10-177-66-8]
host = obs-10-177-66-8
keyring = /etc/ceph/client.radosgw.obs-10-177-66-8.bin
log file = /var/log/radosgw/$name.log
debug ms = 1
; monitors
; You need at least one. You need at least three if you want to
; tolerate any node failures. Always create an odd number.
[mon]
mon data = /vol0/data/mon.$id
; some minimal logging (just message traffic) to aid debugging
debug ms = 0 ; see message traffic
debug mon = 0 ; monitor
debug paxos = 0 ; monitor replication
debug auth = 0 ;
mon allowed clock drift = 2
[mon.0]
host = obs-10-177-66-4
mon addr = 10.177.66.4:6790
; osd
; You need at least one. Two if you want data to be replicated.
; Define as many as you like.
[osd]
; This is where the btrfs volume will be mounted.
osd data = /vol0/data/osd.$id
; Ideally, make this a separate disk or partition. A few GB
; is usually enough; more if you have fast disks. You can use
; a file under the osd data dir if need be
; (e.g. /data/osd$id/journal), but it will be slower than a
; separate disk or partition.
osd journal = /vol0/data/osd.$id/journal
; If the OSD journal is a file, you need to specify the size. This is specified in MB.
keyring = /vol0/data/osd.$id/keyring
osd journal size = 1024
; filestore_xattr_use_omap = 0
filestore journal writeahead = 1
osd heartbeat grace = 8
debug ms = 0 ; message traffic
debug osd = 1
debug filestore = 1 ; local object storage
debug journal = 0 ; local journaling
debug monc = 0
debug rados = 0
[osd.0]
host = obs-10-177-66-4
[osd.1]
host = obs-10-177-66-4
[osd.10]
host = obs-10-177-66-4
[osd.11]
host = obs-10-177-66-4
[osd.12]
host = obs-10-177-66-4
[osd.13]
host = obs-10-177-66-4
[osd.14]
host = obs-10-177-66-4
[osd.15]
host = obs-10-177-66-4
[osd.16]
host = obs-10-177-66-4
[osd.17]
host = obs-10-177-66-4
[osd.18]
host = obs-10-177-66-4
[osd.19]
host = obs-10-177-66-4
[osd.2]
host = obs-10-177-66-4
[osd.20]
host = obs-10-177-66-4
[osd.21]
host = obs-10-177-66-4
[osd.22]
host = obs-10-177-66-4
[osd.23]
host = obs-10-177-66-4
[osd.24]
host = obs-10-177-66-6
[osd.25]
host = obs-10-177-66-6
[osd.26]
host = obs-10-177-66-6
[osd.27]
host = obs-10-177-66-6
[osd.28]
host = obs-10-177-66-6
[osd.29]
host = obs-10-177-66-6
[osd.3]
host = obs-10-177-66-4
[osd.30]
host = obs-10-177-66-6
[osd.31]
host = obs-10-177-66-6
[osd.32]
host = obs-10-177-66-6
[osd.33]
host = obs-10-177-66-6
[osd.34]
host = obs-10-177-66-6
[osd.35]
host = obs-10-177-66-6
[osd.36]
host = obs-10-177-66-6
[osd.37]
host = obs-10-177-66-6
[osd.38]
host = obs-10-177-66-6
[osd.39]
host = obs-10-177-66-6
[osd.4]
host = obs-10-177-66-4
[osd.40]
host = obs-10-177-66-6
[osd.41]
host = obs-10-177-66-6
[osd.42]
host = obs-10-177-66-6
[osd.43]
host = obs-10-177-66-6
[osd.44]
host = obs-10-177-66-6
[osd.45]
host = obs-10-177-66-6
[osd.46]
host = obs-10-177-66-6
[osd.47]
host = obs-10-177-66-6
[osd.48]
host = obs-10-177-66-8
[osd.49]
host = obs-10-177-66-8
[osd.5]
host = obs-10-177-66-4
[osd.50]
host = obs-10-177-66-8
[osd.51]
host = obs-10-177-66-8
[osd.52]
host = obs-10-177-66-8
[osd.53]
host = obs-10-177-66-8
[osd.54]
host = obs-10-177-66-8
[osd.55]
host = obs-10-177-66-8
[osd.56]
host = obs-10-177-66-8
[osd.57]
host = obs-10-177-66-8
[osd.58]
host = obs-10-177-66-8
[osd.59]
host = obs-10-177-66-8
[osd.6]
host = obs-10-177-66-4
[osd.60]
host = obs-10-177-66-8
[osd.61]
host = obs-10-177-66-8
[osd.62]
host = obs-10-177-66-8
[osd.63]
host = obs-10-177-66-8
[osd.64]
host = obs-10-177-66-8
[osd.65]
host = obs-10-177-66-8
[osd.66]
host = obs-10-177-66-8
[osd.67]
host = obs-10-177-66-8
[osd.68]
host = obs-10-177-66-8
[osd.69]
host = obs-10-177-66-8
[osd.7]
host = obs-10-177-66-4
[osd.70]
host = obs-10-177-66-8
[osd.71]
host = obs-10-177-66-8
[osd.8]
host = obs-10-177-66-4
[osd.9]
host = obs-10-177-66-4
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: RBD stale on VM, and RBD cache enable problem
2012-06-11 19:53 RBD stale on VM, and RBD cache enable problem Sławomir Skowron
@ 2012-07-03 17:39 ` Gregory Farnum
2012-07-04 9:28 ` Sławomir Skowron
0 siblings, 1 reply; 3+ messages in thread
From: Gregory Farnum @ 2012-07-03 17:39 UTC (permalink / raw)
To: Sławomir Skowron; +Cc: ceph-devel
On Mon, Jun 11, 2012 at 12:53 PM, Sławomir Skowron <szibis@gmail.com> wrote:
> I have two questions. My newly created cluster with xfs on all osd,
> ubuntu precise, kernel 3.2.0-23-generic. Ceph 0.47.2-1precise
>
> pool 0 'data' rep size 3 crush_ruleset 0 object_hash rjenkins pg_num
> 64 pgp_num 64 last_change 1228 owner 0 crash_replay_interval 45
> pool 1 'metadata' rep size 3 crush_ruleset 1 object_hash rjenkins
> pg_num 64 pgp_num 64 last_change 1226 owner 0
> pool 2 'rbd' rep size 3 crush_ruleset 2 object_hash rjenkins pg_num 64
> pgp_num 64 last_change 1232 owner 0
> pool 3 '.rgw' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 8
> pgp_num 8 last_change 3878 owner 18446744073709551615
>
> 1. After i stop all daemons on 1 machine in my 3 node cluster with 3
> replicas, rbd image operations on vm, staling. DD on this device in VM
> freezing, and after ceph start on this machine everything goes online.
> Is there any problem with my config ?? in this situation ceph should
> go from another copies with reads, and writes into another osd in
> replica chain, yes ??
It should switch to a new "primary" OSD as soon as they detect that
one machine is missing, which by default will be ~25 seconds. How long
did you wait to see if it would continue?
If you'd like to reduce this time, you can turn down some combination of
osd_heartbeat_grace — default 20 seconds, and controls how long an OSD
will wait before it decides a peer is down.
osd_min_down_reporters — default 1, controls how many OSDs need to
report an OSD as down before accepting it. This is already as low as
it should go
osd_min_down_reports — default 3, controls how many failure reports
the monitor needs to receive before accepting an OSD as down. Since
you only have 3 OSDs, and one is down, leaving this at 3 means you're
going to wait for osd_heartbeat_grace plus osd_mon_report_interval_min
(default 5; don't change this) before an OSD is marked down.
Given the logging you include I'm a little concerned that you have 1
PG "stale", indicating that the monitor hasn't gotten a report on that
PG in a very long time. That means either that one PG is somehow
broken, or else that the OSD you turned off isn't getting marked down
and that PG is the only one noticing it.
Could you re-run this test with monitor debugging turned up, see how
long it takes for the OSD to get marked down (using "ceph -w"), and
report back?
-Greg
> Another test iozone on device, and it's stop after daemons stop on 1
> machine, and after osd up, iozone go forward, how can i tune this to
> work without freeze ??
>
> 2012-06-11 21:38:49.583133 pg v88173: 200 pgs: 60 active+clean, 1
> stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
> 18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
> 2012-06-11 21:38:50.582257 pg v88174: 200 pgs: 60 active+clean, 1
> stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
> 18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
> .....
> 2012-06-11 21:39:49.991893 pg v88197: 200 pgs: 60 active+clean, 1
> stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
> 18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
> 2012-06-11 21:39:50.992755 pg v88198: 200 pgs: 60 active+clean, 1
> stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
> 18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
> 2012-06-11 21:39:51.993533 pg v88199: 200 pgs: 60 active+clean, 1
> stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
> 18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
> 2012-06-11 21:39:52.994397 pg v88200: 200 pgs: 60 active+clean, 1
> stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
> 18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
>
> After boot all osd on stoped machine:
>
> 2012-06-11 21:40:37.826619 osd e4162: 72 osds: 53 up, 72 in
> 2012-06-11 21:40:37.825706 mon.0 10.177.66.4:6790/0 348 : [INF] osd.24
> 10.177.66.6:6800/21597 boot
> 2012-06-11 21:40:38.825297 pg v88202: 200 pgs: 54 active+clean, 7
> stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
> 18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
> 2012-06-11 21:40:38.826517 osd e4163: 72 osds: 54 up, 72 in
> 2012-06-11 21:40:38.825250 mon.0 10.177.66.4:6790/0 349 : [INF] osd.25
> 10.177.66.6:6803/21712 boot
> 2012-06-11 21:40:38.825655 mon.0 10.177.66.4:6790/0 350 : [INF] osd.28
> 10.177.66.6:6812/26210 boot
> 2012-06-11 21:40:38.825907 mon.0 10.177.66.4:6790/0 351 : [INF] osd.29
> 10.177.66.6:6815/26327 boot
> 2012-06-11 21:40:39.826738 pg v88203: 200 pgs: 56 active+clean, 4
> stale+active+clean, 3 peering, 137 active+degraded; 783 GB data, 1928
> GB used, 18111 GB / 20040 GB avail; 76921/254952 degraded (30.171%)
> 2012-06-11 21:40:39.830098 osd e4164: 72 osds: 59 up, 72 in
> 2012-06-11 21:40:39.826570 mon.0 10.177.66.4:6790/0 352 : [INF] osd.26
> 10.177.66.6:6806/21835 boot
> 2012-06-11 21:40:39.826961 mon.0 10.177.66.4:6790/0 353 : [INF] osd.27
> 10.177.66.6:6809/21953 boot
> 2012-06-11 21:40:39.828147 mon.0 10.177.66.4:6790/0 354 : [INF] osd.30
> 10.177.66.6:6818/26511 boot
> 2012-06-11 21:40:39.828418 mon.0 10.177.66.4:6790/0 355 : [INF] osd.31
> 10.177.66.6:6821/26583 boot
> 2012-06-11 21:40:39.828935 mon.0 10.177.66.4:6790/0 356 : [INF] osd.33
> 10.177.66.6:6827/26859 boot
> 2012-06-11 21:40:39.829274 mon.0 10.177.66.4:6790/0 357 : [INF] osd.34
> 10.177.66.6:6830/26979 boot
> 2012-06-11 21:40:40.827935 pg v88204: 200 pgs: 56 active+clean, 4
> stale+active+clean, 3 peering, 137 active+degraded; 783 GB data, 1928
> GB used, 18111 GB / 20040 GB avail; 76921/254952 degraded (30.171%)
> 2012-06-11 21:40:40.830059 osd e4165: 72 osds: 62 up, 72 in
> 2012-06-11 21:40:40.827798 mon.0 10.177.66.4:6790/0 358 : [INF] osd.32
> 10.177.66.6:6824/26701 boot
> 2012-06-11 21:40:40.829043 mon.0 10.177.66.4:6790/0 359 : [INF] osd.35
> 10.177.66.6:6833/27165 boot
> 2012-06-11 21:40:40.829316 mon.0 10.177.66.4:6790/0 360 : [INF] osd.36
> 10.177.66.6:6836/27280 boot
> 2012-06-11 21:40:40.829602 mon.0 10.177.66.4:6790/0 361 : [INF] osd.37
> 10.177.66.6:6839/27397 boot
> 2012-06-11 21:40:41.828776 pg v88205: 200 pgs: 56 active+clean, 4
> stale+active+clean, 3 peering, 137 active+degraded; 783 GB data, 1928
> GB used, 18111 GB / 20040 GB avail; 76921/254952 degraded (30.171%)
> 2012-06-11 21:40:41.831823 osd e4166: 72 osds: 68 up, 72 in
> 2012-06-11 21:40:41.828713 mon.0 10.177.66.4:6790/0 362 : [INF] osd.38
> 10.177.66.6:6842/27513 boot
> 2012-06-11 21:40:41.829440 mon.0 10.177.66.4:6790/0 363 : [INF] osd.39
> 10.177.66.6:6845/27628 boot
> 2012-06-11 21:40:41.830226 mon.0 10.177.66.4:6790/0 364 : [INF] osd.40
> 10.177.66.6:6848/27835 boot
> 2012-06-11 21:40:41.830531 mon.0 10.177.66.4:6790/0 365 : [INF] osd.41
> 10.177.66.6:6851/27950 boot
> 2012-06-11 21:40:41.830778 mon.0 10.177.66.4:6790/0 366 : [INF] osd.42
> 10.177.66.6:6854/28065 boot
> 2012-06-11 21:40:41.831249 mon.0 10.177.66.4:6790/0 367 : [INF] osd.43
> 10.177.66.6:6857/28181 boot
> 2012-06-11 21:40:42.830440 pg v88206: 200 pgs: 57 active+clean, 4
> stale+active+clean, 7 peering, 132 active+degraded; 783 GB data, 1928
> GB used, 18111 GB / 20040 GB avail; 75543/254952 degraded (29.630%)
> 2012-06-11 21:40:42.833294 osd e4167: 72 osds: 72 up, 72 in
> 2012-06-11 21:40:42.831046 mon.0 10.177.66.4:6790/0 368 : [INF] osd.44
> 10.177.66.6:6860/28373 boot
> 2012-06-11 21:40:42.832004 mon.0 10.177.66.4:6790/0 369 : [INF] osd.45
> 10.177.66.6:6863/28489 boot
> 2012-06-11 21:40:42.832314 mon.0 10.177.66.4:6790/0 370 : [INF] osd.46
> 10.177.66.6:6866/28607 boot
> 2012-06-11 21:40:42.832545 mon.0 10.177.66.4:6790/0 371 : [INF] osd.47
> 10.177.66.6:6869/28731 boot
> 2012-06-11 21:40:43.830481 pg v88207: 200 pgs: 64 active+clean, 4
> stale+active+clean, 7 peering, 125 active+degraded; 783 GB data, 1928
> GB used, 18111 GB / 20040 GB avail; 72874/254952 degraded (28.583%)
> 2012-06-11 21:40:43.831113 osd e4168: 72 osds: 72 up, 72 in
> 2012-06-11 21:40:44.832521 pg v88208: 200 pgs: 79 active+clean, 1
> stale+active+clean, 4 peering, 113 active+degraded, 3
> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
> avail; 66185/254952 degraded (25.960%)
> 2012-06-11 21:40:45.834077 pg v88209: 200 pgs: 104 active+clean, 1
> stale+active+clean, 4 peering, 85 active+degraded, 6
> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
> avail; 50399/254952 degraded (19.768%)
> 2012-06-11 21:40:46.835367 pg v88210: 200 pgs: 125 active+clean, 1
> stale+active+clean, 4 peering, 59 active+degraded, 11
> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
> avail; 38563/254952 degraded (15.126%)
> 2012-06-11 21:40:47.836516 pg v88211: 200 pgs: 158 active+clean, 1
> stale+active+clean, 26 active+degraded, 15 active+recovering; 783 GB
> data, 1928 GB used, 18111 GB / 20040 GB avail; 18542/254952 degraded
> (7.273%)
> 2012-06-11 21:40:48.853560 pg v88212: 200 pgs: 184 active+clean, 16
> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
> avail; 1/254952 degraded (0.000%)
> 2012-06-11 21:40:49.868514 pg v88213: 200 pgs: 184 active+clean, 16
> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
> avail; 1/254952 degraded (0.000%)
> 2012-06-11 21:40:50.858244 pg v88214: 200 pgs: 184 active+clean, 16
> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
> avail; 1/254952 degraded (0.000%)
> 2012-06-11 21:40:51.845622 pg v88215: 200 pgs: 184 active+clean, 16
> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
> avail
> 2012-06-11 21:40:52.857823 pg v88216: 200 pgs: 184 active+clean, 16
> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
> avail
> 2012-06-11 21:40:53.858281 pg v88217: 200 pgs: 184 active+clean, 16
> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
> avail
> 2012-06-11 21:40:54.855602 pg v88218: 200 pgs: 184 active+clean, 16
> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
> avail
> 2012-06-11 21:40:55.857241 pg v88219: 200 pgs: 184 active+clean, 16
> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
> avail
> 2012-06-11 21:40:56.857631 pg v88220: 200 pgs: 184 active+clean, 16
> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
> avail
> 2012-06-11 21:40:57.858987 pg v88221: 200 pgs: 185 active+clean, 15
> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
> avail
> 2012-06-11 21:40:58.880252 pg v88222: 200 pgs: 185 active+clean, 15
> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
> avail
> 2012-06-11 21:40:59.861910 pg v88223: 200 pgs: 188 active+clean, 12
> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
> avail
> 2012-06-11 21:41:00.902582 pg v88224: 200 pgs: 191 active+clean, 9
> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
> avail
> 2012-06-11 21:41:01.907767 pg v88225: 200 pgs: 196 active+clean, 4
> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
> avail
> 2012-06-11 21:41:02.876377 pg v88226: 200 pgs: 199 active+clean, 1
> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
> avail
> 2012-06-11 21:41:03.876929 pg v88227: 200 pgs: 200 active+clean;
> 783 GB data, 1928 GB used, 18111 GB / 20040 GB avail
>
> <disk type="network" device="disk">
> <driver name="qemu" type="raw"/>
> <source protocol="rbd" name="rbd/foo4">
> </source>
> <target dev="vdf" bus="virtio"/>
> </disk>
>
> 2. When i use rbd_cache=1, or true in my xml, for libvirt i get:
>
> <disk type="network" device="disk">
> <driver name="qemu" type="raw"/>
> <source protocol="rbd" name="rbd/foo5:rbd_cache=1">
> </source>
> <target dev="vdf" bus="virtio"/>
> </disk>
>
> libvirtd.log
> 2012-06-11 18:50:36.992+0000: 1751: error :
> qemuMonitorTextAddDrive:2820 : operation failed: open disk image file
> failed
>
> Libvirt version 0.9.8-2ubuntu17 with some additional patch set before
> ceph 0.46 version appears. Qemu-kvm 1.0+noroms-0ubuntu13.
>
> Do i need any other patch for libvirt ?? Without rbd_cache attaching is ok.
>
> --
> -----
> Pozdrawiam
>
> Sławek "sZiBis" Skowron
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: RBD stale on VM, and RBD cache enable problem
2012-07-03 17:39 ` Gregory Farnum
@ 2012-07-04 9:28 ` Sławomir Skowron
0 siblings, 0 replies; 3+ messages in thread
From: Sławomir Skowron @ 2012-07-04 9:28 UTC (permalink / raw)
To: Gregory Farnum; +Cc: ceph-devel
On Tue, Jul 3, 2012 at 7:39 PM, Gregory Farnum <greg@inktank.com> wrote:
> On Mon, Jun 11, 2012 at 12:53 PM, Sławomir Skowron <szibis@gmail.com> wrote:
>> I have two questions. My newly created cluster with xfs on all osd,
>> ubuntu precise, kernel 3.2.0-23-generic. Ceph 0.47.2-1precise
>>
>> pool 0 'data' rep size 3 crush_ruleset 0 object_hash rjenkins pg_num
>> 64 pgp_num 64 last_change 1228 owner 0 crash_replay_interval 45
>> pool 1 'metadata' rep size 3 crush_ruleset 1 object_hash rjenkins
>> pg_num 64 pgp_num 64 last_change 1226 owner 0
>> pool 2 'rbd' rep size 3 crush_ruleset 2 object_hash rjenkins pg_num 64
>> pgp_num 64 last_change 1232 owner 0
>> pool 3 '.rgw' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 8
>> pgp_num 8 last_change 3878 owner 18446744073709551615
>>
>> 1. After i stop all daemons on 1 machine in my 3 node cluster with 3
>> replicas, rbd image operations on vm, staling. DD on this device in VM
>> freezing, and after ceph start on this machine everything goes online.
>> Is there any problem with my config ?? in this situation ceph should
>> go from another copies with reads, and writes into another osd in
>> replica chain, yes ??
>
> It should switch to a new "primary" OSD as soon as they detect that
> one machine is missing, which by default will be ~25 seconds. How long
> did you wait to see if it would continue?
> If you'd like to reduce this time, you can turn down some combination of
> osd_heartbeat_grace -- default 20 seconds, and controls how long an OSD
> will wait before it decides a peer is down.
> osd_min_down_reporters -- default 1, controls how many OSDs need to
> report an OSD as down before accepting it. This is already as low as
> it should go
> osd_min_down_reports -- default 3, controls how many failure reports
> the monitor needs to receive before accepting an OSD as down. Since
> you only have 3 OSDs, and one is down, leaving this at 3 means you're
> going to wait for osd_heartbeat_grace plus osd_mon_report_interval_min
> (default 5; don't change this) before an OSD is marked down.
Thanks for this options, i will try this on int cluster.
> Given the logging you include I'm a little concerned that you have 1
> PG "stale", indicating that the monitor hasn't gotten a report on that
> PG in a very long time. That means either that one PG is somehow
> broken, or else that the OSD you turned off isn't getting marked down
> and that PG is the only one noticing it.
> Could you re-run this test with monitor debugging turned up, see how
> long it takes for the OSD to get marked down (using "ceph -w"), and
> report back?
There will be some problems with this, because this cluster is newly
re-inited, from ext4, to xfs, and with many other changes. Actual
there are two clusters @ bottom of one application doing data sync to
this clusters.
What is real problem, is that in moment of backfill, when i change
crush config, and rebalancing starting, or machine/group of osd goes
down, radosgw has some problems, with PUT request (writes) in next 9
minutes, cause some 504 (timeout on loadbalancer) in nginx, and down,
some delayed operations in Ceph cluster.
I will try to test this on integration cluster, but in this sprint it
will be difficult :(
> -Greg
>
>> Another test iozone on device, and it's stop after daemons stop on 1
>> machine, and after osd up, iozone go forward, how can i tune this to
>> work without freeze ??
>>
>> 2012-06-11 21:38:49.583133 pg v88173: 200 pgs: 60 active+clean, 1
>> stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
>> 18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
>> 2012-06-11 21:38:50.582257 pg v88174: 200 pgs: 60 active+clean, 1
>> stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
>> 18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
>> .....
>> 2012-06-11 21:39:49.991893 pg v88197: 200 pgs: 60 active+clean, 1
>> stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
>> 18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
>> 2012-06-11 21:39:50.992755 pg v88198: 200 pgs: 60 active+clean, 1
>> stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
>> 18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
>> 2012-06-11 21:39:51.993533 pg v88199: 200 pgs: 60 active+clean, 1
>> stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
>> 18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
>> 2012-06-11 21:39:52.994397 pg v88200: 200 pgs: 60 active+clean, 1
>> stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
>> 18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
>>
>> After boot all osd on stoped machine:
>>
>> 2012-06-11 21:40:37.826619 osd e4162: 72 osds: 53 up, 72 in
>> 2012-06-11 21:40:37.825706 mon.0 10.177.66.4:6790/0 348 : [INF] osd.24
>> 10.177.66.6:6800/21597 boot
>> 2012-06-11 21:40:38.825297 pg v88202: 200 pgs: 54 active+clean, 7
>> stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
>> 18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
>> 2012-06-11 21:40:38.826517 osd e4163: 72 osds: 54 up, 72 in
>> 2012-06-11 21:40:38.825250 mon.0 10.177.66.4:6790/0 349 : [INF] osd.25
>> 10.177.66.6:6803/21712 boot
>> 2012-06-11 21:40:38.825655 mon.0 10.177.66.4:6790/0 350 : [INF] osd.28
>> 10.177.66.6:6812/26210 boot
>> 2012-06-11 21:40:38.825907 mon.0 10.177.66.4:6790/0 351 : [INF] osd.29
>> 10.177.66.6:6815/26327 boot
>> 2012-06-11 21:40:39.826738 pg v88203: 200 pgs: 56 active+clean, 4
>> stale+active+clean, 3 peering, 137 active+degraded; 783 GB data, 1928
>> GB used, 18111 GB / 20040 GB avail; 76921/254952 degraded (30.171%)
>> 2012-06-11 21:40:39.830098 osd e4164: 72 osds: 59 up, 72 in
>> 2012-06-11 21:40:39.826570 mon.0 10.177.66.4:6790/0 352 : [INF] osd.26
>> 10.177.66.6:6806/21835 boot
>> 2012-06-11 21:40:39.826961 mon.0 10.177.66.4:6790/0 353 : [INF] osd.27
>> 10.177.66.6:6809/21953 boot
>> 2012-06-11 21:40:39.828147 mon.0 10.177.66.4:6790/0 354 : [INF] osd.30
>> 10.177.66.6:6818/26511 boot
>> 2012-06-11 21:40:39.828418 mon.0 10.177.66.4:6790/0 355 : [INF] osd.31
>> 10.177.66.6:6821/26583 boot
>> 2012-06-11 21:40:39.828935 mon.0 10.177.66.4:6790/0 356 : [INF] osd.33
>> 10.177.66.6:6827/26859 boot
>> 2012-06-11 21:40:39.829274 mon.0 10.177.66.4:6790/0 357 : [INF] osd.34
>> 10.177.66.6:6830/26979 boot
>> 2012-06-11 21:40:40.827935 pg v88204: 200 pgs: 56 active+clean, 4
>> stale+active+clean, 3 peering, 137 active+degraded; 783 GB data, 1928
>> GB used, 18111 GB / 20040 GB avail; 76921/254952 degraded (30.171%)
>> 2012-06-11 21:40:40.830059 osd e4165: 72 osds: 62 up, 72 in
>> 2012-06-11 21:40:40.827798 mon.0 10.177.66.4:6790/0 358 : [INF] osd.32
>> 10.177.66.6:6824/26701 boot
>> 2012-06-11 21:40:40.829043 mon.0 10.177.66.4:6790/0 359 : [INF] osd.35
>> 10.177.66.6:6833/27165 boot
>> 2012-06-11 21:40:40.829316 mon.0 10.177.66.4:6790/0 360 : [INF] osd.36
>> 10.177.66.6:6836/27280 boot
>> 2012-06-11 21:40:40.829602 mon.0 10.177.66.4:6790/0 361 : [INF] osd.37
>> 10.177.66.6:6839/27397 boot
>> 2012-06-11 21:40:41.828776 pg v88205: 200 pgs: 56 active+clean, 4
>> stale+active+clean, 3 peering, 137 active+degraded; 783 GB data, 1928
>> GB used, 18111 GB / 20040 GB avail; 76921/254952 degraded (30.171%)
>> 2012-06-11 21:40:41.831823 osd e4166: 72 osds: 68 up, 72 in
>> 2012-06-11 21:40:41.828713 mon.0 10.177.66.4:6790/0 362 : [INF] osd.38
>> 10.177.66.6:6842/27513 boot
>> 2012-06-11 21:40:41.829440 mon.0 10.177.66.4:6790/0 363 : [INF] osd.39
>> 10.177.66.6:6845/27628 boot
>> 2012-06-11 21:40:41.830226 mon.0 10.177.66.4:6790/0 364 : [INF] osd.40
>> 10.177.66.6:6848/27835 boot
>> 2012-06-11 21:40:41.830531 mon.0 10.177.66.4:6790/0 365 : [INF] osd.41
>> 10.177.66.6:6851/27950 boot
>> 2012-06-11 21:40:41.830778 mon.0 10.177.66.4:6790/0 366 : [INF] osd.42
>> 10.177.66.6:6854/28065 boot
>> 2012-06-11 21:40:41.831249 mon.0 10.177.66.4:6790/0 367 : [INF] osd.43
>> 10.177.66.6:6857/28181 boot
>> 2012-06-11 21:40:42.830440 pg v88206: 200 pgs: 57 active+clean, 4
>> stale+active+clean, 7 peering, 132 active+degraded; 783 GB data, 1928
>> GB used, 18111 GB / 20040 GB avail; 75543/254952 degraded (29.630%)
>> 2012-06-11 21:40:42.833294 osd e4167: 72 osds: 72 up, 72 in
>> 2012-06-11 21:40:42.831046 mon.0 10.177.66.4:6790/0 368 : [INF] osd.44
>> 10.177.66.6:6860/28373 boot
>> 2012-06-11 21:40:42.832004 mon.0 10.177.66.4:6790/0 369 : [INF] osd.45
>> 10.177.66.6:6863/28489 boot
>> 2012-06-11 21:40:42.832314 mon.0 10.177.66.4:6790/0 370 : [INF] osd.46
>> 10.177.66.6:6866/28607 boot
>> 2012-06-11 21:40:42.832545 mon.0 10.177.66.4:6790/0 371 : [INF] osd.47
>> 10.177.66.6:6869/28731 boot
>> 2012-06-11 21:40:43.830481 pg v88207: 200 pgs: 64 active+clean, 4
>> stale+active+clean, 7 peering, 125 active+degraded; 783 GB data, 1928
>> GB used, 18111 GB / 20040 GB avail; 72874/254952 degraded (28.583%)
>> 2012-06-11 21:40:43.831113 osd e4168: 72 osds: 72 up, 72 in
>> 2012-06-11 21:40:44.832521 pg v88208: 200 pgs: 79 active+clean, 1
>> stale+active+clean, 4 peering, 113 active+degraded, 3
>> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
>> avail; 66185/254952 degraded (25.960%)
>> 2012-06-11 21:40:45.834077 pg v88209: 200 pgs: 104 active+clean, 1
>> stale+active+clean, 4 peering, 85 active+degraded, 6
>> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
>> avail; 50399/254952 degraded (19.768%)
>> 2012-06-11 21:40:46.835367 pg v88210: 200 pgs: 125 active+clean, 1
>> stale+active+clean, 4 peering, 59 active+degraded, 11
>> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
>> avail; 38563/254952 degraded (15.126%)
>> 2012-06-11 21:40:47.836516 pg v88211: 200 pgs: 158 active+clean, 1
>> stale+active+clean, 26 active+degraded, 15 active+recovering; 783 GB
>> data, 1928 GB used, 18111 GB / 20040 GB avail; 18542/254952 degraded
>> (7.273%)
>> 2012-06-11 21:40:48.853560 pg v88212: 200 pgs: 184 active+clean, 16
>> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
>> avail; 1/254952 degraded (0.000%)
>> 2012-06-11 21:40:49.868514 pg v88213: 200 pgs: 184 active+clean, 16
>> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
>> avail; 1/254952 degraded (0.000%)
>> 2012-06-11 21:40:50.858244 pg v88214: 200 pgs: 184 active+clean, 16
>> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
>> avail; 1/254952 degraded (0.000%)
>> 2012-06-11 21:40:51.845622 pg v88215: 200 pgs: 184 active+clean, 16
>> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
>> avail
>> 2012-06-11 21:40:52.857823 pg v88216: 200 pgs: 184 active+clean, 16
>> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
>> avail
>> 2012-06-11 21:40:53.858281 pg v88217: 200 pgs: 184 active+clean, 16
>> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
>> avail
>> 2012-06-11 21:40:54.855602 pg v88218: 200 pgs: 184 active+clean, 16
>> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
>> avail
>> 2012-06-11 21:40:55.857241 pg v88219: 200 pgs: 184 active+clean, 16
>> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
>> avail
>> 2012-06-11 21:40:56.857631 pg v88220: 200 pgs: 184 active+clean, 16
>> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
>> avail
>> 2012-06-11 21:40:57.858987 pg v88221: 200 pgs: 185 active+clean, 15
>> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
>> avail
>> 2012-06-11 21:40:58.880252 pg v88222: 200 pgs: 185 active+clean, 15
>> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
>> avail
>> 2012-06-11 21:40:59.861910 pg v88223: 200 pgs: 188 active+clean, 12
>> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
>> avail
>> 2012-06-11 21:41:00.902582 pg v88224: 200 pgs: 191 active+clean, 9
>> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
>> avail
>> 2012-06-11 21:41:01.907767 pg v88225: 200 pgs: 196 active+clean, 4
>> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
>> avail
>> 2012-06-11 21:41:02.876377 pg v88226: 200 pgs: 199 active+clean, 1
>> active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
>> avail
>> 2012-06-11 21:41:03.876929 pg v88227: 200 pgs: 200 active+clean;
>> 783 GB data, 1928 GB used, 18111 GB / 20040 GB avail
>>
>> <disk type="network" device="disk">
>> <driver name="qemu" type="raw"/>
>> <source protocol="rbd" name="rbd/foo4">
>> </source>
>> <target dev="vdf" bus="virtio"/>
>> </disk>
>>
>> 2. When i use rbd_cache=1, or true in my xml, for libvirt i get:
>>
>> <disk type="network" device="disk">
>> <driver name="qemu" type="raw"/>
>> <source protocol="rbd" name="rbd/foo5:rbd_cache=1">
>> </source>
>> <target dev="vdf" bus="virtio"/>
>> </disk>
>>
>> libvirtd.log
>> 2012-06-11 18:50:36.992+0000: 1751: error :
>> qemuMonitorTextAddDrive:2820 : operation failed: open disk image file
>> failed
>>
>> Libvirt version 0.9.8-2ubuntu17 with some additional patch set before
>> ceph 0.46 version appears. Qemu-kvm 1.0+noroms-0ubuntu13.
>>
>> Do i need any other patch for libvirt ?? Without rbd_cache attaching is ok.
>>
>> --
>> -----
>> Pozdrawiam
>>
>> Sławek "sZiBis" Skowron
--
-----
Pozdrawiam
Sławek "sZiBis" Skowron
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2012-07-04 9:28 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-06-11 19:53 RBD stale on VM, and RBD cache enable problem Sławomir Skowron
2012-07-03 17:39 ` Gregory Farnum
2012-07-04 9:28 ` Sławomir Skowron
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.