* 0.31
@ 2011-07-10 10:48 Fyodor Ustinov
2011-07-10 13:24 ` 0.31 Fyodor Ustinov
2011-07-10 21:12 ` 0.31 Sage Weil
0 siblings, 2 replies; 6+ messages in thread
From: Fyodor Ustinov @ 2011-07-10 10:48 UTC (permalink / raw)
To: ceph-devel
Hi!
I upgraded my cluster and got these messages after upgrade mon/mds:
2011-07-10 13:41:42.950622 log 2011-07-10 13:41:40.891614 mds0
10.5.51.230:6801/23511 1 : [DBG] reconnect by client4705
10.5.51.242:0/8975 after 0.005341
2011-07-10 13:41:42.950622 log 2011-07-10 13:41:40.929079 mds0
10.5.51.230:6801/23511 2 : [DBG] reconnect by client4605
10.5.51.240:0/1543 after 0.042807
2011-07-10 13:41:42.950622 log 2011-07-10 13:41:41.008309 mds0
10.5.51.230:6801/23511 3 : [DBG] reconnect by client4710
10.5.51.241:0/1492286317 after 0.122038
2011-07-10 13:41:42.950622 log 2011-07-10 13:41:41.273756 mds0
10.5.51.230:6801/23511 4 : [DBG] reconnect by client4525
10.5.51.188:0/263432158 after 0.387484
2011-07-10 13:41:42.950622 log 2011-07-10 13:41:41.586636 mds0
10.5.51.230:6801/23511 5 : [ERR] loaded dup inode 1000000a5e9 [2,head]
v244 at
/dcvolia/amanda/state/servers/index/ticket.dcv/_/20110708000502_7.gz,
but inode 1000000a5e9.head v238 already exists at
/dcvolia/amanda/state/servers/index/ticket.dcv/_/20110708000502_7.gz.tmp
2011-07-10 13:41:42.950622 log 2011-07-10 13:41:41.587066 mds0
10.5.51.230:6801/23511 6 : [ERR] loaded dup inode 1000000a5e6 [2,head]
v429 at
/dcvolia/amanda/state/servers/index/mail.naperehresti.info/_usr/20110708000502_0.gz,
but inode 1000000a5e6.head v421 already exists at
/dcvolia/amanda/state/servers/index/mail.naperehresti.info/_usr/20110708000502_0.gz.tmp
2011-07-10 13:41:42.950622 log 2011-07-10 13:41:41.587313 mds0
10.5.51.230:6801/23511 7 : [ERR] loaded dup inode 1000000a5ec [2,head]
v244 at
/dcvolia/amanda/state/servers/index/iprit.dcv/_/20110708000502_6.gz, but
inode 1000000a5ec.head v238 already exists at
/dcvolia/amanda/state/servers/index/iprit.dcv/_/20110708000502_6.gz.tmp
2011-07-10 13:41:42.950622 log 2011-07-10 13:41:41.588392 mds0
10.5.51.230:6801/23511 8 : [ERR] loaded dup inode 1000000a5e8 [2,head]
v244 at
/dcvolia/amanda/state/servers/index/butan.dcv/_/20110708000502_7.gz, but
inode 1000000a5e8.head v238 already exists at
/dcvolia/amanda/state/servers/index/butan.dcv/_/20110708000502_7.gz.tmp
2011-07-10 13:41:42.950622 log 2011-07-10 13:41:41.588509 mds0
10.5.51.230:6801/23511 9 : [ERR] loaded dup inode 1000000a5f2 [2,head]
v274 at
/dcvolia/amanda/state/servers/index/tmg.net.ua/_/20110708000502_6.gz,
but inode 1000000a5f2.head v270 already exists at
/dcvolia/amanda/state/servers/index/tmg.net.ua/_/20110708000502_6.gz.tmp
2011-07-10 13:41:42.950622 log 2011-07-10 13:41:41.592042 mds0
10.5.51.230:6801/23511 10 : [ERR] loaded dup inode 1000000a5e7 [2,head]
v244 at
/dcvolia/amanda/state/servers/index/zoman.dcv/_/20110708000502_7.gz, but
inode 1000000a5e7.head v238 already exists at
/dcvolia/amanda/state/servers/index/zoman.dcv/_/20110708000502_7.gz.tmp
2011-07-10 13:41:42.950622 log 2011-07-10 13:41:41.592998 mds0
10.5.51.230:6801/23511 11 : [ERR] loaded dup inode 1000000a5f1 [2,head]
v204 at
/dcvolia/amanda/state/servers/index/ns4.dc.volia.com/_/20110708000502_0.gz,
but inode 1000000a5f1.head v198 already exists at
/dcvolia/amanda/state/servers/index/ns4.dc.volia.com/_/20110708000502_0.gz.tmp
I can not pay attention or do I need start to fear?
WBR,
Fyodor.
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: 0.31
2011-07-10 10:48 0.31 Fyodor Ustinov
@ 2011-07-10 13:24 ` Fyodor Ustinov
2011-07-10 21:12 ` 0.31 Sage Weil
1 sibling, 0 replies; 6+ messages in thread
From: Fyodor Ustinov @ 2011-07-10 13:24 UTC (permalink / raw)
To: ceph-devel
Hi again.
I start to fear.
After deleting one of file I got:
2011-07-10 16:23:00.062754 log 2011-07-10 16:22:51.410099 mds0
10.5.51.230:6800/24957 5 : [ERR] loaded dup inode 1000000a5e9 [2,head]
v244 at
/dcvolia/amanda/state/servers/index/ticket.dcv/_/20110708000502_7.gz,
but inode 1000000a5e9.head v1011 already exists at ~mds0/stray0/1000000a5e9
P.S. What is it?
2011-07-10 16:21:51.188911 log 2011-07-10 16:21:50.024355 mds0
10.5.51.230:6801/24695 11 : [ERR] unmatched fragstat size on single
dirfrag 600, inode has f(v5 m2011-07-10 16:21:50.023759 21=18+3),
dirfrag has f(v5 m2011-07-10 16:21:50.023759 1=1+0)
2011-07-10 16:21:51.188911 log 2011-07-10 16:21:50.024373 mds0
10.5.51.230:6801/24695 12 : [ERR] unmatched rstat rbytes on single
dirfrag 600, inode has n(v5 rc2011-07-10 16:21:50.023759 b12892122
21=18+3), dirfrag has n(v5 rc2011-07-10 16:21:50.023759 1=1+0)
WBR,
Fyodor.
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: 0.31
2011-07-10 10:48 0.31 Fyodor Ustinov
2011-07-10 13:24 ` 0.31 Fyodor Ustinov
@ 2011-07-10 21:12 ` Sage Weil
2011-07-10 21:29 ` 0.31 Fyodor Ustinov
1 sibling, 1 reply; 6+ messages in thread
From: Sage Weil @ 2011-07-10 21:12 UTC (permalink / raw)
To: Fyodor Ustinov; +Cc: ceph-devel
Hmm, yeah the 'loaded dup inode' messages should never happen. They
appear to be just files, though, not directories, so you can safely ignore
them. The namespace repair will need to clean that up at some point. The
recursive rstat errors you saw are probably just fallout from that, and
can also be ignored.
I'm curious how you got into that state, though. Are you running a single
or clustered mds? Is the workload purely amanda?
sage
On Sun, 10 Jul 2011, Fyodor Ustinov wrote:
> Hi!
>
> I upgraded my cluster and got these messages after upgrade mon/mds:
>
> 2011-07-10 13:41:42.950622 log 2011-07-10 13:41:40.891614 mds0
> 10.5.51.230:6801/23511 1 : [DBG] reconnect by client4705 10.5.51.242:0/8975
> after 0.005341
> 2011-07-10 13:41:42.950622 log 2011-07-10 13:41:40.929079 mds0
> 10.5.51.230:6801/23511 2 : [DBG] reconnect by client4605 10.5.51.240:0/1543
> after 0.042807
> 2011-07-10 13:41:42.950622 log 2011-07-10 13:41:41.008309 mds0
> 10.5.51.230:6801/23511 3 : [DBG] reconnect by client4710
> 10.5.51.241:0/1492286317 after 0.122038
> 2011-07-10 13:41:42.950622 log 2011-07-10 13:41:41.273756 mds0
> 10.5.51.230:6801/23511 4 : [DBG] reconnect by client4525
> 10.5.51.188:0/263432158 after 0.387484
> 2011-07-10 13:41:42.950622 log 2011-07-10 13:41:41.586636 mds0
> 10.5.51.230:6801/23511 5 : [ERR] loaded dup inode 1000000a5e9 [2,head] v244 at
> /dcvolia/amanda/state/servers/index/ticket.dcv/_/20110708000502_7.gz, but
> inode 1000000a5e9.head v238 already exists at
> /dcvolia/amanda/state/servers/index/ticket.dcv/_/20110708000502_7.gz.tmp
> 2011-07-10 13:41:42.950622 log 2011-07-10 13:41:41.587066 mds0
> 10.5.51.230:6801/23511 6 : [ERR] loaded dup inode 1000000a5e6 [2,head] v429 at
> /dcvolia/amanda/state/servers/index/mail.naperehresti.info/_usr/20110708000502_0.gz,
> but inode 1000000a5e6.head v421 already exists at
> /dcvolia/amanda/state/servers/index/mail.naperehresti.info/_usr/20110708000502_0.gz.tmp
> 2011-07-10 13:41:42.950622 log 2011-07-10 13:41:41.587313 mds0
> 10.5.51.230:6801/23511 7 : [ERR] loaded dup inode 1000000a5ec [2,head] v244 at
> /dcvolia/amanda/state/servers/index/iprit.dcv/_/20110708000502_6.gz, but inode
> 1000000a5ec.head v238 already exists at
> /dcvolia/amanda/state/servers/index/iprit.dcv/_/20110708000502_6.gz.tmp
> 2011-07-10 13:41:42.950622 log 2011-07-10 13:41:41.588392 mds0
> 10.5.51.230:6801/23511 8 : [ERR] loaded dup inode 1000000a5e8 [2,head] v244 at
> /dcvolia/amanda/state/servers/index/butan.dcv/_/20110708000502_7.gz, but inode
> 1000000a5e8.head v238 already exists at
> /dcvolia/amanda/state/servers/index/butan.dcv/_/20110708000502_7.gz.tmp
> 2011-07-10 13:41:42.950622 log 2011-07-10 13:41:41.588509 mds0
> 10.5.51.230:6801/23511 9 : [ERR] loaded dup inode 1000000a5f2 [2,head] v274 at
> /dcvolia/amanda/state/servers/index/tmg.net.ua/_/20110708000502_6.gz, but
> inode 1000000a5f2.head v270 already exists at
> /dcvolia/amanda/state/servers/index/tmg.net.ua/_/20110708000502_6.gz.tmp
> 2011-07-10 13:41:42.950622 log 2011-07-10 13:41:41.592042 mds0
> 10.5.51.230:6801/23511 10 : [ERR] loaded dup inode 1000000a5e7 [2,head] v244
> at /dcvolia/amanda/state/servers/index/zoman.dcv/_/20110708000502_7.gz, but
> inode 1000000a5e7.head v238 already exists at
> /dcvolia/amanda/state/servers/index/zoman.dcv/_/20110708000502_7.gz.tmp
> 2011-07-10 13:41:42.950622 log 2011-07-10 13:41:41.592998 mds0
> 10.5.51.230:6801/23511 11 : [ERR] loaded dup inode 1000000a5f1 [2,head] v204
> at /dcvolia/amanda/state/servers/index/ns4.dc.volia.com/_/20110708000502_0.gz,
> but inode 1000000a5f1.head v198 already exists at
> /dcvolia/amanda/state/servers/index/ns4.dc.volia.com/_/20110708000502_0.gz.tmp
>
> I can not pay attention or do I need start to fear?
>
> WBR,
> Fyodor.
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 0.31
2011-07-10 21:12 ` 0.31 Sage Weil
@ 2011-07-10 21:29 ` Fyodor Ustinov
2011-07-10 21:36 ` 0.31 Sage Weil
0 siblings, 1 reply; 6+ messages in thread
From: Fyodor Ustinov @ 2011-07-10 21:29 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-devel
On 07/11/2011 12:12 AM, Sage Weil wrote:
> Hmm, yeah the 'loaded dup inode' messages should never happen. They
> appear to be just files, though, not directories, so you can safely ignore
> them. The namespace repair will need to clean that up at some point. The
> recursive rstat errors you saw are probably just fallout from that, and
> can also be ignored.
I understand it correctly, that until not implemented the Feature # 86
I have no way to fix this?
> I'm curious how you got into that state, though. Are you running a single
> or clustered mds? Is the workload purely amanda?
Single mds. I have these clients:
1. Amanda. Used cfuse.
2. Gate. Used cfuse. NFS gate.
3. One experimental server with many different workloads.
WBR,
Fyodor.
P.S. My friends have always said that I can find a bug in the software,
even where it can not be. Sometimes it interferes. :)
I will try again to use ceph with BackupPC. Last time it was over "brain
explosion" in ceph. :)
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: 0.31
2011-07-10 21:29 ` 0.31 Fyodor Ustinov
@ 2011-07-10 21:36 ` Sage Weil
2011-07-10 21:53 ` 0.31 Fyodor Ustinov
0 siblings, 1 reply; 6+ messages in thread
From: Sage Weil @ 2011-07-10 21:36 UTC (permalink / raw)
To: Fyodor Ustinov; +Cc: ceph-devel
On Mon, 11 Jul 2011, Fyodor Ustinov wrote:
> On 07/11/2011 12:12 AM, Sage Weil wrote:
> > Hmm, yeah the 'loaded dup inode' messages should never happen. They
> > appear to be just files, though, not directories, so you can safely ignore
> > them. The namespace repair will need to clean that up at some point. The
> > recursive rstat errors you saw are probably just fallout from that, and
> > can also be ignored.
> I understand it correctly, that until not implemented the Feature # 86 I have
> no way to fix this?
Right. Well, this is going to come incrementally. We'll have a namespace
scan that fixes some basic issues before there is a full-blown robust
repair tool.
> > I'm curious how you got into that state, though. Are you running a single
> > or clustered mds? Is the workload purely amanda?
>
> Single mds. I have these clients:
> 1. Amanda. Used cfuse.
> 2. Gate. Used cfuse. NFS gate.
> 3. One experimental server with many different workloads.
>
> WBR,
> Fyodor.
>
> P.S. My friends have always said that I can find a bug in the software, even
> where it can not be. Sometimes it interferes. :)
>
> I will try again to use ceph with BackupPC. Last time it was over "brain
> explosion" in ceph. :)
:) Okay. We're putting locking specific tests in place before trying to
reproduce #1150 with Amanda, BTW. It sounds like we need an amanda-like
workload though anyway, given the issues you're seeing. How old is this
fs? How many times has it been upgraded?
sage
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 0.31
2011-07-10 21:36 ` 0.31 Sage Weil
@ 2011-07-10 21:53 ` Fyodor Ustinov
0 siblings, 0 replies; 6+ messages in thread
From: Fyodor Ustinov @ 2011-07-10 21:53 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-devel
On 07/11/2011 12:36 AM, Sage Weil wrote:
> On Mon, 11 Jul 2011, Fyodor Ustinov wrote:
>> On 07/11/2011 12:12 AM, Sage Weil wrote:
>>> Hmm, yeah the 'loaded dup inode' messages should never happen. They
>>> appear to be just files, though, not directories, so you can safely ignore
>>> them. The namespace repair will need to clean that up at some point. The
>>> recursive rstat errors you saw are probably just fallout from that, and
>>> can also be ignored.
>> I understand it correctly, that until not implemented the Feature # 86 I have
>> no way to fix this?
> Right. Well, this is going to come incrementally. We'll have a namespace
> scan that fixes some basic issues before there is a full-blown robust
> repair tool.
ok, I'll be waiting impatiently.
> :) Okay. We're putting locking specific tests in place before trying to
> reproduce #1150 with Amanda, BTW. It sounds like we need an amanda-like
> workload though anyway, given the issues you're seeing. How old is this
> fs? How many times has it been upgraded?
fs has been created not later 2011-06-09.
I'm not sure that I understand correctly "fs upgraded", but cluster
upgraded to each new ceph versions. I.e. 0.29 -> 0.29.1 -> 0.30 -> 0.31
WBR,
Fyodor.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2011-07-10 21:53 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-07-10 10:48 0.31 Fyodor Ustinov
2011-07-10 13:24 ` 0.31 Fyodor Ustinov
2011-07-10 21:12 ` 0.31 Sage Weil
2011-07-10 21:29 ` 0.31 Fyodor Ustinov
2011-07-10 21:36 ` 0.31 Sage Weil
2011-07-10 21:53 ` 0.31 Fyodor Ustinov
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.