0.31

0.31

[Fyodor Ustinov]

Hi!

I upgraded my cluster and  got these messages after upgrade mon/mds:

2011-07-10 13:41:42.950622   log 2011-07-10 13:41:40.891614 mds0 
10.5.51.230:6801/23511 1 : [DBG] reconnect by client4705 
10.5.51.242:0/8975 after 0.005341
2011-07-10 13:41:42.950622   log 2011-07-10 13:41:40.929079 mds0 
10.5.51.230:6801/23511 2 : [DBG] reconnect by client4605 
10.5.51.240:0/1543 after 0.042807
2011-07-10 13:41:42.950622   log 2011-07-10 13:41:41.008309 mds0 
10.5.51.230:6801/23511 3 : [DBG] reconnect by client4710 
10.5.51.241:0/1492286317 after 0.122038
2011-07-10 13:41:42.950622   log 2011-07-10 13:41:41.273756 mds0 
10.5.51.230:6801/23511 4 : [DBG] reconnect by client4525 
10.5.51.188:0/263432158 after 0.387484
2011-07-10 13:41:42.950622   log 2011-07-10 13:41:41.586636 mds0 
10.5.51.230:6801/23511 5 : [ERR] loaded dup inode 1000000a5e9 [2,head] 
v244 at 
/dcvolia/amanda/state/servers/index/ticket.dcv/_/20110708000502_7.gz, 
but inode 1000000a5e9.head v238 already exists at 
/dcvolia/amanda/state/servers/index/ticket.dcv/_/20110708000502_7.gz.tmp
2011-07-10 13:41:42.950622   log 2011-07-10 13:41:41.587066 mds0 
10.5.51.230:6801/23511 6 : [ERR] loaded dup inode 1000000a5e6 [2,head] 
v429 at 
/dcvolia/amanda/state/servers/index/mail.naperehresti.info/_usr/20110708000502_0.gz, 
but inode 1000000a5e6.head v421 already exists at 
/dcvolia/amanda/state/servers/index/mail.naperehresti.info/_usr/20110708000502_0.gz.tmp
2011-07-10 13:41:42.950622   log 2011-07-10 13:41:41.587313 mds0 
10.5.51.230:6801/23511 7 : [ERR] loaded dup inode 1000000a5ec [2,head] 
v244 at 
/dcvolia/amanda/state/servers/index/iprit.dcv/_/20110708000502_6.gz, but 
inode 1000000a5ec.head v238 already exists at 
/dcvolia/amanda/state/servers/index/iprit.dcv/_/20110708000502_6.gz.tmp
2011-07-10 13:41:42.950622   log 2011-07-10 13:41:41.588392 mds0 
10.5.51.230:6801/23511 8 : [ERR] loaded dup inode 1000000a5e8 [2,head] 
v244 at 
/dcvolia/amanda/state/servers/index/butan.dcv/_/20110708000502_7.gz, but 
inode 1000000a5e8.head v238 already exists at 
/dcvolia/amanda/state/servers/index/butan.dcv/_/20110708000502_7.gz.tmp
2011-07-10 13:41:42.950622   log 2011-07-10 13:41:41.588509 mds0 
10.5.51.230:6801/23511 9 : [ERR] loaded dup inode 1000000a5f2 [2,head] 
v274 at 
/dcvolia/amanda/state/servers/index/tmg.net.ua/_/20110708000502_6.gz, 
but inode 1000000a5f2.head v270 already exists at 
/dcvolia/amanda/state/servers/index/tmg.net.ua/_/20110708000502_6.gz.tmp
2011-07-10 13:41:42.950622   log 2011-07-10 13:41:41.592042 mds0 
10.5.51.230:6801/23511 10 : [ERR] loaded dup inode 1000000a5e7 [2,head] 
v244 at 
/dcvolia/amanda/state/servers/index/zoman.dcv/_/20110708000502_7.gz, but 
inode 1000000a5e7.head v238 already exists at 
/dcvolia/amanda/state/servers/index/zoman.dcv/_/20110708000502_7.gz.tmp
2011-07-10 13:41:42.950622   log 2011-07-10 13:41:41.592998 mds0 
10.5.51.230:6801/23511 11 : [ERR] loaded dup inode 1000000a5f1 [2,head] 
v204 at 
/dcvolia/amanda/state/servers/index/ns4.dc.volia.com/_/20110708000502_0.gz, 
but inode 1000000a5f1.head v198 already exists at 
/dcvolia/amanda/state/servers/index/ns4.dc.volia.com/_/20110708000502_0.gz.tmp

I can not pay attention or do I need start to fear?

WBR,
     Fyodor.

Re: 0.31

[Fyodor Ustinov]

Hi again.

I start to fear.

After deleting one of file I got:

2011-07-10 16:23:00.062754   log 2011-07-10 16:22:51.410099 mds0 
10.5.51.230:6800/24957 5 : [ERR] loaded dup inode 1000000a5e9 [2,head] 
v244 at 
/dcvolia/amanda/state/servers/index/ticket.dcv/_/20110708000502_7.gz, 
but inode 1000000a5e9.head v1011 already exists at ~mds0/stray0/1000000a5e9

P.S. What is it?

2011-07-10 16:21:51.188911   log 2011-07-10 16:21:50.024355 mds0 
10.5.51.230:6801/24695 11 : [ERR] unmatched fragstat size on single 
dirfrag 600, inode has f(v5 m2011-07-10 16:21:50.023759 21=18+3), 
dirfrag has f(v5 m2011-07-10 16:21:50.023759 1=1+0)
2011-07-10 16:21:51.188911   log 2011-07-10 16:21:50.024373 mds0 
10.5.51.230:6801/24695 12 : [ERR] unmatched rstat rbytes on single 
dirfrag 600, inode has n(v5 rc2011-07-10 16:21:50.023759 b12892122 
21=18+3), dirfrag has n(v5 rc2011-07-10 16:21:50.023759 1=1+0)

WBR,
     Fyodor.

Re: 0.31

[Sage Weil]

Hmm, yeah the 'loaded dup inode' messages should never happen.  They 
appear to be just files, though, not directories, so you can safely ignore 
them.  The namespace repair will need to clean that up at some point. The 
recursive rstat errors you saw are probably just fallout from that, and 
can also be ignored.

I'm curious how you got into that state, though.  Are you running a single 
or clustered mds?  Is the workload purely amanda?

sage


On Sun, 10 Jul 2011, Fyodor Ustinov wrote:

> Hi!
> 
> I upgraded my cluster and  got these messages after upgrade mon/mds:
> 
> 2011-07-10 13:41:42.950622   log 2011-07-10 13:41:40.891614 mds0
> 10.5.51.230:6801/23511 1 : [DBG] reconnect by client4705 10.5.51.242:0/8975
> after 0.005341
> 2011-07-10 13:41:42.950622   log 2011-07-10 13:41:40.929079 mds0
> 10.5.51.230:6801/23511 2 : [DBG] reconnect by client4605 10.5.51.240:0/1543
> after 0.042807
> 2011-07-10 13:41:42.950622   log 2011-07-10 13:41:41.008309 mds0
> 10.5.51.230:6801/23511 3 : [DBG] reconnect by client4710
> 10.5.51.241:0/1492286317 after 0.122038
> 2011-07-10 13:41:42.950622   log 2011-07-10 13:41:41.273756 mds0
> 10.5.51.230:6801/23511 4 : [DBG] reconnect by client4525
> 10.5.51.188:0/263432158 after 0.387484
> 2011-07-10 13:41:42.950622   log 2011-07-10 13:41:41.586636 mds0
> 10.5.51.230:6801/23511 5 : [ERR] loaded dup inode 1000000a5e9 [2,head] v244 at
> /dcvolia/amanda/state/servers/index/ticket.dcv/_/20110708000502_7.gz, but
> inode 1000000a5e9.head v238 already exists at
> /dcvolia/amanda/state/servers/index/ticket.dcv/_/20110708000502_7.gz.tmp
> 2011-07-10 13:41:42.950622   log 2011-07-10 13:41:41.587066 mds0
> 10.5.51.230:6801/23511 6 : [ERR] loaded dup inode 1000000a5e6 [2,head] v429 at
> /dcvolia/amanda/state/servers/index/mail.naperehresti.info/_usr/20110708000502_0.gz,
> but inode 1000000a5e6.head v421 already exists at
> /dcvolia/amanda/state/servers/index/mail.naperehresti.info/_usr/20110708000502_0.gz.tmp
> 2011-07-10 13:41:42.950622   log 2011-07-10 13:41:41.587313 mds0
> 10.5.51.230:6801/23511 7 : [ERR] loaded dup inode 1000000a5ec [2,head] v244 at
> /dcvolia/amanda/state/servers/index/iprit.dcv/_/20110708000502_6.gz, but inode
> 1000000a5ec.head v238 already exists at
> /dcvolia/amanda/state/servers/index/iprit.dcv/_/20110708000502_6.gz.tmp
> 2011-07-10 13:41:42.950622   log 2011-07-10 13:41:41.588392 mds0
> 10.5.51.230:6801/23511 8 : [ERR] loaded dup inode 1000000a5e8 [2,head] v244 at
> /dcvolia/amanda/state/servers/index/butan.dcv/_/20110708000502_7.gz, but inode
> 1000000a5e8.head v238 already exists at
> /dcvolia/amanda/state/servers/index/butan.dcv/_/20110708000502_7.gz.tmp
> 2011-07-10 13:41:42.950622   log 2011-07-10 13:41:41.588509 mds0
> 10.5.51.230:6801/23511 9 : [ERR] loaded dup inode 1000000a5f2 [2,head] v274 at
> /dcvolia/amanda/state/servers/index/tmg.net.ua/_/20110708000502_6.gz, but
> inode 1000000a5f2.head v270 already exists at
> /dcvolia/amanda/state/servers/index/tmg.net.ua/_/20110708000502_6.gz.tmp
> 2011-07-10 13:41:42.950622   log 2011-07-10 13:41:41.592042 mds0
> 10.5.51.230:6801/23511 10 : [ERR] loaded dup inode 1000000a5e7 [2,head] v244
> at /dcvolia/amanda/state/servers/index/zoman.dcv/_/20110708000502_7.gz, but
> inode 1000000a5e7.head v238 already exists at
> /dcvolia/amanda/state/servers/index/zoman.dcv/_/20110708000502_7.gz.tmp
> 2011-07-10 13:41:42.950622   log 2011-07-10 13:41:41.592998 mds0
> 10.5.51.230:6801/23511 11 : [ERR] loaded dup inode 1000000a5f1 [2,head] v204
> at /dcvolia/amanda/state/servers/index/ns4.dc.volia.com/_/20110708000502_0.gz,
> but inode 1000000a5f1.head v198 already exists at
> /dcvolia/amanda/state/servers/index/ns4.dc.volia.com/_/20110708000502_0.gz.tmp
> 
> I can not pay attention or do I need start to fear?
> 
> WBR,
>     Fyodor.
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: 0.31

[Fyodor Ustinov]

On 07/11/2011 12:12 AM, Sage Weil wrote:
> Hmm, yeah the 'loaded dup inode' messages should never happen.  They
> appear to be just files, though, not directories, so you can safely ignore
> them.  The namespace repair will need to clean that up at some point. The
> recursive rstat errors you saw are probably just fallout from that, and
> can also be ignored.
I understand it correctly, that until not  implemented the Feature # 86 
I have no way to fix this?
> I'm curious how you got into that state, though.  Are you running a single
> or clustered mds?  Is the workload purely amanda?
Single mds. I have these clients:
1. Amanda. Used cfuse.
2. Gate. Used cfuse. NFS gate.
3. One experimental server with many different workloads.

WBR,
     Fyodor.

P.S. My friends have always said that I can find a bug in the software, 
even where it can not be. Sometimes it interferes. :)

I will try again to use ceph with BackupPC. Last time it was over "brain 
explosion" in ceph. :)

Re: 0.31

[Sage Weil]

On Mon, 11 Jul 2011, Fyodor Ustinov wrote:
> On 07/11/2011 12:12 AM, Sage Weil wrote:
> > Hmm, yeah the 'loaded dup inode' messages should never happen.  They
> > appear to be just files, though, not directories, so you can safely ignore
> > them.  The namespace repair will need to clean that up at some point. The
> > recursive rstat errors you saw are probably just fallout from that, and
> > can also be ignored.
> I understand it correctly, that until not  implemented the Feature # 86 I have
> no way to fix this?

Right.  Well, this is going to come incrementally.  We'll have a namespace 
scan that fixes some basic issues before there is a full-blown robust 
repair tool.

> > I'm curious how you got into that state, though.  Are you running a single
> > or clustered mds?  Is the workload purely amanda?
>
> Single mds. I have these clients:
> 1. Amanda. Used cfuse.
> 2. Gate. Used cfuse. NFS gate.
> 3. One experimental server with many different workloads.
> 
> WBR,
>     Fyodor.
> 
> P.S. My friends have always said that I can find a bug in the software, even
> where it can not be. Sometimes it interferes. :)
> 
> I will try again to use ceph with BackupPC. Last time it was over "brain
> explosion" in ceph. :)

:) Okay.  We're putting locking specific tests in place before trying to 
reproduce #1150 with Amanda, BTW.  It sounds like we need an amanda-like 
workload though anyway, given the issues you're seeing.  How old is this 
fs?  How many times has it been upgraded?

sage

Re: 0.31

[Fyodor Ustinov]

On 07/11/2011 12:36 AM, Sage Weil wrote:
> On Mon, 11 Jul 2011, Fyodor Ustinov wrote:
>> On 07/11/2011 12:12 AM, Sage Weil wrote:
>>> Hmm, yeah the 'loaded dup inode' messages should never happen.  They
>>> appear to be just files, though, not directories, so you can safely ignore
>>> them.  The namespace repair will need to clean that up at some point. The
>>> recursive rstat errors you saw are probably just fallout from that, and
>>> can also be ignored.
>> I understand it correctly, that until not  implemented the Feature # 86 I have
>> no way to fix this?
> Right.  Well, this is going to come incrementally.  We'll have a namespace
> scan that fixes some basic issues before there is a full-blown robust
> repair tool.
ok, I'll be waiting impatiently.

> :) Okay.  We're putting locking specific tests in place before trying to
> reproduce #1150 with Amanda, BTW.  It sounds like we need an amanda-like
> workload though anyway, given the issues you're seeing.  How old is this
> fs?  How many times has it been upgraded?
fs has been created not later 2011-06-09.

I'm not sure that I understand correctly "fs upgraded", but cluster 
upgraded to each new ceph versions. I.e. 0.29 -> 0.29.1 -> 0.30 -> 0.31

WBR,
     Fyodor.