amanda oddments of recent occurance

amanda oddments of recent occurance

[Gene Heskett]

Greetings, and I hope everybody had a very happy Christmas.

For some odd reason, I've had several instances of amandad turning a 
pair of copies of itself into zombies or something here on the server 
box, evidenced by getting an email report and printout that lists 
only the client, and not the servers stuff.  It happened again 
tonight, so I killed them, but could not run anything amanda related 
as it claimed the server was not responding as a client.  I also note 
that kpm wouldn't connect to the system to monitor things, claiming 
that the dcopserver apparently wasn't running.  htop did just fine 
however.  So I rebooted and now the missed backup is proceeding 
apparently normally, with one head scratcher showing up in an 
amstatus report, from a section of it:

coyote:/usr/dlds-misc/FC3-i386-disk2.iso           1      637m wait 
for dumping
coyote:/usr/dlds-misc/FC3-i386-rescuecd.iso        0       76m wait 
for dumping
coyote:/usr/dlds-misc/FC3/FC3-i386-SRPMS-disc1.iso 1      562m wait 
for dumping
coyote:/usr/dlds-misc/FC3/FC3-i386-SRPMS-disk2.iso 0      562m wait 
for dumping
coyote:/usr/dlds-misc/FC3/FC3-i386-SRPMS-disk3.iso 0      562m wait 
for dumping
coyote:/usr/dlds-misc/FC3/FC3-i386-SRPMS-disk4.iso 0      562m wait 
for dumping
coyote:/usr/dlds-misc/FC3/FC3-i386-disc1.iso       1      617m wait 
for dumping
coyote:/usr/dlds-misc/FC3/FC3-i386-disk3.iso       1      636m wait 
for dumping
coyote:/usr/dlds-misc/FC3/FC3-i386-disk4.iso       1      386m wait 
for dumping

Looks normal at first glance, but wait, there is more!  Those first 2 
iso images are not *in* the /usr/dlds-misc directory as shown but are 
in fact in the /usr/dlds-misc/FC3 directory as the other 7 images 
shown correctly. 

?????????????????

This was after just over 24 hours of uptime with the brand new 2.6.10 
kernel.  Previous to that, I'd been running 2.6.10-rc3-mm1-V0.33-04 
for over a week, this being the realtime-preemptable patches Ingo 
Molnar is working on.  I might add that they appeared to work quite 
well, with the machine remaining much more responsive from the 
keyboard while amanda was running, or while kmail was doing a mail 
fetch compared to how its acting now.  Running 2.6.10, I've had lags 
of over a minute between what I type, and its appearance on the 
screen if kmail is doing a fetch or amanda has a few copies of gzip 
running.

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.30% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2004 by Maurice Eugene Heskett, all rights reserved.

Re: amanda oddments of recent occurance

[Gene Heskett]

On Monday 27 December 2004 03:28, Paul Bijnens wrote:
>Gene Heskett wrote:

And I'm Cc:ing this to the lkml too, Paul.  Somebody there needs to 
know we now have an amanda problem with 2.6.10 that I didn't have 
with 2.6.10-rc3-mm1-V0.7.33-04.

>> coyote:/usr/dlds-misc/FC3-i386-disk2.iso           1      637m
>> wait for dumping coyote:/usr/dlds-misc/FC3-i386-rescuecd.iso      
>>  0       76m wait for dumping
>> coyote:/usr/dlds-misc/FC3/FC3-i386-SRPMS-disc1.iso 1      562m
>> wait for dumping
>> coyote:/usr/dlds-misc/FC3/FC3-i386-SRPMS-disk2.iso 0      562m
>> wait for dumping
>> coyote:/usr/dlds-misc/FC3/FC3-i386-SRPMS-disk3.iso 0      562m
>> wait for dumping
>> coyote:/usr/dlds-misc/FC3/FC3-i386-SRPMS-disk4.iso 0      562m
>> wait for dumping coyote:/usr/dlds-misc/FC3/FC3-i386-disc1.iso     
>>  1      617m wait for dumping
>> coyote:/usr/dlds-misc/FC3/FC3-i386-disk3.iso       1      636m
>> wait for dumping coyote:/usr/dlds-misc/FC3/FC3-i386-disk4.iso     
>>  1      386m wait for dumping
>>
>> Looks normal at first glance, but wait, there is more!  Those
>> first 2 iso images are not *in* the /usr/dlds-misc directory as
>> shown but are in fact in the /usr/dlds-misc/FC3 directory as the
>> other 7 images shown correctly.
>
>To be able to back up a single file (the large iso's) you probably
>created special DLE-entries using includes, with "diskname"
> different than "diskdevice".
>
>coyote  /usr/dlds-misc/FC3-i386-disk2.iso /usr/dlds-misc/FC3 {
> comp-user-tar
> include "./FC3-i386-disk2.iso
> }  1
>
>What you see in is the "diskname" (second argument in DLE), which
> you may name anything you like;  it just has to be unique in the
> disklist. And apparently you used a different scheme for the first
> two.
>
>Is this what has happened?
>
Absolutely correct Paul.  Fixed now.

But I've also another problem, amanda barfed again last night, about 
90% done, leaving:
[amanda@coyote root]$ ps -ea|grep amandad
 3126 ?        00:00:00 amandad
 3128 ?        00:00:00 amandad <defunct>

still present, and it missed several of the larger dle's due to data 
timeouts after amandad went on strike.  I can kill them, but even if 
I do, an 'amcheck Daily' returns this:
[amanda@coyote root]$ amcheck Daily
Amanda Tape Server Host Check
-----------------------------
Holding disk /dumps: 26558 MB disk space available, using 26058 MB
amcheck-server: slot 11: date 20041227 label Dailys-11 (active tape)
amcheck-server: slot 12: date 20041213 label Dailys-12 (exact label 
match)
NOTE: skipping tape-writable test
Tape Dailys-12 label ok
Server check took 0.600 seconds

Amanda Backup Client Hosts Check
--------------------------------
WARNING: coyote: selfcheck reply timed out.
Client check: 2 hosts checked in 21.313 seconds, 1 problem found

(brought to you by Amanda 2.4.5b1-20041221)

Now, I had shut down an smb mounted box in the basement about 2:25 am, 
and that got this in the logs:
--------------
Dec 27 02:25:11 coyote mount.smbfs[22699]: [2004/12/27 02:25:11, 0] 
tdb/tdbutil.c:tdb_log(725)
Dec 27 02:25:11 coyote mount.smbfs[22699]:   
tdb(/var/cache/samba/gencache.tdb): tdb_lock failed on list 10 
ltype=0
 (Bad file descriptor)
Dec 27 02:25:11 coyote mount.smbfs[22699]: [2004/12/27 02:25:11, 0] 
tdb/tdbutil.c:tdb_log(725)
Dec 27 02:25:11 coyote mount.smbfs[22699]:   
tdb(/var/cache/samba/gencache.tdb): tdb_lock failed on list 10 
ltype=1
 (Bad file descriptor)
Dec 27 02:25:11 coyote mount.smbfs[22699]: [2004/12/27 02:25:11, 0] 
tdb/tdbutil.c:tdb_log(725)
Dec 27 02:25:11 coyote mount.smbfs[22699]:   
tdb(/var/cache/samba/gencache.tdb): tdb_lock failed on list 10 
ltype=1
 (Bad file descriptor)
Dec 27 02:25:11 coyote mount.smbfs[22699]: [2004/12/27 02:25:11, 0] 
client/smbmount.c:do_connection(156)
Dec 27 02:25:11 coyote mount.smbfs[22699]:   22699: Connection to 
shop.coyote.den failed
Dec 27 02:25:40 coyote kernel: smb_add_request: request [dd0c7ec0, 
mid=24027] timed out!
Dec 27 02:26:10 coyote kernel: smb_add_request: request [dd0c7ec0, 
mid=24028] timed out!
Dec 27 02:26:40 coyote kernel: smb_add_request: request [dd0c7ec0, 
mid=24029] timed out!
Dec 27 02:27:10 coyote kernel: smb_add_request: request [dd0c7ec0, 
mid=24030] timed out!
Dec 27 02:27:39 coyote kernel: smb_add_request: request [dd0c7ec0, 
mid=24031] timed out!
Dec 27 02:28:09 coyote kernel: smb_add_request: request [dd0c7ec0, 
mid=24032] timed out!
Dec 27 02:28:39 coyote kernel: smb_add_request: request [dd0c7ec0, 
mid=24033] timed out!
Dec 27 02:29:09 coyote kernel: smb_add_request: request [dd0c7ec0, 
mid=24034] timed out!
Dec 27 02:29:38 coyote kernel: smb_add_request: request [dd0c7ec0, 
mid=24035] timed out!
Dec 27 02:30:08 coyote kernel: smb_add_request: request [dd0c7ec0, 
mid=24036] timed out!
--------------
and this last message continues for another megabyte or so till now.

And the only way I know how to fix this is to reboot to a good kernel, 
2.6.10 apparently has a problem with amanda and samba that earlier 
kernels don't have.  I had 8 days of uptime with Ingo Molnars last 
realtime patch, so guess what I'll be running till this gets fixed...

I've no idea why samba is mucking with amanda, if indeed thats the 
problem, that machine is not part of amanda's backup configuration in 
any way, shape or form.

And, still booted to 2.6.10, I have a script in my init.d dir just 
like the smb one but it starts & stops my local samba shares.
--------
[root@coyote root]# service asmb restart
Stopping share gene:
Stopping share dlds:
stopping share shop:
umount: /mnt/shop: device is busy
umount: /mnt/shop: device is busy
Starting share gene:
Starting share dlds:
Starting share shop:
Could not resolve mount point /mnt/shop
[root@coyote root]# ls /mnt/shop
ls: /mnt/shop: Input/output error
[root@coyote root]#
-------------
So whatever has been done to the kernels smb code in 2.6.10, has 
managed to break that too.  If I reboot to 2.6.10, it will work 
again, for maybe 24 hours, 28 hours is the longest uptime I've 
managed so far, but problems start popping up at about 20-21 hours.

I did try the CIFS code for one build, but its apparently incompatible 
with an older samba on the other two machines of my local network.

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.30% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2004 by Maurice Eugene Heskett, all rights reserved.