Undo aic7xxx changes

Undo aic7xxx changes

[Klaus Dittrich]

I today compiled  2.4.21-rc2 smp with aic79xx-linux-2.4-20030502-tar.gz 
and discovered no problems or hangs. (Tyan-S2665 with AIC-7902)

I haven't had any problems with the driver since I got this 
motherboard starting with aic79xx-linux-2.4-20030318-tar.gz 
and linux-2.4.21-pre5.

-- 
Regards Klaus

Undo aic7xxx changes

[Marcelo Tosatti]

Hi,

I've undone aic7xxx changes which were locking up some machines on
initialization.

The new driver is now named drivers/scsi/aic79xx and is under
CONFIG_AIC79XX.

Justin, unfortunately I can't even THINK about updating aic7xxx to your
new driver at the current release stage. I will do so in the 2.4.22.

The update also contains a PCI posting flush fix from Arjan.

People, please test the driver.

Re: Undo aic7xxx changes

[Justin T. Gibbs]

> Hi,
> 
> I've undone aic7xxx changes which were locking up some machines on
> initialization.

Hmm.  It would have been nice to have the oportunity to fix this correctly.
As it stands now, I have really no idea what people were testing or not
since by taking Alan's patch you have lost the complete change history
and the ability to step people through the changes.  I have preserved
this history in the bk send output that is available on my site if at
some point that is useful to you.

> The new driver is now named drivers/scsi/aic79xx and is under
> CONFIG_AIC79XX.

So we now have an extra copy of the assembler, the Config files, and
the aiclib files.  This is not a solution.  If you wanted to selectively
update the aic79xx driver, all you had to do was ask me for the requisite
change sets.  This is what a mainatiner is for.

> Justin, unfortunately I can't even THINK about updating aic7xxx to your
> new driver at the current release stage. I will do so in the 2.4.22.

Does this mean that you will actually take BK changes form me instead of
from just about anyone else that sends you aic7xxx driver updates?  I had
pretty much given up on this.

> The update also contains a PCI posting flush fix from Arjan.

Which is completely unnecessary and in fact will cause hangs and crashes
on many Dell servers.  The "fix" for the VIA systems that violate the
PCI spec is to either:

1) Update the driver correctly so that it's detection logic will
   automatically disable memory mapped I/O for these broken systems.

or 

2) Just disable the BIOS options that configure the system to violate
   the PCI prefetching rules.

Slowing down all systems, even the ones that are *not broken* by doing
extra, random, PCI read cycles is not a fix.

If you want some verification of the Dell issue (which I'm sure will
cause problems on other "fast" systems too), just ask Matt Domsh.

Again, if you have concerns about the aic7xxx or aic79xx drivers, my
mail box is always open.  Waiting to contact me until the last minute
where I can only sit on the sidelines and watch another train wreck is
not the best way to ensure that the drivers function correctly in 2.4.X.

What this basically boils down to is trust.  If you don't trust me,
tell me how I can build that trust.  Without it, I can only continue
to tell most people that contact me with bug reports, "It's already
fixed in the official driver.  You can pull the latest from ..."

--
Justin

Re: Undo aic7xxx changes

[Stephan von Krawczynski]

On Thu, 08 May 2003 18:45:42 -0600
"Justin T. Gibbs" <gibbs@scsiguy.com> wrote:

> > Hi,
> > [...]
> > Justin, unfortunately I can't even THINK about updating aic7xxx to your
> > new driver at the current release stage. I will do so in the 2.4.22.
> 
> [...]
> Again, if you have concerns about the aic7xxx or aic79xx drivers, my
> mail box is always open.  Waiting to contact me until the last minute
> where I can only sit on the sidelines and watch another train wreck is
> not the best way to ensure that the drivers function correctly in 2.4.X.
> 
> What this basically boils down to is trust.  If you don't trust me,
> tell me how I can build that trust.  Without it, I can only continue
> to tell most people that contact me with bug reports, "It's already
> fixed in the official driver.  You can pull the latest from ..."

Justin, just to complete the picture: as I wrote some days ago concerning your
hint to "use the latest from ..." your latest driver does not complete booting
on (at least) my system but freezes - which I wrote to LKML. I have not yet
heard
anything about this issue. You cannot expect to include a newer driver which
performs obviously worse in some cases.
"Worse" here means "fails" and not "performs bad". Marcelos' decision on the
topic looks pretty reasonable to me...

Regards,
Stephan

Re: Undo aic7xxx changes

[Willy Tarreau]

On Fri, May 09, 2003 at 12:06:48PM +0200, Stephan von Krawczynski wrote:
 
> Justin, just to complete the picture: as I wrote some days ago concerning your
> hint to "use the latest from ..." your latest driver does not complete booting
> on (at least) my system but freezes - which I wrote to LKML. I have not yet
> heard
> anything about this issue. You cannot expect to include a newer driver which
> performs obviously worse in some cases.
> "Worse" here means "fails" and not "performs bad". Marcelos' decision on the
> topic looks pretty reasonable to me...

What's your setup ? Are you in SMP ? I was hit by a lock bug introduced near
6.2.30, which Justin fixed recently and included in his latest driver
(20030502). Justin suggested to me to try the NMI watchdog to find what was
wrong and it pointed us to a spinlock problem. Have you tried to debug
something ? I must say that this driver seems really robust now on my setup
(dual athlon), but perhaps your problem is of the same order and could be fixed
easily with some help, which would be good for you and everyone else.

Regards,
Willy

Re: Undo aic7xxx changes

[Stephan von Krawczynski]

On Fri, 9 May 2003 14:06:59 +0200
Willy Tarreau <willy@w.ods.org> wrote:

> On Fri, May 09, 2003 at 12:06:48PM +0200, Stephan von Krawczynski wrote:
>  
> > Justin, just to complete the picture: as I wrote some days ago concerning
> > your hint to "use the latest from ..." your latest driver does not complete
> > booting on (at least) my system but freezes - which I wrote to LKML. I have
> > not yet heard
> > anything about this issue. You cannot expect to include a newer driver
> > which performs obviously worse in some cases.
> > "Worse" here means "fails" and not "performs bad". Marcelos' decision on
> > the topic looks pretty reasonable to me...
> 
> What's your setup ? Are you in SMP ?

SMP PIII 1.4 GHz, dual Adaptec AIC-7899P U160/m (rev 01)

> I was hit by a lock bug introduced near
> 6.2.30, which Justin fixed recently and included in his latest driver
> (20030502). Justin suggested to me to try the NMI watchdog to find what was
> wrong and it pointed us to a spinlock problem. Have you tried to debug
> something ?

I cannot say which version of the driver it was, the only thing I can tell you
is that the archive was called aic79xx-linux-2.4-20030410-tar.gz.

> I must say that this driver seems really robust now on my setup
> (dual athlon), but perhaps your problem is of the same order and could be
> fixed easily with some help, which would be good for you and everyone else.

I can't tell, basic problem in my setup is that it seems virtually impossible
to bring some 100GB of data onto a streamer connected to the above aic. It
crashes almost every day with a freeze and no oops or other message. I am at
the moment willing to await 2.4.21 and see, and if that does not solve it, then
I will probably go back to a dual symbios controller which I used before and
never had any glitches with.
This is a system in production and not particularly useful for debugging a lot
and correspoding downtime.

Regards,
Stephan

Re: Undo aic7xxx changes

[Willy Tarreau]

On Fri, May 09, 2003 at 03:02:07PM +0200, Stephan von Krawczynski wrote:
 
> I cannot say which version of the driver it was, the only thing I can tell you
> is that the archive was called aic79xx-linux-2.4-20030410-tar.gz.

That's really interesting, because I got the bug since around this version
(20030417 IIRC), and it locked up only on SMP, sometimes during boot, or
during heavy disk accesses caused by "updatedb" and "make -j dep". It's
fixed in 20030502 from http://people.freebsd.org/~gibbs/linux/SRC/

> I can't tell, basic problem in my setup is that it seems virtually impossible
> to bring some 100GB of data onto a streamer connected to the above aic. It
> crashes almost every day with a freeze and no oops or other message.

I had the same symptom which is very frustrating, I agree. I even had
difficulties to catch the NMI watchdog output which was often truncated.

> I am at the moment willing to await 2.4.21 and see, and if that does not solve it,

Well, would you at least agree to retest current version from the above URL ?
I find it a bit of a shame that the driver goes back in -rc stage.

Marcelo, do you have some information about the setup from the people who reported
hangs to you ? Perhaps we could even ask them to confirm that Justin's updated
driver fixes their problems ?

> This is a system in production and not particularly useful for debugging a lot
> and correspoding downtime.

I certainly can understand ;-)

Regards,
Willy

Re: Undo aic7xxx changes

[Stephan von Krawczynski]

On Fri, 9 May 2003 15:27:57 +0200
Willy Tarreau <willy@w.ods.org> wrote:

> On Fri, May 09, 2003 at 03:02:07PM +0200, Stephan von Krawczynski wrote:
>  
> > I cannot say which version of the driver it was, the only thing I can tell
> > you is that the archive was called aic79xx-linux-2.4-20030410-tar.gz.
> 
> That's really interesting, because I got the bug since around this version
> (20030417 IIRC), and it locked up only on SMP, sometimes during boot, or
> during heavy disk accesses caused by "updatedb" and "make -j dep". It's
> fixed in 20030502 from http://people.freebsd.org/~gibbs/linux/SRC/

I tried to merge the latest aic archive into 2.4.21-rc2, besides the "usual"
signed/unsigned warnings I got this one:

aic7xxx_osm.c: In function `ahc_linux_map_seg':
aic7xxx_osm.c:770: warning: integer constant is too large for "long" type

FYI

-- 
Regards,
Stephan

Re: Undo aic7xxx changes

[Willy Tarreau]

On Fri, May 09, 2003 at 03:46:37PM +0200, Stephan von Krawczynski wrote:
> On Fri, 9 May 2003 15:27:57 +0200
> Willy Tarreau <willy@w.ods.org> wrote:
> 
> > On Fri, May 09, 2003 at 03:02:07PM +0200, Stephan von Krawczynski wrote:
> >  
> > > I cannot say which version of the driver it was, the only thing I can tell
> > > you is that the archive was called aic79xx-linux-2.4-20030410-tar.gz.
> > 
> > That's really interesting, because I got the bug since around this version
> > (20030417 IIRC), and it locked up only on SMP, sometimes during boot, or
> > during heavy disk accesses caused by "updatedb" and "make -j dep". It's
> > fixed in 20030502 from http://people.freebsd.org/~gibbs/linux/SRC/
> 
> I tried to merge the latest aic archive into 2.4.21-rc2, besides the "usual"
> signed/unsigned warnings I got this one:
> 
> aic7xxx_osm.c: In function `ahc_linux_map_seg':
> aic7xxx_osm.c:770: warning: integer constant is too large for "long" type

Good catch, but in fact, it's more this line which worries me :

758:                if ((addr ^ (addr + len - 1)) & ~0xFFFFFFFF) {

I don't see how ~0xFFFFFFFF can be non-null on 32 bits archs, because addr is
a bus_addr_t which is in turn dma_addr_t which itself is u32. So unless I don't
find the trick this would mean that this code should never be executed. Perhaps
~0xFFFFFFFFULL would be more appropriate, or even >0xFFFFFFFF, since this can be
detected with u32 using the carry left by the addition.

Regards,
Willy

Re: Undo aic7xxx changes

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 251 bytes --]

> ull on 32 bits archs, because addr is
> a bus_addr_t which is in turn dma_addr_t which itself is u32. So unless I don't
> find the trick this would mean that this code should never be executed. Perhaps

dma_addr_t is either u32 or u64 on x86


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Undo aic7xxx changes

[Willy Tarreau]

On Fri, May 09, 2003 at 05:08:03PM +0200, Arjan van de Ven wrote:
> > ull on 32 bits archs, because addr is
> > a bus_addr_t which is in turn dma_addr_t which itself is u32. So unless I don't
> > find the trick this would mean that this code should never be executed. Perhaps
> 
> dma_addr_t is either u32 or u64 on x86

Yes Arjan, but it's u64 only if CONFIG_HIGHMEM is set. So I repost my question
in another way : is this code supposed to be executed when CONFIG_HIGHMEM=n
since (u32)(~0xFFFFFFFF) = 0 ?

Regards,
Willy

Re: Undo aic7xxx changes

[Andreas Schwab]

Willy Tarreau <willy@w.ods.org> writes:

|> On Fri, May 09, 2003 at 03:46:37PM +0200, Stephan von Krawczynski wrote:
|> > On Fri, 9 May 2003 15:27:57 +0200
|> > Willy Tarreau <willy@w.ods.org> wrote:
|> > 
|> > > On Fri, May 09, 2003 at 03:02:07PM +0200, Stephan von Krawczynski wrote:
|> > >  
|> > > > I cannot say which version of the driver it was, the only thing I can tell
|> > > > you is that the archive was called aic79xx-linux-2.4-20030410-tar.gz.
|> > > 
|> > > That's really interesting, because I got the bug since around this version
|> > > (20030417 IIRC), and it locked up only on SMP, sometimes during boot, or
|> > > during heavy disk accesses caused by "updatedb" and "make -j dep". It's
|> > > fixed in 20030502 from http://people.freebsd.org/~gibbs/linux/SRC/
|> > 
|> > I tried to merge the latest aic archive into 2.4.21-rc2, besides the "usual"
|> > signed/unsigned warnings I got this one:
|> > 
|> > aic7xxx_osm.c: In function `ahc_linux_map_seg':
|> > aic7xxx_osm.c:770: warning: integer constant is too large for "long" type
|> 
|> Good catch, but in fact, it's more this line which worries me :
|> 
|> 758:                if ((addr ^ (addr + len - 1)) & ~0xFFFFFFFF) {
|> 
|> I don't see how ~0xFFFFFFFF can be non-null on 32 bits archs

It will always be zero even on 64 bit archs, because ~0xFFFFFFFF is of
type unsigned int.  The context doesn't matter.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux AG, Deutschherrnstr. 15-19, D-90429 Nürnberg
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Undo aic7xxx changes

[William Lee Irwin III]

On Fri, May 09, 2003 at 04:56:21PM +0200, Willy Tarreau wrote:
> I don't see how ~0xFFFFFFFF can be non-null on 32 bits archs, because addr is
> a bus_addr_t which is in turn dma_addr_t which itself is u32. So unless I don't
> find the trick this would mean that this code should never be executed. Perhaps
> ~0xFFFFFFFFULL would be more appropriate, or even >0xFFFFFFFF, since this can be
> detected with u32 using the carry left by the addition.

include/asm-i386/types.h line 55

#ifdef CONFIG_HIGHMEM
typedef u64 dma_addr_t;
#else
typedef u32 dma_addr_t;
#endif


-- wli

Re: Undo aic7xxx changes

[Stephan von Krawczynski]

On Fri, 9 May 2003 15:27:57 +0200
Willy Tarreau <willy@w.ods.org> wrote:

> Well, would you at least agree to retest current version from the above URL ?
> I find it a bit of a shame that the driver goes back in -rc stage.

Ok, I can tell you at least this: it boots. Just did it. I can tell tomorrow
how it behaves with my specific problem.

This is a setup with 2.4.21-rc2 and aic79xx-linux-2.4-20030502-tar.gz.

-- 
Regards,
Stephan

Re: Undo aic7xxx changes

[Willy Tarreau]

On Fri, May 09, 2003 at 04:11:06PM +0200, Stephan von Krawczynski wrote:
> On Fri, 9 May 2003 15:27:57 +0200
> Willy Tarreau <willy@w.ods.org> wrote:
> 
> > Well, would you at least agree to retest current version from the above URL ?
> > I find it a bit of a shame that the driver goes back in -rc stage.
> 
> Ok, I can tell you at least this: it boots. Just did it. I can tell tomorrow
> how it behaves with my specific problem.

Thanks for having tried ;-)

Willy

Re: Undo aic7xxx changes

[Stephan von Krawczynski]

On Fri, 9 May 2003 16:57:38 +0200
Willy Tarreau <willy@w.ods.org> wrote:

> On Fri, May 09, 2003 at 04:11:06PM +0200, Stephan von Krawczynski wrote:
> > On Fri, 9 May 2003 15:27:57 +0200
> > Willy Tarreau <willy@w.ods.org> wrote:
> > 
> > > Well, would you at least agree to retest current version from the above
> > > URL ? I find it a bit of a shame that the driver goes back in -rc stage.
> > 
> > Ok, I can tell you at least this: it boots. Just did it. I can tell
> > tomorrow how it behaves with my specific problem.
> 
> Thanks for having tried ;-)

Hello all,

I have tried 2.4.21-rc2 with aic79xx-linux-2.4-20030502-tar.gz for three days
now and have to say it performs well. I had no freezes any more and nothing
weird happening. Everything is smooth and ok. This is the best performance I
have seen comparing all 2.4.21-X versions tested.

Thanks a lot.

I will proceed with further stress tests...

Regards,
Stephan

Re: Undo aic7xxx changes

[Marc-Christian Petersen]

On Monday 12 May 2003 11:02, Stephan von Krawczynski wrote:

> I have tried 2.4.21-rc2 with aic79xx-linux-2.4-20030502-tar.gz for three
> days now and have to say it performs well. I had no freezes any more and
> nothing weird happening. Everything is smooth and ok. This is the best
> performance I have seen comparing all 2.4.21-X versions tested.
>
> Thanks a lot.
same here. 0 Problems at all.

ciao, Marc

Re: Undo aic7xxx changes

[Willy Tarreau]

Hi All,

On Mon, May 12, 2003 at 11:02:18AM +0200, Stephan von Krawczynski wrote:

> I have tried 2.4.21-rc2 with aic79xx-linux-2.4-20030502-tar.gz for three days
> now and have to say it performs well. I had no freezes any more and nothing
> weird happening. Everything is smooth and ok. This is the best performance I
> have seen comparing all 2.4.21-X versions tested.

Same here, it seems rock solid on my dual athlon and has survived several
hours of 5 simultaneous make -j 8 bzImage modules with swapping. Definitely the
most stable for me since I've switched from Doug's to Justin's driver.

Marcelo, would it be unreasonable to include it in -rc3 ? After all, it would
not be a radical update, since it was removed from -rc2 ? Just a few bug fixes.

What do you think ?

Regards,
Willy

Re: Undo aic7xxx changes

[Stephan von Krawczynski]

On Mon, 12 May 2003 11:02:18 +0200
Stephan von Krawczynski <skraw@ithnet.com> wrote:

> On Fri, 9 May 2003 16:57:38 +0200
> Willy Tarreau <willy@w.ods.org> wrote:
> 
> > On Fri, May 09, 2003 at 04:11:06PM +0200, Stephan von Krawczynski wrote:
> > > On Fri, 9 May 2003 15:27:57 +0200
> > > Willy Tarreau <willy@w.ods.org> wrote:
> > > 
> > > > Well, would you at least agree to retest current version from the above
> > > > URL ? I find it a bit of a shame that the driver goes back in -rc
> > > > stage.
> > > 
> > > Ok, I can tell you at least this: it boots. Just did it. I can tell
> > > tomorrow how it behaves with my specific problem.
> > 
> > Thanks for having tried ;-)
> 
> Hello all,
> 
> I have tried 2.4.21-rc2 with aic79xx-linux-2.4-20030502-tar.gz for three days
> now and have to say it performs well. I had no freezes any more and nothing
> weird happening. Everything is smooth and ok. This is the best performance I
> have seen comparing all 2.4.21-X versions tested.
> 
> Thanks a lot.
> 
> I will proceed with further stress tests...

Ok. I managed to crash the tested machine after 14 days now. The crash itself
is exactly like former 2.4.21-X. It just freezes, no oops no nothing. It looks
like things got better, but not solved.

Regards,
Stephan

Re: Undo aic7xxx changes

[Justin T. Gibbs]

> Ok. I managed to crash the tested machine after 14 days now. The crash itself
> is exactly like former 2.4.21-X. It just freezes, no oops no nothing. It looks
> like things got better, but not solved.

What is telling you that the freeze is SCSI related?  Are you running
with the nmi watchdog and have a trace?  Do you have driver messages
that you aren't sharing?

--
Justin

Re: Undo aic7xxx changes

[Stephan von Krawczynski]

On Fri, 23 May 2003 06:58:41 -0600
"Justin T. Gibbs" <gibbs@scsiguy.com> wrote:

> > Ok. I managed to crash the tested machine after 14 days now. The crash
> > itself is exactly like former 2.4.21-X. It just freezes, no oops no
> > nothing. It looks like things got better, but not solved.
> 
> What is telling you that the freeze is SCSI related?  Are you running
> with the nmi watchdog and have a trace?  Do you have driver messages
> that you aren't sharing?

Hello Justin,

to make that clear: I am in no way sure _what_ is causing the problem. I am
only updating the (very few) infos I gave/could give during the last weeks.
>From looking at the ongoings I would say your driver patch (URL already sent
several times) made things better. This does obviously not mean that the
kernel-included aic-driver is the sole cause of the troubles.

I am in fact very pleased that rc2/aic-20030502 made things quite noticably
better than every 21-rc/pre before.

What I am giving is a positive feedback, but I have as few logs for it as I had
for the very negative I sent times ago.

Anyway, I am continuing with stress-tests on rc3/aic-20030520. 

Regards,
Stephan

Re: Undo aic7xxx changes

[Willy Tarreau]

Hello !

On Fri, May 23, 2003 at 06:58:41AM -0600, Justin T. Gibbs wrote:
> > Ok. I managed to crash the tested machine after 14 days now. The crash itself
> > is exactly like former 2.4.21-X. It just freezes, no oops no nothing. It looks
> > like things got better, but not solved.
> 
> What is telling you that the freeze is SCSI related?  Are you running
> with the nmi watchdog and have a trace?  Do you have driver messages
> that you aren't sharing?

Stephen,

Justin is right, you should run it through the NMI watchdog, in the hope to
find something useful. If it hangs again in 14 days, you won't know why and
that may be frustrating. With the NMI watchdog, you at least have a chance to
see where it locks up, and you may find it to be within the driver, which would
help Justin stabilize it, or within any other kernel subsystem.

I had to use nmi_watchdog=2 at boot time, but other people use 1.

Regards,
Willy

Re: Undo aic7xxx changes

[Stephan von Krawczynski]

On Fri, 23 May 2003 21:57:57 +0200
Willy Tarreau <willy@w.ods.org> wrote:

> Hello !
> 
> On Fri, May 23, 2003 at 06:58:41AM -0600, Justin T. Gibbs wrote:
> > > Ok. I managed to crash the tested machine after 14 days now. The crash
> > > itself is exactly like former 2.4.21-X. It just freezes, no oops no
> > > nothing. It looks like things got better, but not solved.
> > 
> > What is telling you that the freeze is SCSI related?  Are you running
> > with the nmi watchdog and have a trace?  Do you have driver messages
> > that you aren't sharing?
> 
> Stephen,
> 
> Justin is right, you should run it through the NMI watchdog, in the hope to
> find something useful. If it hangs again in 14 days, you won't know why and
> that may be frustrating. With the NMI watchdog, you at least have a chance to
> see where it locks up, and you may find it to be within the driver, which
> would help Justin stabilize it, or within any other kernel subsystem.
> 
> I had to use nmi_watchdog=2 at boot time, but other people use 1.
> 
> Regards,
> Willy

Hello Willy,

I will do that, but I am not so confident about this, because the box runs X
and a console oops output from nmi may as well not be visible nor written to
disk.

Regards,
Stephan

Re: Undo aic7xxx changes

[Willy Tarreau]

On Sat, May 24, 2003 at 12:52:52PM +0200, Stephan von Krawczynski wrote:
> On Fri, 23 May 2003 21:57:57 +0200
> Willy Tarreau <willy@w.ods.org> wrote:
> 
> > Hello !
> > 
> > On Fri, May 23, 2003 at 06:58:41AM -0600, Justin T. Gibbs wrote:
> > > > Ok. I managed to crash the tested machine after 14 days now. The crash
> > > > itself is exactly like former 2.4.21-X. It just freezes, no oops no
> > > > nothing. It looks like things got better, but not solved.
> > > 
> > > What is telling you that the freeze is SCSI related?  Are you running
> > > with the nmi watchdog and have a trace?  Do you have driver messages
> > > that you aren't sharing?
> > 
> > Stephen,
> > 
> > Justin is right, you should run it through the NMI watchdog, in the hope to
> > find something useful. If it hangs again in 14 days, you won't know why and
> > that may be frustrating. With the NMI watchdog, you at least have a chance to
> > see where it locks up, and you may find it to be within the driver, which
> > would help Justin stabilize it, or within any other kernel subsystem.
> > 
> > I had to use nmi_watchdog=2 at boot time, but other people use 1.
> > 
> > Regards,
> > Willy
> 
> Hello Willy,
> 
> I will do that, but I am not so confident about this, because the box runs X
> and a console oops output from nmi may as well not be visible nor written to
> disk.

OK, I understand. Other options are : serial console (worked for me after
several retries), remote syslogd (sometimes works if the system can still
schedule a bit), or patches such as netconsole, which sends the logs to a
remote host, and kmsgdump which tries to get them onto a floppy after a
panic or a forced dump.

Regards,
Willy

Re: Undo aic7xxx changes

[Stephan von Krawczynski]

On Sat, 24 May 2003 13:16:08 +0200
Willy Tarreau <willy@w.ods.org> wrote:

> > Hello Willy,
> > 
> > I will do that, but I am not so confident about this, because the box runs
> > X and a console oops output from nmi may as well not be visible nor written
> > to disk.
> 
> OK, I understand. Other options are : serial console (worked for me after
> several retries), remote syslogd (sometimes works if the system can still
> schedule a bit), or patches such as netconsole, which sends the logs to a
> remote host, and kmsgdump which tries to get them onto a floppy after a
> panic or a forced dump.
> 
> Regards,
> Willy

Hello all,

it did not take really long for rc3+aic20030520 to freeze - exactly one day.

Though I used nmi_watchdog there are no presentable outputs. As I expected the
screen simply is black and no messages are in any logfiles.
Again it froze while tar-ing about 80 GB of data onto an aic-driven SDLT. Data
is coming from IDE drive connected to a 3ware 7500-8 (though no raid
configuration). 

I conclude that rc2+aic20030502 was way better.

Ah yes, one more thing: I can ping the box, but keyboard, mouse, display is
dead and usually working processes stopped (like snmp).

Willy: I am willing to try a serial console setup (as it does not interfere
with X). I have tried this before with no luck. Can you provide some hints how
you got that working (yes, I read Documentation/serial-console.txt, but I could
not manage any output on the serial line).

Regards,
Stephan

Re: Undo aic7xxx changes

[Willy TARREAU]

Hello !

On Sun, May 25, 2003 at 12:58:11PM +0200, Stephan von Krawczynski wrote:
> it did not take really long for rc3+aic20030520 to freeze - exactly one day.

Well, in some ways, it will be easier to debug it than when it took 14 days, if
it's the same bug, of course.

> Though I used nmi_watchdog there are no presentable outputs. As I expected the
> screen simply is black and no messages are in any logfiles.
> Again it froze while tar-ing about 80 GB of data onto an aic-driven SDLT. Data
> is coming from IDE drive connected to a 3ware 7500-8 (though no raid
> configuration). 

OK, so there's a high probability that the problem is related to either SCSI or
IDE (or both), and less likely implies any other parts.

> Ah yes, one more thing: I can ping the box, but keyboard, mouse, display is
> dead and usually working processes stopped (like snmp).

that's surprizing, mine was completely dead IIRC. It's like it doesn't schedule
anymore but still processes interrupts. I don't know if a deadlock can cause
this behaviour.

> Willy: I am willing to try a serial console setup (as it does not interfere
> with X). I have tried this before with no luck. Can you provide some hints how
> you got that working (yes, I read Documentation/serial-console.txt, but I could
> not manage any output on the serial line).

I had to try several times, because the freeze was so sudden that I often 
caught only a few chars. Justin even didn't believe me. First, you have to
check that CONFIG_SERIAL_CONSOLE is enabled. After that, you'll need a remote
console which can work at high speeds (I could get interesting results at 38400
bps). Surprizingly, above I had mangled output. Perhaps my cable wasn't good
enough (flat cisco RJ45 console cable). I also disabled hard and soft flow
control. But as I already stated, in my case it was easier because it froze
every 2-3 boots, and when it didn't I only had to start a "make -j dep" to get
it. So if I got frozen with no messages, I simply hit the reset button and tried
again. It seems more complicated in your case (although your big tar may be
helping).

When your setup seems OK, you should test it to be sure. I often use "mdir"
with nothing in the drive, or AltGr-SysRq-P to get console messages. If you
don't see anything on your serial console, then your setup is not ready yet
for a test.

Oh and by the way, if you're using modules, you may find interesting to keep
copies of lsmod output, and /proc/ksyms to get a more accurate decoding with a
further ksymoops.

If you really cannot catch anything, I suggest one of these solutions :
  - apply the netconsole patch and have a linux box on the same lan with the
    netconsole server. You can find it in -aa kernels for example.
  - apply the kmsgdump patch, only if you have a floppy drive or a parallel
    printer. It will try to reset the system after a panic, and use bios calls
    to write the kernel messages buffer on the media. This usually works, but
    there are some corner cases where it doesn't. But it's easy to try with
    AltGr-SysRq-D. Download it from http://w.ods.org/tools/kmsgdump/

Good luck !
Willy

Re: Undo aic7xxx changes

[Marc-Christian Petersen]

On Sunday 25 May 2003 12:58, Stephan von Krawczynski wrote:

Hi Stephan,

> Though I used nmi_watchdog there are no presentable outputs. As I expected
> the screen simply is black and no messages are in any logfiles.
> Again it froze while tar-ing about 80 GB of data onto an aic-driven SDLT.
> Data is coming from IDE drive connected to a 3ware 7500-8 (though no raid
> configuration).
>
> I conclude that rc2+aic20030502 was way better.
> Ah yes, one more thing: I can ping the box, but keyboard, mouse, display is
> dead and usually working processes stopped (like snmp).
> Willy: I am willing to try a serial console setup (as it does not interfere
> with X). I have tried this before with no luck. Can you provide some hints
> how you got that working (yes, I read Documentation/serial-console.txt, but
> I could not manage any output on the serial line).
before trying this, could you please update to aic20030523? Thank you.


ciao, Marc

Re: Undo aic7xxx changes

[Stephan von Krawczynski]

On Sun, 25 May 2003 14:47:56 +0200
Marc-Christian Petersen <m.c.p@wolk-project.de> wrote:

> On Sunday 25 May 2003 12:58, Stephan von Krawczynski wrote:
> 
> Hi Stephan,

> before trying this, could you please update to aic20030523? Thank you.

Is there a changelog somewhere? What is the difference between 20030520 and 20030523 ?

Regards,
Stephan

Re: Undo aic7xxx changes

[Marc-Christian Petersen]

On Sunday 25 May 2003 15:50, Stephan von Krawczynski wrote:

Hi Stephan,

> > before trying this, could you please update to aic20030523? Thank you.
> Is there a changelog somewhere? What is the difference between 20030520 and
> 20030523 ?
yes, there is a changelog. Unfortunately in the tar.gz package because the one 
on Justins website isn't up2date. I've made it available on my website.

http://wolk.sf.net/tmp/AIC-CHANGELOG

ciao, Marc

Re: Undo aic7xxx changes

[Geller Sandor]

On Sun, 25 May 2003, Stephan von Krawczynski wrote:

> Is there a changelog somewhere? What is the difference between 20030520
> and 20030523 ?

See drivers/scsi/aic7xxx/CHANGELOG

  Geller Sandor <wildy@petra.hos.u-szeged.hu>

Re: Undo aic7xxx changes

[Stephan von Krawczynski]

On Sun, 25 May 2003 14:47:56 +0200
Marc-Christian Petersen <m.c.p@wolk-project.de> wrote:

> On Sunday 25 May 2003 12:58, Stephan von Krawczynski wrote:
> 
> Hi Stephan,
> before trying this, could you please update to aic20030523? Thank you.
> 
> 
> ciao, Marc

Hello Marc,

I did this. The combination rc3+aic20030523 survived the first day of tests. So
it seems at least better than rc3+aic20030520.

I'll keep you informed.

Regards,
Stephan

Re: Undo aic7xxx changes

[Willy Tarreau]

On Mon, May 26, 2003 at 05:00:58PM +0200, Stephan von Krawczynski wrote:
> On Sun, 25 May 2003 14:47:56 +0200
> Marc-Christian Petersen <m.c.p@wolk-project.de> wrote:
> 
> > On Sunday 25 May 2003 12:58, Stephan von Krawczynski wrote:
> > 
> > Hi Stephan,
> > before trying this, could you please update to aic20030523? Thank you.
> > 
> > 
> > ciao, Marc
> 
> Hello Marc,
> 
> I did this. The combination rc3+aic20030523 survived the first day of tests. So
> it seems at least better than rc3+aic20030520.

The same has been running on my Alpha since yesterday evening on a 54GB raid0
which I transformed to raid5 (39 GB backed up to IDE ; mkraid ; 39GB restored).

Still alive.
Cheers,
Willy

Re: Undo aic7xxx changes

[Stephan von Krawczynski]

Hello Marcelo,

I tried plain rc6 now and have to tell you it does not survive a single day of
my usual tests. It freezes during tar from 3ware-driven IDE to aic-driven SDLT.
This is identical to all previous rc (and some pre) releases of 2.4.21. So far
I can tell you that the only thing that has recently cured this problem is
replacing the aic-driver with latest of justins' releases.
As plain rc6 does definitely not work I will now switch over to
rc6+aic-20030523. Remember that rc3+aic-20030523 already worked quite ok (4
days test survived).

My personal opinion is a known-to-be-broken 2.4.21 should not be released, as a
lot of people only try/use the releases and therefore an immediately released
2.4.22-pre1 with justins driver will not be a good solution.

Regards,
Stephan

Re: Undo aic7xxx changes

[Marc-Christian Petersen]

On Friday 30 May 2003 10:09, Stephan von Krawczynski wrote:

Hi Stephan,

> I tried plain rc6 now and have to tell you it does not survive a single day
> of my usual tests. It freezes during tar from 3ware-driven IDE to
> aic-driven SDLT. This is identical to all previous rc (and some pre)
> releases of 2.4.21. So far I can tell you that the only thing that has
> recently cured this problem is replacing the aic-driver with latest of
> justins' releases.
> As plain rc6 does definitely not work I will now switch over to
> rc6+aic-20030523. Remember that rc3+aic-20030523 already worked quite ok (4
> days test survived).
same experience on my boxen (quite much with AIC)

> My personal opinion is a known-to-be-broken 2.4.21 should not be released,
> as a lot of people only try/use the releases and therefore an immediately
> released 2.4.22-pre1 with justins driver will not be a good solution.
ACK!

Maybe we should disable AIC Config option and instead add a comment like:

comment 'For AICXXXX, please go to http://people.freebsd.org/~gibbs/linux/'
comment 'and download the latest tar.gz and unpack these drivers!'
comment 'After unpacking, enable Config.in option in drivers/scsi/Config.in'

*scnr* ;)

ciao, Marc

Re: Undo aic7xxx changes

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 555 bytes --]



> My personal opinion is a known-to-be-broken 2.4.21 should not be released, as a
> lot of people only try/use the releases and therefore an immediately released
> 2.4.22-pre1 with justins driver will not be a good solution.

I think you missed the point entirely before. 2.4.21 CANNOT cause
regressions most of all. At this point there is no way to know if the
thing that fixes your machine breaks on 100s others that DO work
correctly in 2.4.20. Even if it would fix 100s and break 1 it's still
not acceptable for stable kernel releases.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Undo aic7xxx changes

[Stephan von Krawczynski]

On 30 May 2003 10:21:33 +0200
Arjan van de Ven <arjanv@redhat.com> wrote:

> 
> 
> > My personal opinion is a known-to-be-broken 2.4.21 should not be released,
> > as a lot of people only try/use the releases and therefore an immediately
> > released 2.4.22-pre1 with justins driver will not be a good solution.
> 
> I think you missed the point entirely before. 2.4.21 CANNOT cause
> regressions most of all. At this point there is no way to know if the
> thing that fixes your machine breaks on 100s others that DO work
> correctly in 2.4.20. Even if it would fix 100s and break 1 it's still
> not acceptable for stable kernel releases.

Unfortunately you miss my point (which is probably too simple to be clearly
visible):
I want to give some feedback on a topic/problem I am experiencing since _long_.
I was _asked_ to do so. Additionally I am stating my _opinion_. I am _not_
telling anybody what to do. I am not in a position to do so. Very likely only
_few_ people are in such a position, very likely the maintainer of aic and
hopefully Marcelo.
Have you read all available bug reports Justin got? If you have not, don't play
with numbers.

Another personal opinion: software development tends to make things possible
that "cannot be". ;-)

Regards,
Stephan

Re: Undo aic7xxx changes

[Jeff Garzik]

On Fri, May 30, 2003 at 10:09:00AM +0200, Stephan von Krawczynski wrote:
> Hello Marcelo,
> 
> I tried plain rc6 now and have to tell you it does not survive a single day of
> my usual tests. It freezes during tar from 3ware-driven IDE to aic-driven SDLT.
> This is identical to all previous rc (and some pre) releases of 2.4.21. So far
> I can tell you that the only thing that has recently cured this problem is
> replacing the aic-driver with latest of justins' releases.

So Justin's driver fixes your 3ware problems???

And exactly what -rc/-pre release stopped working for you?

	Jeff

Re: Undo aic7xxx changes

[Stephan von Krawczynski]

On Fri, 30 May 2003 09:34:56 -0400
Jeff Garzik <jgarzik@pobox.com> wrote:

> On Fri, May 30, 2003 at 10:09:00AM +0200, Stephan von Krawczynski wrote:
> > Hello Marcelo,
> > 
> > I tried plain rc6 now and have to tell you it does not survive a single day
> > of my usual tests. It freezes during tar from 3ware-driven IDE to
> > aic-driven SDLT. This is identical to all previous rc (and some pre)
> > releases of 2.4.21. So far I can tell you that the only thing that has
> > recently cured this problem is replacing the aic-driver with latest of
> > justins' releases.
> 
> So Justin's driver fixes your 3ware problems???

This is _no_ 3ware problem. As I told you data comes from 3ware and goes to
aic. The problem occurs if using plain-version aic and is gone if using justins
latest releases.
As long as we do nothing with the aic driver there is no problem at all (3ware
works fine here).

> And exactly what -rc/-pre release stopped working for you?

Very good question. I can check, but I need one day per version to check. It
may well be that in fact none of the pre/rc releases worked, we have this box
since about pre3 and to my knowledge we always had the problem. Boy, we were
quite happy when we found out that Justins stuff got it going - it already got
on our nerves quite a bit ;-)

If you want to know about some special kernel release just tell me and I will
try it.

Maybe I should tell again details about the test setup as not all may remember
in this long-lasting thread.
Basically the problem seldomly arises after booting. I have the impression that
this got in fact better over the releases, earlier pre's froze earlier.
what we do:
1) copy around 50 - 100 GB of data via nfs to a 3ware drive (always works well)
2) tar this data on the nfs server from 3ware drive to aic(-driven) SDLT
(quantum)
3) verify the archived data via tar

freezes happen while 2) or 3). If you reboot after 1) they are very rare, never
on any later rc-release.
As this whole things takes time we do it overnight and have a look at the box
next morning. Not a single plain release is ok on the next morning. Checking
the logs we find out it froze in 2) or 3).
If you do exactly the same thing on exactly the same box with exactly the same
data but Justins driver everything is ok (aic-20030523). It was not ok with
aic-20030520 (just to mention this), aic-20030502 was quite ok (survived 14 days).

What else can I tell you?

Regards,
Stephan

Re: Undo aic7xxx changes

[Jeff Garzik]

On Fri, May 30, 2003 at 10:09:00AM +0200, Stephan von Krawczynski wrote:
> Hello Marcelo,
> 
> I tried plain rc6 now and have to tell you it does not survive a single day of
> my usual tests. It freezes during tar from 3ware-driven IDE to aic-driven SDLT.
> This is identical to all previous rc (and some pre) releases of 2.4.21. So far
> I can tell you that the only thing that has recently cured this problem is
> replacing the aic-driver with latest of justins' releases.
> As plain rc6 does definitely not work I will now switch over to
> rc6+aic-20030523. Remember that rc3+aic-20030523 already worked quite ok (4
> days test survived).

Also, does the aic7xxx_old driver work for you?

The "old" part is only in regards to lack of support for very-new
aic7xxx hardware.

	Jeff

Re: Undo aic7xxx changes

[Justin T. Gibbs]

> Willy: I am willing to try a serial console setup (as it does not interfere
> with X).

Are you still running all of your tests with X up?  You then have no chance
of getting any useful diagnostics without a serial console.  Can't you switch
back to a vty while the test is running?

>I have tried this before with no luck. Can you provide some hints how
> you got that working (yes, I read Documentation/serial-console.txt, but
> I could not manage any output on the serial line).

You will need a null modem cable.  Config a kernel with serial console
support enabled.  Use a fairly high speed for your console (115200).  To
enable your first serial port as a console add something like the following
to your kenrel command line:

console=ttyS0,115200 console=vty0

This will retain console output on the vty too.

--
Justin

Re: Undo aic7xxx changes

[Marcelo Tosatti]

On Fri, 23 May 2003, Stephan von Krawczynski wrote:

> On Mon, 12 May 2003 11:02:18 +0200
> Stephan von Krawczynski <skraw@ithnet.com> wrote:
>
> > On Fri, 9 May 2003 16:57:38 +0200
> > Willy Tarreau <willy@w.ods.org> wrote:
> >
> > > On Fri, May 09, 2003 at 04:11:06PM +0200, Stephan von Krawczynski wrote:
> > > > On Fri, 9 May 2003 15:27:57 +0200
> > > > Willy Tarreau <willy@w.ods.org> wrote:
> > > >
> > > > > Well, would you at least agree to retest current version from the above
> > > > > URL ? I find it a bit of a shame that the driver goes back in -rc
> > > > > stage.
> > > >
> > > > Ok, I can tell you at least this: it boots. Just did it. I can tell
> > > > tomorrow how it behaves with my specific problem.
> > >
> > > Thanks for having tried ;-)
> >
> > Hello all,
> >
> > I have tried 2.4.21-rc2 with aic79xx-linux-2.4-20030502-tar.gz for three days
> > now and have to say it performs well. I had no freezes any more and nothing
> > weird happening. Everything is smooth and ok. This is the best performance I
> > have seen comparing all 2.4.21-X versions tested.
> >
> > Thanks a lot.
> >
> > I will proceed with further stress tests...
>
> Ok. I managed to crash the tested machine after 14 days now. The crash itself
> is exactly like former 2.4.21-X. It just freezes, no oops no nothing. It looks
> like things got better, but not solved.
>

What about rc3?

Re: Undo aic7xxx changes

[Stephan von Krawczynski]

On Fri, 23 May 2003 15:30:33 -0300 (BRT)
Marcelo Tosatti <marcelo@conectiva.com.br> wrote:

> What about rc3?

I will inform you if anything bad happens :-)

rc3+aic20030520 tests started today.

Regards,
Stephan