HFSC warning issue

HFSC warning issue

[François Delawarde]

Hi all,

I'm having the same HFSC issues as described here:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=631945

Some user reports that he made it work by reverting this patch:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blobdiff;f=net/sched/sch_hfsc.c;h=47496098d35c2876cd0b5a88e1365922cc369ce9;hp=abd904be428717462395d62adb682023e64e17a5;hb=3b2eb6131e2f6ff646abb0fc69648179b8b70216;hpb=9c01ae58d4fee39e2af5b1379ee5431dd585cf62


Is this the correct solution? Any "clean" patch I could test that would
resolve this issue?

Thanks,
François.

--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: HFSC warning issue

[Michal Soltys]

W dniu 11.07.2011 11:26, François Delawarde pisze:
> Hi all,
> 
> I'm having the same HFSC issues as described here:
> 
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=631945
> 
> Some user reports that he made it work by reverting this patch:
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blobdiff;f=net/sched/sch_hfsc.c;h=47496098d35c2876cd0b5a88e1365922cc369ce9;hp=abd904be428717462395d62adb682023e64e17a5;hb=3b2eb6131e2f6ff646abb0fc69648179b8b70216;hpb=9c01ae58d4fee39e2af5b1379ee5431dd585cf62
> 
> 
> Is this the correct solution? Any "clean" patch I could test that would
> resolve this issue?
> 
> Thanks,
> François.
> 

Hmmm,

This commit in mention is the patch I submitted 
a while ago, namely:

http://kerneltrap.org/mailarchive/linux-netdev/2010/8/30/6284141

Whole rationale and short discussion regarding the 
change is in that thread. 

In a nutshell - it fixed certain scenario which could cause 
bursty traffic, but the change is just a minor adjustment. Let 
me sit a bit on this thing, and why it manages to trigger that 
warning ...

For a quick fix - reverting that commit should be perfectly 
safe - there were no other patches related to this one.

--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: HFSC warning issue

[Michal Soltys]

Missed explanation behind the change:

http://www.spinics.net/lists/netdev/msg139383.html

Re: HFSC warning issue

[François Delawarde]

Not sure if its relevant but it seems that the ones that reported the
issue either use IFB or IMQ devices.

In my case, I use IMQ.

François.

On Mon, 2011-07-11 at 14:11 +0200, Michal Soltys wrote: 
> Missed explanation behind the change:
> 
> http://www.spinics.net/lists/netdev/msg139383.html
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Bug#631945: HFSC warning issue

[00bormoj]

Oops - my logs are again filled with the WARN_ON from
net/sched/sch_hfsc.c:1427. Sorry for the confusion, it looks like
"initialize parent's cl_cfmin properly in init_vf()" is not to blame.

This is an amd64 system. Vanilla kernel.org kernel except with fglrx
(the amd/ati radeon proprietary driver - yes it is a router and an
HTPC...) loaded. I should probably start bisecting then, but it's a
little tough when it can take more than two days to trigger :(

[1115475.034494] WARNING: at net/sched/sch_hfsc.c:1427
hfsc_dequeue+0x155/0x28c [sch_hfsc]()
[1115475.034497] Hardware name: TA780G M2+
[1115475.034498] Modules linked in: ifb act_police sch_ingress cls_u32
sch_sfq sch_hfsc joydev hid_apple hidp binfmt_misc rfcomm bnep
ipt_ULOG cryptd aes_x86_64 aes_generic lirc_serial(C) ipt_MASQUERADE
xt_NOTRACK xt_state xt_dscp xt_length xt_mark nf_conntrack_ipv6
nf_defrag_ipv6 xt_connmark xt_tcpudp ip6t_REJECT ipt_REJECT ip6t_LOG
ipt_LOG xt_limit nf_conntrack_ftp kvm_amd kvm ip6table_raw iptable_raw
iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4
ip6table_mangle sit tunnel4 iptable_mangle ip6table_filter nfsd
ip6_tables ipv6 nfs iptable_filter lockd ip_tables x_tables
auth_rpcgss nfs_acl sunrpc arc4 ecb saa7134_alsa snd_hda_codec_hdmi
snd_hda_codec_realtek fglrx(P) snd_hda_intel snd_hda_codec snd_hwdep
ir_lirc_codec lirc_dev snd_pcm_oss snd_mixer_oss snd_pcm
ir_sony_decoder tda827x snd_seq_dummy snd_seq_oss snd_seq_midi
ir_jvc_decoder rt61pci crc_itu_t snd_rawmidi tda8290 ir_rc6_decoder
snd_seq_midi_event tuner ir_rc5_decoder snd_seq saa7134 rt2x00pci
ir_nec_decoder snd_timer snd_seq_device rt2x00lib rc_core mac80211
videobuf_dma_sg videobuf_core xfs v4l2_common btusb snd bluetooth
videodev v4l2_compat_ioctl32 cfg80211 i2c_piix4 uinput soundcore
cpufreq_ondemand powernow_k8 freq_table edac_core mperf tveeprom evdev
i2c_core pl2303 k8temp edac_mce_amd ppdev usbserial eeprom_93cx6
snd_page_alloc rfkill usblp wmi parport_pc parport processor button
loop ext4 mbcache jbd2 crc16 usb_storage uas usbhid hid dm_mod sg
raid10 sd_mod crc_t10dif raid456 async_pq async_xor pata_acpi xor
async_memcpy async_raid6_recov raid6_pq async_tx raid1 raid0 multipath
linear md_mod btrfs lzo_compress ahci pata_atiixp zlib_deflate libahci
libata thermal thermal_sys hwmon scsi_mod r8169 crc32c mii ehci_hcd
ohci_hcd libcrc32c [last unloaded: ifb]
[1115475.034594] Pid: 3, comm: ksoftirqd/0 Tainted: P        WC  2.6.39.1 #1
[1115475.034596] Call Trace:
[1115475.034600]  [<ffffffff8103b785>] ? warn_slowpath_common+0x78/0x8c
[1115475.034604]  [<ffffffffa0371e1c>] ? hfsc_dequeue+0x155/0x28c [sch_hfsc]
[1115475.034607]  [<ffffffff81244dfa>] ? __qdisc_run+0x93/0x11a
[1115475.034610]  [<ffffffff8122c9bc>] ? net_tx_action+0x134/0x19d
[1115475.034613]  [<ffffffff81041403>] ? __do_softirq+0xff/0x1e0
[1115475.034617]  [<ffffffff81041588>] ? run_ksoftirqd+0xa4/0x18c
[1115475.034620]  [<ffffffff810414e4>] ? __do_softirq+0x1e0/0x1e0
[1115475.034623]  [<ffffffff810414e4>] ? __do_softirq+0x1e0/0x1e0
[1115475.034626]  [<ffffffff81055b04>] ? kthread+0x7a/0x82
[1115475.034630]  [<ffffffff812b4ad4>] ? kernel_thread_helper+0x4/0x10
[1115475.034633]  [<ffffffff81055a8a>] ? kthread_worker_fn+0x14d/0x14d
[1115475.034637]  [<ffffffff812b4ad0>] ? gs_change+0xb/0xb

Re: Bug#631945: HFSC warning issue

[Michal Soltys]

On 11-07-11 17:07, 00bormoj@gmail.com wrote:
> Oops - my logs are again filled with the WARN_ON from
> net/sched/sch_hfsc.c:1427. Sorry for the confusion, it looks like
> "initialize parent's cl_cfmin properly in init_vf()" is not to blame.
>

Ah, ok then. It would be really weird if that was the actual cause. I
have few patches stacked for later, so it made me feel a bit uneasy :)

> This is an amd64 system. Vanilla kernel.org kernel except with fglrx
> (the amd/ati radeon proprietary driver - yes it is a router and an
> HTPC...) loaded. I should probably start bisecting then, but it's a
> little tough when it can take more than two days to trigger :(
>

 From what I can see, and if I haven't missed or misread anything:

That warning that gets triggered when next_time == 0 in
hfsc_schedule_watchdog() implies, that hfsc_dequeue() tried to dequeue a
packet, but no leaf had anything eligible for scheduling (realtime
criterion) and linksharing was upperlimited.

Now if there IS something to dequeue, then I don't see how next_time
could possibly be zero - that would mean there's no packet to schedule
at all. But if that happened, then hfsc_dequeue() would simply exit at
the very beginning due to: if (sch->q.qlen == 0) .

So it does look weird (like if something external messed with hfsc's
qlen) ...

Also note - the changes to sch_hfsc.c were pretty minimal since the last
kernel that worked for you. You might be searching for something else ...


ps.

You don't have to upperlimit every single class (which will cripple
linksharing between siblings) - if your aim is to match some interface's
speed, just set it in the single top class of your class hierarchy.

Re: Bug#631945: HFSC warning issue

[François Delawarde]

On Wed, 2011-07-13 at 01:28 +0200, Michal Soltys wrote: 
> That warning that gets triggered when next_time == 0 in
> hfsc_schedule_watchdog() implies, that hfsc_dequeue() tried to dequeue a
> packet, but no leaf had anything eligible for scheduling (realtime
> criterion) and linksharing was upperlimited.
> 
> Now if there IS something to dequeue, then I don't see how next_time
> could possibly be zero - that would mean there's no packet to schedule
> at all. But if that happened, then hfsc_dequeue() would simply exit at
> the very beginning due to: if (sch->q.qlen == 0) .
> 
> So it does look weird (like if something external messed with hfsc's
> qlen) ...

Could it be related with setting a custom device qlen with iproute (wild
guess)? 

> Also note - the changes to sch_hfsc.c were pretty minimal since the last
> kernel that worked for you. You might be searching for something else ...

Any idea where we could start searching?

Is there any configuration we could try (with upperlimits maybe?) that
could help identify or workaround this problem?

Thanks,
François.

--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: HFSC warning issue

[Michal Pokrywka]

Through serveral days my friend has done some tests with kernels 2.6.39.x

2.6.39.1, 2.6.39.2 and 2.6.39.3 are still affected.

2.6.39 (debian version 2.6.39-1 from http://snapshot.debian.org/binary/?cat=l)
works for several hours without triggering the bug. If it persist for some hours
more we will bisect 2.6.39 and 2.6.39.1 and try to find the related patch.