【Ask for help】 A question for __ib_cache_gid_add()

【Ask for help】 A question for __ib_cache_gid_add()

[oulijun]

Hi
  I am using the ubuntu system(5.0.0 kernel) to test the hip08 NIC port,. When I modify the perr mac1 to mac2,then restore to mac1, it will cause
the gid0 and gid 1 of the roce to be unavailable, and check that the /sys/class/infiniband/hns_0/ports/1/gid_attrs/ndevs/0 is show invalid.
the protocol stack print will appear.

  Oct 16 17:59:36 ubuntu kernel: [200635.496317] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:4600:4dff:fea7:9599 error=-28
Oct 16 17:59:37 ubuntu kernel: [200636.705848] 8021q: adding VLAN 0 to HW filter on device enp189s0f0
Oct 16 17:59:37 ubuntu kernel: [200636.705854] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:4600:4dff:fea7:9599 error=-28
Oct 16 17:59:39 ubuntu kernel: [200638.755828] hns3 0000:bd:00.0 enp189s0f0: link up
Oct 16 17:59:39 ubuntu kernel: [200638.755847] IPv6: ADDRCONF(NETDEV_CHANGE): enp189s0f0: link becomes ready
Oct 16 18:00:56 ubuntu kernel: [200715.699961] hns3 0000:bd:00.0 enp189s0f0: link down
Oct 16 18:00:56 ubuntu kernel: [200716.016142] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:4600:4dff:fea7:95f4 error=-28
Oct 16 18:00:58 ubuntu kernel: [200717.229857] 8021q: adding VLAN 0 to HW filter on device enp189s0f0
Oct 16 18:00:58 ubuntu kernel: [200717.229863] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:4600:4dff:fea7:95f4 error=-28

Has anyone else encounterd a similar problem ? I wonder if the _ib_cache_add_gid() is defective in 5.0 kernel?

Thanks
Lijun

Re: 【Ask for help】 A question for __ib_cache_gid_add()

[Jason Gunthorpe]

On Fri, Nov 01, 2019 at 05:36:36PM +0800, oulijun wrote:
> Hi
>   I am using the ubuntu system(5.0.0 kernel) to test the hip08 NIC port,. When I modify the perr mac1 to mac2,then restore to mac1, it will cause
> the gid0 and gid 1 of the roce to be unavailable, and check that the /sys/class/infiniband/hns_0/ports/1/gid_attrs/ndevs/0 is show invalid.
> the protocol stack print will appear.
> 
>   Oct 16 17:59:36 ubuntu kernel: [200635.496317] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:4600:4dff:fea7:9599 error=-28
> Oct 16 17:59:37 ubuntu kernel: [200636.705848] 8021q: adding VLAN 0 to HW filter on device enp189s0f0
> Oct 16 17:59:37 ubuntu kernel: [200636.705854] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:4600:4dff:fea7:9599 error=-28
> Oct 16 17:59:39 ubuntu kernel: [200638.755828] hns3 0000:bd:00.0 enp189s0f0: link up
> Oct 16 17:59:39 ubuntu kernel: [200638.755847] IPv6: ADDRCONF(NETDEV_CHANGE): enp189s0f0: link becomes ready
> Oct 16 18:00:56 ubuntu kernel: [200715.699961] hns3 0000:bd:00.0 enp189s0f0: link down
> Oct 16 18:00:56 ubuntu kernel: [200716.016142] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:4600:4dff:fea7:95f4 error=-28
> Oct 16 18:00:58 ubuntu kernel: [200717.229857] 8021q: adding VLAN 0 to HW filter on device enp189s0f0
> Oct 16 18:00:58 ubuntu kernel: [200717.229863] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:4600:4dff:fea7:95f4 error=-28
> 
> Has anyone else encounterd a similar problem ? I wonder if the _ib_cache_add_gid() is defective in 5.0 kernel?

Maybe Parav knows?

Jason

Re: 【Ask for help】 A question for __ib_cache_gid_add()

[Mark Zhang]

On 11/1/2019 9:05 PM, Jason Gunthorpe wrote:
> On Fri, Nov 01, 2019 at 05:36:36PM +0800, oulijun wrote:
>> Hi
>>    I am using the ubuntu system(5.0.0 kernel) to test the hip08 NIC port,. When I modify the perr mac1 to mac2,then restore to mac1, it will cause
>> the gid0 and gid 1 of the roce to be unavailable, and check that the /sys/class/infiniband/hns_0/ports/1/gid_attrs/ndevs/0 is show invalid.
>> the protocol stack print will appear.
>>
>>    Oct 16 17:59:36 ubuntu kernel: [200635.496317] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:4600:4dff:fea7:9599 error=-28
>> Oct 16 17:59:37 ubuntu kernel: [200636.705848] 8021q: adding VLAN 0 to HW filter on device enp189s0f0
>> Oct 16 17:59:37 ubuntu kernel: [200636.705854] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:4600:4dff:fea7:9599 error=-28
>> Oct 16 17:59:39 ubuntu kernel: [200638.755828] hns3 0000:bd:00.0 enp189s0f0: link up
>> Oct 16 17:59:39 ubuntu kernel: [200638.755847] IPv6: ADDRCONF(NETDEV_CHANGE): enp189s0f0: link becomes ready
>> Oct 16 18:00:56 ubuntu kernel: [200715.699961] hns3 0000:bd:00.0 enp189s0f0: link down
>> Oct 16 18:00:56 ubuntu kernel: [200716.016142] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:4600:4dff:fea7:95f4 error=-28
>> Oct 16 18:00:58 ubuntu kernel: [200717.229857] 8021q: adding VLAN 0 to HW filter on device enp189s0f0
>> Oct 16 18:00:58 ubuntu kernel: [200717.229863] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:4600:4dff:fea7:95f4 error=-28
>>
>> Has anyone else encounterd a similar problem ? I wonder if the _ib_cache_add_gid() is defective in 5.0 kernel?
> 
> Maybe Parav knows?
> 
One possibility is, during the operation you have port reset; Then
all gids will be deleted and re-created. But during gids re-build
gid entry #0 and #1 is in use, so they cannot be deleted then you
get this error.

> Jason
> 

Re: 【Ask for help】 A question for __ib_cache_gid_add()

[Mark Zhang]

On 11/1/2019 9:21 PM, Mark Zhang wrote:
> On 11/1/2019 9:05 PM, Jason Gunthorpe wrote:
>> On Fri, Nov 01, 2019 at 05:36:36PM +0800, oulijun wrote:
>>> Hi
>>>     I am using the ubuntu system(5.0.0 kernel) to test the hip08 NIC port,. When I modify the perr mac1 to mac2,then restore to mac1, it will cause
>>> the gid0 and gid 1 of the roce to be unavailable, and check that the /sys/class/infiniband/hns_0/ports/1/gid_attrs/ndevs/0 is show invalid.
>>> the protocol stack print will appear.
>>>
>>>     Oct 16 17:59:36 ubuntu kernel: [200635.496317] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:4600:4dff:fea7:9599 error=-28
>>> Oct 16 17:59:37 ubuntu kernel: [200636.705848] 8021q: adding VLAN 0 to HW filter on device enp189s0f0
>>> Oct 16 17:59:37 ubuntu kernel: [200636.705854] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:4600:4dff:fea7:9599 error=-28
>>> Oct 16 17:59:39 ubuntu kernel: [200638.755828] hns3 0000:bd:00.0 enp189s0f0: link up
>>> Oct 16 17:59:39 ubuntu kernel: [200638.755847] IPv6: ADDRCONF(NETDEV_CHANGE): enp189s0f0: link becomes ready
>>> Oct 16 18:00:56 ubuntu kernel: [200715.699961] hns3 0000:bd:00.0 enp189s0f0: link down
>>> Oct 16 18:00:56 ubuntu kernel: [200716.016142] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:4600:4dff:fea7:95f4 error=-28
>>> Oct 16 18:00:58 ubuntu kernel: [200717.229857] 8021q: adding VLAN 0 to HW filter on device enp189s0f0
>>> Oct 16 18:00:58 ubuntu kernel: [200717.229863] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:4600:4dff:fea7:95f4 error=-28
>>>
>>> Has anyone else encounterd a similar problem ? I wonder if the _ib_cache_add_gid() is defective in 5.0 kernel?
>>
>> Maybe Parav knows?
>>
> One possibility is, during the operation you have port reset; Then
> all gids will be deleted and re-created. But during gids re-build
> gid entry #0 and #1 is in use, so they cannot be deleted then you
> get this error.
> 
I mean then you get this error message, which is expected. Check
all valid gid entries before and after the operation.

>> Jason
>>
> 

RE: 【Ask for help】 A question for __ib_cache_gid_add()

[Parav Pandit]

Hi Lijun,

> -----Original Message-----
> From: Jason Gunthorpe <jgg@ziepe.ca>
> Sent: Friday, November 1, 2019 8:06 AM
> To: oulijun <oulijun@huawei.com>; Parav Pandit <parav@mellanox.com>
> Cc: Doug Ledford <dledford@redhat.com>; linux-rdma <linux-
> rdma@vger.kernel.org>
> Subject: Re: 【Ask for help】 A question for __ib_cache_gid_add()
> 
> On Fri, Nov 01, 2019 at 05:36:36PM +0800, oulijun wrote:
> > Hi
> >   I am using the ubuntu system(5.0.0 kernel) to test the hip08 NIC
> > port,. When I modify the perr mac1 to mac2,then restore to mac1, it will
> cause the gid0 and gid 1 of the roce to be unavailable, and check that the
> /sys/class/infiniband/hns_0/ports/1/gid_attrs/ndevs/0 is show invalid.
> > the protocol stack print will appear.
> >
> >   Oct 16 17:59:36 ubuntu kernel: [200635.496317] __ib_cache_gid_add:
> > unable to add gid fe80:0000:0000:0000:4600:4dff:fea7:9599 error=-28
> > Oct 16 17:59:37 ubuntu kernel: [200636.705848] 8021q: adding VLAN 0 to
> > HW filter on device enp189s0f0 Oct 16 17:59:37 ubuntu kernel:
> > [200636.705854] __ib_cache_gid_add: unable to add gid
> > fe80:0000:0000:0000:4600:4dff:fea7:9599 error=-28 Oct 16 17:59:39
> > ubuntu kernel: [200638.755828] hns3 0000:bd:00.0 enp189s0f0: link up
> > Oct 16 17:59:39 ubuntu kernel: [200638.755847] IPv6:
> > ADDRCONF(NETDEV_CHANGE): enp189s0f0: link becomes ready Oct 16
> > 18:00:56 ubuntu kernel: [200715.699961] hns3 0000:bd:00.0 enp189s0f0:
> > link down Oct 16 18:00:56 ubuntu kernel: [200716.016142]
> > __ib_cache_gid_add: unable to add gid
> > fe80:0000:0000:0000:4600:4dff:fea7:95f4 error=-28 Oct 16 18:00:58
> > ubuntu kernel: [200717.229857] 8021q: adding VLAN 0 to HW filter on
> > device enp189s0f0 Oct 16 18:00:58 ubuntu kernel: [200717.229863]
> > __ib_cache_gid_add: unable to add gid
> > fe80:0000:0000:0000:4600:4dff:fea7:95f4 error=-28
> >
> > Has anyone else encounterd a similar problem ? I wonder if the
> _ib_cache_add_gid() is defective in 5.0 kernel?
> 
> Maybe Parav knows?

I used the kernel from [1], which seems to be fine; it has the required commits [2], [3], [4].

Are you running RDMA traffic/applications which are using GID 0 and 1 when changing MAC?
If so, administrative operation such as MAC address change during active RDMA traffic is unsupported, which can lead to this error.
Can you please confirm?

If you are not running RDMA traffic while changing the mac, I need more debug logs.
Can you please enable ftrace and share the output file mac_change_trace.txt using below steps?

echo 0 > /sys/kernel/debug/tracing/tracing_on
echo function_graph > /sys/kernel/debug/tracing/current_tracer
echo > /sys/kernel/debug/tracing/trace
echo > /sys/kernel/debug/tracing/set_ftrace_filter
echo ':mod:ib*' > /sys/kernel/debug/tracing/set_ftrace_filter
echo ':mod:rdma*' >> /sys/kernel/debug/tracing/set_ftrace_filter
echo 1 > /sys/kernel/debug/tracing/tracing_on

ip link set <netdev> address <new_mac1>
ip link set <netdev> address <new_mac2>
cat /sys/kernel/debug/tracing/trace > mac_change_trace.txt

[1] git://git.launchpad.net/~ubuntu-kernel-test/ubuntu/+source/linux/+git/mainline-crack v5.0
[2] commit 5c5702e259dc ("RDMA/core: Set right entry state before releasing reference")
[3] commit be5914c124bc ("RDMA/core: Delete RoCE GID in hw when corresponding IP is deleted")
[4] commit d12e2eed2743 ("IB/core: Update GID entries for netdevice whose mac address changes")

Re: 【Ask for help】 A question for __ib_cache_gid_add()

[oulijun]

在 2019/11/1 21:21, Mark Zhang 写道:
> On 11/1/2019 9:05 PM, Jason Gunthorpe wrote:
>> On Fri, Nov 01, 2019 at 05:36:36PM +0800, oulijun wrote:
>>> Hi
>>>    I am using the ubuntu system(5.0.0 kernel) to test the hip08 NIC port,. When I modify the perr mac1 to mac2,then restore to mac1, it will cause
>>> the gid0 and gid 1 of the roce to be unavailable, and check that the /sys/class/infiniband/hns_0/ports/1/gid_attrs/ndevs/0 is show invalid.
>>> the protocol stack print will appear.
>>>
>>>    Oct 16 17:59:36 ubuntu kernel: [200635.496317] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:4600:4dff:fea7:9599 error=-28
>>> Oct 16 17:59:37 ubuntu kernel: [200636.705848] 8021q: adding VLAN 0 to HW filter on device enp189s0f0
>>> Oct 16 17:59:37 ubuntu kernel: [200636.705854] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:4600:4dff:fea7:9599 error=-28
>>> Oct 16 17:59:39 ubuntu kernel: [200638.755828] hns3 0000:bd:00.0 enp189s0f0: link up
>>> Oct 16 17:59:39 ubuntu kernel: [200638.755847] IPv6: ADDRCONF(NETDEV_CHANGE): enp189s0f0: link becomes ready
>>> Oct 16 18:00:56 ubuntu kernel: [200715.699961] hns3 0000:bd:00.0 enp189s0f0: link down
>>> Oct 16 18:00:56 ubuntu kernel: [200716.016142] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:4600:4dff:fea7:95f4 error=-28
>>> Oct 16 18:00:58 ubuntu kernel: [200717.229857] 8021q: adding VLAN 0 to HW filter on device enp189s0f0
>>> Oct 16 18:00:58 ubuntu kernel: [200717.229863] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:4600:4dff:fea7:95f4 error=-28
>>>
>>> Has anyone else encounterd a similar problem ? I wonder if the _ib_cache_add_gid() is defective in 5.0 kernel?
>> Maybe Parav knows?
>>
> One possibility is, during the operation you have port reset; Then
> all gids will be deleted and re-created. But during gids re-build
> gid entry #0 and #1 is in use, so they cannot be deleted then you
> get this error.
Thank your reply. I will check it acording to my logs.
>> Jason
>>

Re: 【Ask for help】 A question for __ib_cache_gid_add()

[oulijun]

在 2019/11/2 0:00, Parav Pandit 写道:
> Hi Lijun,
>
>> -----Original Message-----
>> From: Jason Gunthorpe <jgg@ziepe.ca>
>> Sent: Friday, November 1, 2019 8:06 AM
>> To: oulijun <oulijun@huawei.com>; Parav Pandit <parav@mellanox.com>
>> Cc: Doug Ledford <dledford@redhat.com>; linux-rdma <linux-
>> rdma@vger.kernel.org>
>> Subject: Re: 【Ask for help】 A question for __ib_cache_gid_add()
>>
>> On Fri, Nov 01, 2019 at 05:36:36PM +0800, oulijun wrote:
>>> Hi
>>>   I am using the ubuntu system(5.0.0 kernel) to test the hip08 NIC
>>> port,. When I modify the perr mac1 to mac2,then restore to mac1, it will
>> cause the gid0 and gid 1 of the roce to be unavailable, and check that the
>> /sys/class/infiniband/hns_0/ports/1/gid_attrs/ndevs/0 is show invalid.
>>> the protocol stack print will appear.
>>>
>>>   Oct 16 17:59:36 ubuntu kernel: [200635.496317] __ib_cache_gid_add:
>>> unable to add gid fe80:0000:0000:0000:4600:4dff:fea7:9599 error=-28
>>> Oct 16 17:59:37 ubuntu kernel: [200636.705848] 8021q: adding VLAN 0 to
>>> HW filter on device enp189s0f0 Oct 16 17:59:37 ubuntu kernel:
>>> [200636.705854] __ib_cache_gid_add: unable to add gid
>>> fe80:0000:0000:0000:4600:4dff:fea7:9599 error=-28 Oct 16 17:59:39
>>> ubuntu kernel: [200638.755828] hns3 0000:bd:00.0 enp189s0f0: link up
>>> Oct 16 17:59:39 ubuntu kernel: [200638.755847] IPv6:
>>> ADDRCONF(NETDEV_CHANGE): enp189s0f0: link becomes ready Oct 16
>>> 18:00:56 ubuntu kernel: [200715.699961] hns3 0000:bd:00.0 enp189s0f0:
>>> link down Oct 16 18:00:56 ubuntu kernel: [200716.016142]
>>> __ib_cache_gid_add: unable to add gid
>>> fe80:0000:0000:0000:4600:4dff:fea7:95f4 error=-28 Oct 16 18:00:58
>>> ubuntu kernel: [200717.229857] 8021q: adding VLAN 0 to HW filter on
>>> device enp189s0f0 Oct 16 18:00:58 ubuntu kernel: [200717.229863]
>>> __ib_cache_gid_add: unable to add gid
>>> fe80:0000:0000:0000:4600:4dff:fea7:95f4 error=-28
>>>
>>> Has anyone else encounterd a similar problem ? I wonder if the
>> _ib_cache_add_gid() is defective in 5.0 kernel?
>>
>> Maybe Parav knows?
> I used the kernel from [1], which seems to be fine; it has the required commits [2], [3], [4].
>
> Are you running RDMA traffic/applications which are using GID 0 and 1 when changing MAC?
> If so, administrative operation such as MAC address change during active RDMA traffic is unsupported, which can lead to this error.
> Can you please confirm?
Hi, parav Pandit
    if running RDMA traffic/application which are using vlan gid when vconfig rem the vlan, it will happen the following error?
   
 Oct 11 13:51:13 ubuntu kernel: [10408.846497] unregister_netdevice: waiting for eno1.1000 to become free. Usage count = 4
Oct 11 13:51:23 ubuntu kernel: [10418.926477] unregister_netdevice: waiting for eno1.1000 to become free. Usage count = 4
Oct 11 13:51:33 ubuntu kernel: [10429.006489] unregister_netdevice: waiting for eno1.1000 to become free. Usage count = 4
Oct 11 13:51:43 ubuntu kernel: [10439.086477] unregister_netdevice: waiting for eno1.1000 to become free. Usage count = 4
Oct 11 13:51:53 ubuntu kernel: [10449.166493] unregister_netdevice: waiting for eno1.1000 to become free. Usage count = 4
Oct 11 13:52:04 ubuntu kernel: [10459.246473] unregister_netdevice: waiting for eno1.1000 to become free. Usage count = 4
Oct 11 13:52:14 ubuntu kernel: [10469.326478] unregister_netdevice: waiting for eno1.1000 to become free. Usage count = 4
Oct 11 13:52:24 ubuntu kernel: [10479.406470] unregister_netdevice: waiting for eno1.1000 to become free. Usage count = 4
Oct 11 13:52:34 ubuntu kernel: [10489.486495] unregister_netdevice: waiting for eno1.1000 to become free. Usage count = 4
Oct 11 13:52:44 ubuntu kernel: [10499.566476] unregister_netdevice: waiting for eno1.1000 to become free. Usage count = 4
Oct 11 13:52:54 ubuntu kernel: [10509.646600] unregister_netdevice: waiting for eno1.1000 to become free. Usage count = 4


Thanks

> If you are not running RDMA traffic while changing the mac, I need more debug logs.
> Can you please enable ftrace and share the output file mac_change_trace.txt using below steps?
>
> echo 0 > /sys/kernel/debug/tracing/tracing_on
> echo function_graph > /sys/kernel/debug/tracing/current_tracer
> echo > /sys/kernel/debug/tracing/trace
> echo > /sys/kernel/debug/tracing/set_ftrace_filter
> echo ':mod:ib*' > /sys/kernel/debug/tracing/set_ftrace_filter
> echo ':mod:rdma*' >> /sys/kernel/debug/tracing/set_ftrace_filter
> echo 1 > /sys/kernel/debug/tracing/tracing_on
>
> ip link set <netdev> address <new_mac1>
> ip link set <netdev> address <new_mac2>
> cat /sys/kernel/debug/tracing/trace > mac_change_trace.txt
>
> [1] git://git.launchpad.net/~ubuntu-kernel-test/ubuntu/+source/linux/+git/mainline-crack v5.0
> [2] commit 5c5702e259dc ("RDMA/core: Set right entry state before releasing reference")
> [3] commit be5914c124bc ("RDMA/core: Delete RoCE GID in hw when corresponding IP is deleted")
> [4] commit d12e2eed2743 ("IB/core: Update GID entries for netdevice whose mac address changes")
>
>
> .
>

RE: 【Ask for help】 A question for __ib_cache_gid_add()

[Parav Pandit]

Hi Lijun,

> From: linux-rdma-owner@vger.kernel.org <linux-rdma-
> owner@vger.kernel.org> On Behalf Of oulijun
> Sent: Wednesday, November 13, 2019 6:36 AM
> 在 2019/11/2 0:00, Parav Pandit 写道:
> > Hi Lijun,
> >
> >> -----Original Message-----
> >> From: Jason Gunthorpe <jgg@ziepe.ca>
> >> Sent: Friday, November 1, 2019 8:06 AM
> >> To: oulijun <oulijun@huawei.com>; Parav Pandit <parav@mellanox.com>
> >> Cc: Doug Ledford <dledford@redhat.com>; linux-rdma <linux-
> >> rdma@vger.kernel.org>
> >> Subject: Re: 【Ask for help】 A question for __ib_cache_gid_add()
> >>
> >> On Fri, Nov 01, 2019 at 05:36:36PM +0800, oulijun wrote:
> >>> Hi
> >>>   I am using the ubuntu system(5.0.0 kernel) to test the hip08 NIC
> >>> port,. When I modify the perr mac1 to mac2,then restore to mac1, it
> >>> will
> >> cause the gid0 and gid 1 of the roce to be unavailable, and check
> >> that the
> >> /sys/class/infiniband/hns_0/ports/1/gid_attrs/ndevs/0 is show invalid.
> >>> the protocol stack print will appear.
> >>>
> >>>   Oct 16 17:59:36 ubuntu kernel: [200635.496317] __ib_cache_gid_add:
> >>> unable to add gid fe80:0000:0000:0000:4600:4dff:fea7:9599 error=-28
> >>> Oct 16 17:59:37 ubuntu kernel: [200636.705848] 8021q: adding VLAN 0
> >>> to HW filter on device enp189s0f0 Oct 16 17:59:37 ubuntu kernel:
> >>> [200636.705854] __ib_cache_gid_add: unable to add gid
> >>> fe80:0000:0000:0000:4600:4dff:fea7:9599 error=-28 Oct 16 17:59:39
> >>> ubuntu kernel: [200638.755828] hns3 0000:bd:00.0 enp189s0f0: link up
> >>> Oct 16 17:59:39 ubuntu kernel: [200638.755847] IPv6:
> >>> ADDRCONF(NETDEV_CHANGE): enp189s0f0: link becomes ready Oct 16
> >>> 18:00:56 ubuntu kernel: [200715.699961] hns3 0000:bd:00.0 enp189s0f0:
> >>> link down Oct 16 18:00:56 ubuntu kernel: [200716.016142]
> >>> __ib_cache_gid_add: unable to add gid
> >>> fe80:0000:0000:0000:4600:4dff:fea7:95f4 error=-28 Oct 16 18:00:58
> >>> ubuntu kernel: [200717.229857] 8021q: adding VLAN 0 to HW filter on
> >>> device enp189s0f0 Oct 16 18:00:58 ubuntu kernel: [200717.229863]
> >>> __ib_cache_gid_add: unable to add gid
> >>> fe80:0000:0000:0000:4600:4dff:fea7:95f4 error=-28
> >>>
> >>> Has anyone else encounterd a similar problem ? I wonder if the
> >> _ib_cache_add_gid() is defective in 5.0 kernel?
> >>
> >> Maybe Parav knows?
> > I used the kernel from [1], which seems to be fine; it has the required
> commits [2], [3], [4].
> >
> > Are you running RDMA traffic/applications which are using GID 0 and 1 when
> changing MAC?
> > If so, administrative operation such as MAC address change during active
> RDMA traffic is unsupported, which can lead to this error.
> > Can you please confirm?
> Hi, parav Pandit
>     if running RDMA traffic/application which are using vlan gid when vconfig
> rem the vlan, it will happen the following error?

Yes, that is correct. Application is bound the rdma device and its associated vlan netdevice, holding reference to it while QPs are active.
Once application releases the QPs, unregistration should progress.

Supporting dynamic detachment of netdevices from the RoCE device was added using below 7 patches, those are not present in the Ubuntu 5.0.0. kernel that I looked at.

3bf3e2b881c1 RDMA/rxe: Consider skb reserve space based on netdev of GID
8f9748602491 IB/cm: Reduce dependency on gid attribute ndev check
a70c07397fd8 RDMA: Introduce and use GID attr helper to read RoCE L2 fields
adb4a57a7a1d RDMA/cma: Use rdma_read_gid_attr_ndev_rcu to access netdev
dab2175800ef RDMA/rxe: Use rdma_read_gid_attr_ndev_rcu to access netdev
5102eca9039b net/smc: Use rdma_read_gid_l2_fields to L2 fields
943bd984b108 RDMA/core: Allow detaching gid attribute netdevice for RoCE