Re: [B.A.T.M.A.N.] Paper "Performance Evaluation of BATMAN-adv Wireless Mesh Network Routing Algorithms "

[Sven Eckelmann <sven@narfation.org>]

[-- Attachment #1: Type: text/plain, Size: 7190 bytes --]

On Montag, 13. August 2018 04:24:08 CEST ligang.liu@mail.sim.ac.cn wrote:
[...]
> I'm so sorry for late reply, because I was in vocation last week.

No problem. But please change your reply style [2]. It is not very helpful 
when I first have to open my old mail to find out what you've added to the 
text and what was my original text :)

 
> Thank you very much for your concern on our work.
> 
> First of all, I have to say that the batman-adv V has performed very good in our recent tests.
> The main reason why it performed poor when we tested it  is the cfg80211 bug.
> When we made the test in Jan. we did not know about the cfg80211 uninitialization bug.
> We never thought of this kind of bug at that time, which is far beyond our scope.

Thanks for checking that. In the light of this and the other problems 
mentioned in my mail, it would be a good idea to retract the paper with the 
wrong information and revisit the topic with a new paper. I would be really 
interested in your results and the overhead comparison - but when comparing 
the different B.A.T.M.A.N. algorithm versions instead of bugs in other 
components.

This paper now could cause problems when other research use it as base for 
their own research - not noticing that this paper is flawed.

@kekegai@smart-com.org: Are you handling the retraction of CSCloud/EdgeCom 2018
conference papers [1]?

> I noticed this problem in the mail-list until in June.
> 
> Recently, I tried to fix the bug by modified the cfg80211 source code, ref https://patchwork.kernel.org/patch/10449857/ 
[...]
> Some tests have revealed that V has very good performance, at least not bad than IV.
> We will report our results after more careful tests.

Thanks


> Here is my response of your questions.
> 
> ************************************************************************************** 
> * What is OpenWrt 1703? (you've referenced a batman-adv config tutorial here 
>   and there is no such version as OpenWrt 1703)
> 
> ANS: 
> The base OpenWrt we used was built near in Sep.  2017, with
> git clone https://git.lede-project.org/source.git
> 
> I think the OpenWrt version should be LEDE 17.01.03

No, this would *not* result in LEDE 17.01.3 (released in Oct 3, 2017) - 
because you would have to specify the correct branch branch (actually the 
correct tag). The work on the stable versions v17.01.x was separated from the 
rest of the development with the branch created on on 2017-01-16. And your
commands will neither use the separate branch or the actual commit - it will 
just use whatever master is at that time. Rather bad when somebody wants to 
understand how to reproduce the results of your experiment.

Please make sure in your revisited paper that you make correct 
statements about the used software (LEDE vs. OpenWrt, git tags or actual git 
version's, applied patches, used feed versions, ...). Btw. there was just an 
actual OpenWrt release (after v15.05.1): 18.06.0 [3].

And also make sure that you're references are actually about the software. 
Right now your paper makes wrong statements about that kind of stuff.

* don't call it OpenWRT when it is LEDE
* don't name it 1703 when there is no such version and you've actually used 
  the development branch
* don't add a reference for OpenWRT 1703 which is then pointing to an 
  unrelated config tutorial on open-mesh.org

Having these incorrect statements in your paper doesn't make it look like it
is of high quality. And this also causes problems when other try to use your 
paper as base for their own research.

> **************************************************************************************
> * When did you want to contact the authors of the B.A.T.M.A.N. V 
>   implementation in batman-adv about your findings?
> 
> ANS: 
> I have tried to contact the community by sending a email to the mail-list,
> with a title "What's the exact definition of expected_throughput in BATMAN-ADV V?".
> In that email, I described my concern that the value of the expected_throughput was confused,
> and asked for help.
> 
> Unfortunately there are no response from there.

There was no such mail on the mailing list [4]. Please make sure that you 
followed our requirements for this list [5] (no HTML mails). And please use 
shorter, more concrete subjects than that.

But it seems that you never contacted the batman-adv developers via the 
mailing list about the other problems.

> **************************************************************************************
> * How did you solve the cfg80211 bug which returns uninitialized (random)
>   data when asked for the expected throughput of a station? Maybe you remember 
>   that the expected throughput is *the value* which B.A.T.M.A.N. V needs to 
>   calculate routes.
> 
> ANS:
> Our test was done in Jan. 2018 and the paper was submitted in Feb. 2018.
> At that time, I even had not realized that the problem of expected_throughput
> was from cfg80211 bug.

So your paper therefore (without you noticing) doesn't compare B.A.T.M.A.N. IV 
vs. B.A.T.M.A.N. V but still tries to establish the narrative that 
B.A.T.M.A.N. V is worse. 


> **************************************************************************************
> * What batman-adv version was used?
> 
> ANS:
> I think the version of batman-adv is 2017.3
> because the source file in OpenWrt named as:
> batman-adv-2017.3
> batctl-2017.3

This already shows that you did not use LEDE 17.01.3 - because this version 
did never include that particular batman-adv release. You were more likely on 
some random LEDE development branch commit.

> 
> **************************************************************************************
> * Did you use the openwrt-routing feed (which exact version) for
>   batctl+batman-adv or how did you build it?
> 
> ANS:
> Here are the main scripts.
> git clone https://git.lede-project.org/source.git
> ./scripts/feeds update -a
> ./scripts/feeds install -a
> 
> make download V=s
> make V=99
> 
> At last I flushed the generated 2 files:
> lede-ar71xx-generic-uImage-lzma.bin
> lede-ar71xx-generic-root.squashfs

These instruction will result in different version depending on when you run 
them. Neither the used LEDE commit is specified here nor the used feed commit.

> **************************************************************************************
> * Did you generate graphs for the chosen best next hop and how it changes
>   over time in comparison to B.A.T.M.A.N. IV?
>  
> ANS:
> I had checked the route by batctl ping -R xxx.xxx.xxx.xxx
> Unfortunately, we did not record and compare the route between IV and V.

I am currently assuming that the chosen routes were wrong because of the 
cfg80211 bug [6]. Your brief tests with the fixed cfg80211 seem to suggest 
this but no way to check this now.

Kind regards,
	Sven

> [1] https://ieeexplore.ieee.org/document/8421863/
[2] https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
[3] https://github.com/openwrt/openwrt/releases/tag/v18.06.0
[4] https://lists.open-mesh.org/pipermail/b.a.t.m.a.n/
[5] https://www.open-mesh.org/projects/open-mesh/wiki/MailingList
[6] https://patchwork.kernel.org/patch/10449857/

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]