[PATCH 1/9] knownnetworks: copy network_config on update

[PATCH 1/9] knownnetworks: copy network_config on update

[James Prestwood]

[-- Attachment #1: Type: text/plain, Size: 699 bytes --]

The network_config was not being copied to network_info when
updated. This caused any new settings to be lost if the network
configuration file was updated during runtime.
---
 src/knownnetworks.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/src/knownnetworks.c b/src/knownnetworks.c
index 09acdd3c..85648ead 100644
--- a/src/knownnetworks.c
+++ b/src/knownnetworks.c
@@ -464,6 +464,8 @@ void known_network_update(struct network_info *network,
 	}
 
 	known_network_set_autoconnect(network, new->is_autoconnectable);
+
+	memcpy(&network->config, new, sizeof(struct network_config));
 }
 
 bool known_networks_foreach(known_networks_foreach_func_t function,
-- 
2.31.1

[PATCH 2/9] auto-t: hostapd: catch local socket remove exception

[James Prestwood]

[-- Attachment #1: Type: text/plain, Size: 683 bytes --]

This happens if hostapd is killed ungracefully which is done in
some tests.
---
 autotests/util/hostapd.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/autotests/util/hostapd.py b/autotests/util/hostapd.py
index 4e694c84..6031b1dc 100644
--- a/autotests/util/hostapd.py
+++ b/autotests/util/hostapd.py
@@ -118,7 +118,11 @@ class HostapdCLI:
             return
 
         self.ctrl_sock.close()
-        os.remove(self.local_ctrl)
+
+        try:
+            os.remove(self.local_ctrl)
+        except:
+            pass
 
         if self._hostapd_restarted:
             ctx.stop_process(ctx.hostapd.process, force)
-- 
2.31.1

[PATCH 3/9] test-runner: remove TLS debugging artifact

[James Prestwood]

[-- Attachment #1: Type: text/plain, Size: 832 bytes --]

When logging is enabled TLS debugging is turned on which creates
a PEM file during runtime. There is no way for IWD itself to clean
this up since its meant to be there for debugging.
---
 tools/test-runner | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tools/test-runner b/tools/test-runner
index 98164f7a..fa3c434f 100755
--- a/tools/test-runner
+++ b/tools/test-runner
@@ -1192,6 +1192,10 @@ def post_test(ctx, to_copy):
 				dbg("\n")
 				os.remove('/tmp/' + f)
 
+	# Special case for when logging is enabled
+	if os.path.isfile('/tmp/iwd-tls-debug-server-cert.pem'):
+		os.remove('/tmp/iwd-tls-debug-server-cert.pem')
+
 	allowed = ['phonesim.conf', 'certs', 'secrets', 'iwd']
 	for f in [f for f in os.listdir('/tmp') if f not in allowed]:
 		dbg("File %s was not cleaned up!" % f)
-- 
2.31.1

[PATCH 4/9] test-runner: set msize for --monitor

[James Prestwood]

[-- Attachment #1: Type: text/plain, Size: 712 bytes --]

This was added for logging but --monitor suffers the same warning
without setting msize as well.
---
 tools/test-runner | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/test-runner b/tools/test-runner
index fa3c434f..2d4b6b5a 100755
--- a/tools/test-runner
+++ b/tools/test-runner
@@ -1379,7 +1379,7 @@ def run_tests():
 				os.remove(f)
 	elif args.monitor:
 		parent = os.path.abspath(os.path.join(args.monitor, os.pardir))
-		mount('mondir', parent, '9p', 0, 'trans=virtio,version=9p2000.L')
+		mount('mondir', parent, '9p', 0, 'trans=virtio,version=9p2000.L,msize=10240')
 
 	if config.ctx.args.unit_tests is None:
 		run_auto_tests(config.ctx, args)
-- 
2.31.1

[PATCH 5/9] test-runner: increase RAM to 384MB

[James Prestwood]

[-- Attachment #1: Type: text/plain, Size: 580 bytes --]

Occationally python will fatally terminate trying to load a test
using importlib with an out of memory exception. Increasing RAM
allows reliable exection of all tests.
---
 tools/test-runner | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/test-runner b/tools/test-runner
index 2d4b6b5a..ff7d9586 100755
--- a/tools/test-runner
+++ b/tools/test-runner
@@ -1585,7 +1585,7 @@ class Main:
 		if self.args.valgrind:
 			ram = 512
 		else:
-			ram = 256
+			ram = 384
 
 		print("Using %d cores, %d RAM for VM" % (smp, ram))
 
-- 
2.31.1

[PATCH 6/9] test-runner: don't reset dbus object on reset()

[James Prestwood]

[-- Attachment #1: Type: text/plain, Size: 573 bytes --]

Doing this is redundant since the dbus object is initialized
anyways when a new test starts.
---
 tools/test-runner | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/tools/test-runner b/tools/test-runner
index ff7d9586..cb97657e 100755
--- a/tools/test-runner
+++ b/tools/test-runner
@@ -593,9 +593,6 @@ class Namespace:
 
 		self.radios = []
 
-		if self.name == "root":
-			self._bus = dbus.bus.BusConnection(address_or_type=self.dbus_address)
-
 		for p in list(self.processes):
 			print("Killing process %s" % p.name)
 			p.kill()
-- 
2.31.1

[PATCH 7/9] auto-t: remove unneeded hidden networks

[James Prestwood]

[-- Attachment #1: Type: text/plain, Size: 7993 bytes --]

testScan was creating 10 separate hidden networks which
sometimes bogged down hostapd to the point that it would
not start up in time before test-runner's timeouts fired.
This appeared to be due to hostapd needing to create 10
separate interfaces which would sometimes fail with -ENFILE.

The test itself only needed two separate networks, so instead
the additional 8 can be completely removed.
---
 autotests/testScan/hw.conf                | 11 +++---
 autotests/testScan/periodic_scan_test.py  | 12 ++-----
 autotests/testScan/requested_scan_test.py |  8 -----
 autotests/testScan/ssid_hidden.conf       | 41 -----------------------
 autotests/testScan/ssid_hidden_0.conf     |  4 +++
 autotests/testScan/ssid_hidden_1.conf     |  4 +++
 autotests/testScan/ssid_hidden_2.open     |  3 --
 autotests/testScan/ssid_hidden_3.open     |  3 --
 autotests/testScan/ssid_hidden_4.open     |  3 --
 autotests/testScan/ssid_hidden_5.open     |  3 --
 autotests/testScan/ssid_hidden_6.open     |  3 --
 autotests/testScan/ssid_hidden_7.open     |  3 --
 autotests/testScan/ssid_hidden_8.open     |  3 --
 autotests/testScan/ssid_hidden_9.open     |  3 --
 14 files changed, 16 insertions(+), 88 deletions(-)
 delete mode 100644 autotests/testScan/ssid_hidden.conf
 create mode 100644 autotests/testScan/ssid_hidden_0.conf
 create mode 100644 autotests/testScan/ssid_hidden_1.conf
 delete mode 100644 autotests/testScan/ssid_hidden_2.open
 delete mode 100644 autotests/testScan/ssid_hidden_3.open
 delete mode 100644 autotests/testScan/ssid_hidden_4.open
 delete mode 100644 autotests/testScan/ssid_hidden_5.open
 delete mode 100644 autotests/testScan/ssid_hidden_6.open
 delete mode 100644 autotests/testScan/ssid_hidden_7.open
 delete mode 100644 autotests/testScan/ssid_hidden_8.open
 delete mode 100644 autotests/testScan/ssid_hidden_9.open

diff --git a/autotests/testScan/hw.conf b/autotests/testScan/hw.conf
index b2464baf..660c874e 100644
--- a/autotests/testScan/hw.conf
+++ b/autotests/testScan/hw.conf
@@ -1,9 +1,10 @@
 [SETUP]
-num_radios=5
+num_radios=6
 start_iwd=0
 
 [HOSTAPD]
-rad0=ssid_hidden.conf
-rad1=ssid_open.conf
-rad2=ssid_8021x.conf
-rad3=ssid_psk.conf
+rad0=ssid_hidden_0.conf
+rad1=ssid_hidden_1.conf
+rad2=ssid_open.conf
+rad3=ssid_8021x.conf
+rad4=ssid_psk.conf
diff --git a/autotests/testScan/periodic_scan_test.py b/autotests/testScan/periodic_scan_test.py
index 3252f79f..439b844d 100644
--- a/autotests/testScan/periodic_scan_test.py
+++ b/autotests/testScan/periodic_scan_test.py
@@ -12,11 +12,7 @@ class Test(unittest.TestCase):
         'ssid_open': False,
         'ssid_psk': False,
         'ssid_8021x': False,
-        'ssid_hidden_5': False,
-        'ssid_hidden_6': False,
-        'ssid_hidden_7': False,
-        'ssid_hidden_8': False,
-        'ssid_hidden_9': False,
+        'ssid_hidden_1': False,
     }
 
     def set_network(self, ssid):
@@ -61,11 +57,7 @@ class Test(unittest.TestCase):
 
     @classmethod
     def setUpClass(cls):
-        IWD.copy_to_storage('ssid_hidden_5.open')
-        IWD.copy_to_storage('ssid_hidden_6.open')
-        IWD.copy_to_storage('ssid_hidden_7.open')
-        IWD.copy_to_storage('ssid_hidden_8.open')
-        IWD.copy_to_storage('ssid_hidden_9.open')
+        IWD.copy_to_storage('ssid_hidden_1.open')
 
     @classmethod
     def tearDownClass(cls):
diff --git a/autotests/testScan/requested_scan_test.py b/autotests/testScan/requested_scan_test.py
index 3858b4f5..69297f27 100644
--- a/autotests/testScan/requested_scan_test.py
+++ b/autotests/testScan/requested_scan_test.py
@@ -15,10 +15,6 @@ class Test(unittest.TestCase):
         'ssid_psk': False,
         'ssid_8021x': False,
         'ssid_hidden_0': False,
-        'ssid_hidden_1': False,
-        'ssid_hidden_2': False,
-        'ssid_hidden_3': False,
-        'ssid_hidden_4': False,
     }
 
     def set_network(self, ssid):
@@ -63,10 +59,6 @@ class Test(unittest.TestCase):
     @classmethod
     def setUpClass(cls):
         IWD.copy_to_storage('ssid_hidden_0.open')
-        IWD.copy_to_storage('ssid_hidden_1.open')
-        IWD.copy_to_storage('ssid_hidden_2.open')
-        IWD.copy_to_storage('ssid_hidden_3.open')
-        IWD.copy_to_storage('ssid_hidden_4.open')
 
     @classmethod
     def tearDownClass(cls):
diff --git a/autotests/testScan/ssid_hidden.conf b/autotests/testScan/ssid_hidden.conf
deleted file mode 100644
index 2dfed56f..00000000
--- a/autotests/testScan/ssid_hidden.conf
+++ /dev/null
@@ -1,41 +0,0 @@
-ssid=ssid_hidden_0
-interface=wln0
-hw_mode=g
-channel=1
-ignore_broadcast_ssid=1
-
-bss=wln0_1
-ssid=ssid_hidden_1
-ignore_broadcast_ssid=1
-
-bss=wln0_2
-ssid=ssid_hidden_2
-ignore_broadcast_ssid=1
-
-bss=wln0_3
-ssid=ssid_hidden_3
-ignore_broadcast_ssid=1
-
-bss=wln0_4
-ssid=ssid_hidden_4
-ignore_broadcast_ssid=1
-
-bss=wln0_5
-ssid=ssid_hidden_5
-ignore_broadcast_ssid=1
-
-bss=wln0_6
-ssid=ssid_hidden_6
-ignore_broadcast_ssid=1
-
-bss=wln0_7
-ssid=ssid_hidden_7
-ignore_broadcast_ssid=1
-
-bss=wln0_8
-ssid=ssid_hidden_8
-ignore_broadcast_ssid=1
-
-bss=wln0_9
-ssid=ssid_hidden_9
-ignore_broadcast_ssid=1
diff --git a/autotests/testScan/ssid_hidden_0.conf b/autotests/testScan/ssid_hidden_0.conf
new file mode 100644
index 00000000..5d15e360
--- /dev/null
+++ b/autotests/testScan/ssid_hidden_0.conf
@@ -0,0 +1,4 @@
+ssid=ssid_hidden_0
+hw_mode=g
+channel=1
+ignore_broadcast_ssid=1
diff --git a/autotests/testScan/ssid_hidden_1.conf b/autotests/testScan/ssid_hidden_1.conf
new file mode 100644
index 00000000..cc11803f
--- /dev/null
+++ b/autotests/testScan/ssid_hidden_1.conf
@@ -0,0 +1,4 @@
+ssid=ssid_hidden_1
+hw_mode=g
+channel=1
+ignore_broadcast_ssid=1
diff --git a/autotests/testScan/ssid_hidden_2.open b/autotests/testScan/ssid_hidden_2.open
deleted file mode 100644
index 28622594..00000000
--- a/autotests/testScan/ssid_hidden_2.open
+++ /dev/null
@@ -1,3 +0,0 @@
-[Settings]
-AutoConnect=false
-Hidden=true
diff --git a/autotests/testScan/ssid_hidden_3.open b/autotests/testScan/ssid_hidden_3.open
deleted file mode 100644
index 28622594..00000000
--- a/autotests/testScan/ssid_hidden_3.open
+++ /dev/null
@@ -1,3 +0,0 @@
-[Settings]
-AutoConnect=false
-Hidden=true
diff --git a/autotests/testScan/ssid_hidden_4.open b/autotests/testScan/ssid_hidden_4.open
deleted file mode 100644
index 28622594..00000000
--- a/autotests/testScan/ssid_hidden_4.open
+++ /dev/null
@@ -1,3 +0,0 @@
-[Settings]
-AutoConnect=false
-Hidden=true
diff --git a/autotests/testScan/ssid_hidden_5.open b/autotests/testScan/ssid_hidden_5.open
deleted file mode 100644
index 28622594..00000000
--- a/autotests/testScan/ssid_hidden_5.open
+++ /dev/null
@@ -1,3 +0,0 @@
-[Settings]
-AutoConnect=false
-Hidden=true
diff --git a/autotests/testScan/ssid_hidden_6.open b/autotests/testScan/ssid_hidden_6.open
deleted file mode 100644
index 28622594..00000000
--- a/autotests/testScan/ssid_hidden_6.open
+++ /dev/null
@@ -1,3 +0,0 @@
-[Settings]
-AutoConnect=false
-Hidden=true
diff --git a/autotests/testScan/ssid_hidden_7.open b/autotests/testScan/ssid_hidden_7.open
deleted file mode 100644
index 28622594..00000000
--- a/autotests/testScan/ssid_hidden_7.open
+++ /dev/null
@@ -1,3 +0,0 @@
-[Settings]
-AutoConnect=false
-Hidden=true
diff --git a/autotests/testScan/ssid_hidden_8.open b/autotests/testScan/ssid_hidden_8.open
deleted file mode 100644
index 28622594..00000000
--- a/autotests/testScan/ssid_hidden_8.open
+++ /dev/null
@@ -1,3 +0,0 @@
-[Settings]
-AutoConnect=false
-Hidden=true
diff --git a/autotests/testScan/ssid_hidden_9.open b/autotests/testScan/ssid_hidden_9.open
deleted file mode 100644
index 28622594..00000000
--- a/autotests/testScan/ssid_hidden_9.open
+++ /dev/null
@@ -1,3 +0,0 @@
-[Settings]
-AutoConnect=false
-Hidden=true
-- 
2.31.1

[PATCH 8/9] auto-t: add regular 8021x AKM to FT-FILS tests

[James Prestwood]

[-- Attachment #1: Type: text/plain, Size: 1757 bytes --]

After some code changes the FT-FILS AKM was no longer selectable
inside network_can_connect_bss. This normally shouldn't matter
since station ends up selecting the AKM explicitly, including
passing the fils_hint, but since the autotests only included
FT-FILS AKMs this caused the transition to fail with no available
BSS's.

To fix this the standard 8021x AKM was added to the hostapd
configs. This allows these BSS's to be selected when attempting
to roam, but since FT-FILS is the only other AKM it will be used
for the actual transition.
---
 autotests/testFT-FILS-SHA256/ft-eap-ccmp-2.conf | 2 +-
 autotests/testFT-FILS-SHA384/ft-eap-ccmp-2.conf | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/autotests/testFT-FILS-SHA256/ft-eap-ccmp-2.conf b/autotests/testFT-FILS-SHA256/ft-eap-ccmp-2.conf
index 24266ea9..585f1ca1 100644
--- a/autotests/testFT-FILS-SHA256/ft-eap-ccmp-2.conf
+++ b/autotests/testFT-FILS-SHA256/ft-eap-ccmp-2.conf
@@ -9,7 +9,7 @@ nas_identifier=dummy2
 wpa=2
 # Can support WPA-EAP and FT-EAP (space separated list) at the same
 # time but we want to force FT
-wpa_key_mgmt=FT-FILS-SHA256
+wpa_key_mgmt=WPA-EAP FT-FILS-SHA256
 wpa_pairwise=CCMP
 ieee8021x=1
 
diff --git a/autotests/testFT-FILS-SHA384/ft-eap-ccmp-2.conf b/autotests/testFT-FILS-SHA384/ft-eap-ccmp-2.conf
index 8af73cbc..a6bdf641 100644
--- a/autotests/testFT-FILS-SHA384/ft-eap-ccmp-2.conf
+++ b/autotests/testFT-FILS-SHA384/ft-eap-ccmp-2.conf
@@ -9,7 +9,7 @@ nas_identifier=dummy2
 wpa=2
 # Can support WPA-EAP and FT-EAP (space separated list) at the same
 # time but we want to force FT
-wpa_key_mgmt=FT-FILS-SHA384
+wpa_key_mgmt=WPA-EAP FT-FILS-SHA384
 wpa_pairwise=CCMP
 ieee8021x=1
 
-- 
2.31.1

[PATCH 9/9] test-runner: refactor process output code

[James Prestwood]

[-- Attachment #1: Type: text/plain, Size: 4945 bytes --]

The Process class requires the ability to write out any processes
output to stdout, logging, or an explicit file, as well as store
it inside python for processing by test utilities. To accomplish
this each process was given a temporary file to write to, and that
file had an IO watch set on it. Any data that was written was then
read, and re-written out to where it needed to go. This ended up
being very buggy and quite complex due to needing to mess with
read/write pointers inside the file.

Popen already creates pipes to stdout if told, and they are accessable
via the p.stdout. Its then as simple as setting an IO watch on that
pipe and keeping the same code for reading out new data and writing
it to any files we want. This greatly reduces the complexity.
---
 tools/test-runner | 77 +++++++++++++++++------------------------------
 1 file changed, 27 insertions(+), 50 deletions(-)

diff --git a/tools/test-runner b/tools/test-runner
index cb97657e..75e46444 100755
--- a/tools/test-runner
+++ b/tools/test-runner
@@ -175,24 +175,12 @@ class Process:
 		self.io_watch = None
 		self.cleanup = cleanup
 		self.verbose = False
-
-		if not namespace:
-			self.output_name = '/tmp/%s-out' % self.name
-		else:
-			self.output_name = '/tmp/%s-%s-out' % (self.name, namespace)
+		self.out = ''
 
 		if namespace:
 			self.args = ['ip', 'netns', 'exec', namespace]
 			self.args.extend(args)
 
-		#
-		# For simplicity all processes will write to a temp file
-		# (/tmp/<name>-out). If any verbose options are required this file
-		# will get an IO watch and write out any bytes to the needed FDs.
-		#
-		self.stdout = open(self.output_name, 'a+')
-		self.io_position = self.stdout.tell()
-
 		if ctx:
 			# Verbose requested, add stdout/stderr to write FD list.
 			# This will end up always returning true if logging is
@@ -228,18 +216,18 @@ class Process:
 				os.fchown(f.fileno(), int(ctx.args.log_uid), int(ctx.args.log_gid))
 				self.write_fds.append(f)
 
-			#
-			# Only add an IO watch for long running processes. If
-			# the process is being waited for, the log/outfile bits
-			# will be handled after the process exists.
-			#
-			if self.write_fds != [] and not wait and not check or self.verbose:
-				self.io_watch = GLib.io_add_watch(self.stdout, GLib.IO_IN,
-								self.io_callback)
-
-		self.pid = subprocess.Popen(self.args, stdout=self.stdout, stderr=self.stdout,
+		self.pid = subprocess.Popen(self.args, stdout=subprocess.PIPE,
+						stderr=subprocess.STDOUT,
 						env=env, cwd=os.getcwd())
 
+		# Set as non-blocking so read() in the IO callback doesn't block forever
+		fl = fcntl.fcntl(self.pid.stdout, fcntl.F_GETFL)
+		fcntl.fcntl(self.pid.stdout, fcntl.F_SETFL, fl | os.O_NONBLOCK)
+
+		if not wait and not check:
+			self.io_watch = GLib.io_add_watch(self.pid.stdout, GLib.IO_IN |
+						GLib.IO_HUP | GLib.IO_ERR, self.io_callback)
+
 		print("Starting process {}".format(self.pid.args))
 
 		if not wait and not check:
@@ -249,12 +237,7 @@ class Process:
 		self.killed = True
 		self.ret = self.pid.returncode
 
-		self.stdout.seek(self.io_position)
-		self.out = self.stdout.read()
-		self.stdout.seek(0, 2)
-
-		if len(self.write_fds) > 0:
-			self.process_io(self.stdout)
+		self.process_io(self.pid.stdout)
 
 		self.write_fds = []
 
@@ -263,21 +246,16 @@ class Process:
 			raise subprocess.CalledProcessError(returncode=self.ret, cmd=self.args)
 
 	def process_io(self, source):
-		#
-		# The file will have already been written to, meaning the seek
-		# position points to EOF. This is why the position is saved so
-		# we can seek to where we were last time, read data, and seek
-		# back to EOF.
-		#
-		source.seek(self.io_position)
 		data = source.read()
 
-		self.io_position += len(data)
-		source.seek(self.io_position)
-
-		if len(data) == 0:
+		if not data:
 			return True
 
+		data = data.decode('utf-8')
+
+		# Save data away in case the caller needs it (e.g. list_sta)
+		self.out += data
+
 		for f in self.write_fds:
 			f.write(data)
 			f.flush()
@@ -294,10 +272,6 @@ class Process:
 	def __del__(self):
 		print("Del process %s" % self.args)
 
-		os.remove(self.output_name)
-
-		self.stdout.close()
-
 		if not self.killed:
 			self.kill()
 
@@ -312,19 +286,22 @@ class Process:
 		else:
 			self.pid.terminate()
 
-		self.pid.wait(timeout=15)
-		self.pid = None
+		try:
+			self.pid.wait(timeout=15)
+		except:
+			dbg("Process %s did not complete in 15 seconds!" % self.name)
+			self.pid.kill()
 
 		if self.ctx and self in self.ctx.processes:
 			self.ctx.processes.remove(self)
 
-		self.ctx = None
-
-		self.process_io(self.stdout)
-
 		if self.cleanup:
 			self.cleanup()
 
+		self.process_io(self.pid.stdout)
+
+		self.ctx = None
+		self.pid = None
 		self.write_fds = []
 
 		if self.io_watch:
-- 
2.31.1

Re: [PATCH 1/9] knownnetworks: copy network_config on update

[Denis Kenzior]

[-- Attachment #1: Type: text/plain, Size: 358 bytes --]

Hi James,

On 7/29/21 5:21 PM, James Prestwood wrote:
> The network_config was not being copied to network_info when
> updated. This caused any new settings to be lost if the network
> configuration file was updated during runtime.
> ---
>   src/knownnetworks.c | 2 ++
>   1 file changed, 2 insertions(+)

All applied, thanks.

Regards,
-Denis