Reference Edge Test Plan

IoT Edge Test Plan using the Horton Test Framework

This document is an outline of the specific test scenarios we are covering with the IoT Edge tests.

Related documents

• For a big-picture of the Horton test framework, look here
• Some definitions useful in understanding the Horton framework are outlined here
• Examples and visualizations for understanding the definitions are provided here

Definitions

In addition to the definitions used by the Horton framework, the following definitions are useful in describing the Edge E2E tests.

• test module - This is the Edge Module, living inside a docker container, which contains the code that we are testing.
• friend module - This is a secondary Edge module which is used to help validate the test module. The friend module is used:
  • as a destination for output events that the test module sends
  • as a source for input messages that the test module receives
  • as a receiver for method calls that the test module invokes
  • as an invoker for method calls that the test module receives
• leaf device - This is a limited Azure IoT Device client which is also used to help validate the test module. The leaf device is used:
  • as a receiver for method calls that the test module invokes

The following names are also known, but they are part of a implementation details that are being phased out.

• nodeMod, cMod, csharpMod, javaMod, pythonMod, and pythonPreviewMod - these are the names of docker containers that are used as test modules for different langauges.
• friendMod - this is the name of the docker container which is responsible for acting as:
  • The friend module as defined above.
  • The leaf device as defined above.
  • a default implementation of the Service and Registry APIs, in case the module under test does not have an implementation to use for testing.

The fact that friendMod is an Edge module, and also a leaf device and an implementation of other APIs is confusing and should not be dwelt on as these pieces of functionality are eventually moving into their own containers

Test Environments

Edge Tests are currently being run only under AMD64 Linux (16.04) with the folloviwing Docker containers. Base container choices are somewhat arbitrary.

SDK	Docker Base Image Name	OS	Language Variant
C	ubuntu:18.04	Ubuntu 18.04	GCC
CSharp	microsoft/dotnet:2.2-sdk	Debian Stretch (v9)	.net core 2.2
Java	maven:3.3-jdk-8	Debian Jesie (v8)	JDK-8
Node	node:6-slim	Debian Stretch (v9)	Node 6.17
Python	ubuntu:18.04	Ubuntu 18.04	GCC + Python 3.6
Python-preview	python:3.6	Debian Stretch (v9)	Python 3.6.8

Test Suites (or "the matrix of suites")

Tests for the Module Client SDK are run using both EdgeHub and IoTHub as destinations with 4 different transports (AMQP, AMQP-WS, MQTT, MQTT-WS). This leads us to 8 different suites:

• edgehub_module_amqp
• edgehub_module_amqp_ws
• edgehub_module_mqtt
• edgehub_module_mqtt_ws
• iothub_module_amqp
• iothub_module_amqp_ws
• iothub_module_mqtt
• iothub_module_mqtt_ws

Test Cases

edgehub	iothub	edghub fi	test name
X	X	X	test_module_client_connect_disconnect
X	X	X	test_module_client_connect_enable_twin_disconnect
X	X	X	test_module_client_connect_enable_methods_disconnect
X	-	X	test_module_client_connect_enable_input_messages_disconnect
X	X	-	test_registry_client_connect_disconnect
X	X	-	test_service_client_connect_disconnect
X	X	-	test_device_client_connect_disconnect
X	X	X	test_device_client_connect_enable_methods_disconnect
X	-	X	test_device_method_from_service_to_leaf_device
X	-	X	test_device_method_from_module_to_leaf_device
X	-	-	test_module_input_output_loopback
X	X	X	test_module_method_call_invoked_from_service
X	-	X	test_module_method_from_test_to_friend
X	-	X	test_module_method_from_friend_to_test
X	-	X	test_module_output_routed_upstream
X	X	X	test_module_send_event_to_iothub
X	-	X	test_module_to_friend_routing
X	-	X	test_friend_to_module_routing
X	-	-	test_module_test_to_friend_and_back
X	X	X	test_service_can_set_desired_properties_and_module_can_retrieve_them
X	X	X	test_service_can_set_multiple_desired_property_patches_and_module_can_retrieve_them_as_events
X	X	X	test_module_can_set_reported_properties_and_service_can_retrieve_them

Current Exclusions

With the exception of Node, we don't test any leaf device functionality

All SDKs run all suites, with two exceptions:

1. C and Python don't run edgehub_module_amqp and edgehub_module_amqpws
2. python-preview only runs edgehub_module_mqtt and iothub_module_amqtt

Several tests are also skipped:

• C AMQP currently skips all tests that send output events or receive input messages.
• C MQTTWS currently skips all tests that receive input messages
• C, when using a connection string instead of the environment, skips any tests that invoke methods
• Java skips all twin tests.