Reference Framework Definitions Visualized

(slightly more) Concrete Examples for Framework Definitions

Test Definition Example

An example test matrix might be the set of tests that get run as part of the commit gate for a given SDK. To easily visualize this, remember:

• A matrix contains a list of suites
• A suite defines how to runs a scenario with some set of parameters (transport, environment, and destination)
• A scenario contains a bunch of related test cases along with some smaller set of static parameters (destination and connection method)

A test matrix

This matrix (called build-ci in this example) would run the following suites. One purpose of this suite might be to validate code before it gets committed to master.

• edgehub_module_mqtt - Runs all IoT Edge Module tests using MQTT
• edgehub_module_mqttws - ditto, only with a different transport
• edgehub_module_amqp - ditto
• edgehub_module_amqpws - ditto
• iothub_module_mqtt - Runs all IoT Hub Module tests using MQTT
• iothub_module_mqttws - ditto, only with a different transport
• iothub_module_amqp - ditto
• iothub_module_amqpws - ditto

A test suite

The edgehub_module_mqtt suite is simple enough. It runs the edgehub-module scenario, using the MQTT transport, inside a node-v6-ubuntu-slim container.

A test scenario

The edgehub_module test scenario has the following test cases. These all exercise different ModuleApi functionality using IoT Edge as a destination. Additionally, it will test any RegistryApi and ServiceApi functions involved with using IoT Edge Modules. It knows that it uses an IotEdge deployment to deploy this container to a test machine and it knows to use the GatewayHostName= parameter on the connection string so it can route traffic through EdgeHub.

• test_module_client_connect_disconnect
• test_module_client_connect_enable_twin_disconnect
• test_module_client_connect_enable_methods_disconnect
• test_module_client_connect_enable_input_messages_disconnect
• test_registry_client_connect_disconnect
• test_service_client_connect_disconnect
• test_device_client_connect_disconnect
• test_device_client_connect_enable_methods_disconnect
• test_device_method_from_service_to_leaf_device
• test_device_method_from_module_to_leaf_device
• test_module_input_output_loopback
• test_module_method_call_invoked_from_service
• test_module_method_from_test_to_friend
• test_module_method_from_friend_to_test
• test_module_output_routed_upstream
• test_module_send_event_to_iothub
• test_module_to_friend_routing
• test_friend_to_module_routing
• test_module_test_to_friend_and_back
• test_service_can_set_desired_properties_and_module_can_retrieve_them
• test_service_can_set_multiple_desired_property_patches_and_module_can_retrieve_them_as_events
• test_module_can_set_reported_properties_and_service_can_retrieve_them

A test case

If we look at the test called test_module_method_from_friend_to_test, this test will

1. Connect the module that is being tested to EdgeHub
2. Connect another module to edgeHub (the "friend" module)
3. From the friend module, invoke a direct method on the module being tested.
4. On the module being tested, verify that the method call arrives and the parameters are correct. Return a result.
5. On the friend module, verify that the result is returned correctly.

Test Implementation Example

This explains how code being tested gets executed. to easily visualize this, remember:

• The top layers are all written in Python and are shared among all SDKs:
  • The test case calls into the adapter
  • The adapter makes an interop call into the wrapper
• After we interop into the wrapper, we're executing using whatever language we're testing.
  • The wrapper routes the call through the glue
  • The glue calls into the SDK.

The test case

If we zoom in on the test case above, we see lots of setup and teardown, and this line, which invokes a method call and waits for the response to arrive back.

    response = source_module.call_module_method_async(
        destination_device_id, destination_module_id, method_invoke_parameters
    ).get()

The adapter

The call_module_method_async is a function on the module client adapter interface (class AbstractModuleApi), and since most of our tests use REST for interop, the implementation is in class ModuleApi inside rest_module_api.py. The implementation is actually more complex because of decorators and syntax and other complications, but the core of it is this function, which calls into an object that was generated by autorest which crates the actual HTTP transaction for the REST call.

    self.rest_endpoint.invoke_module_method(self.connection_id, device_id, module_id, method_invoke_parameters)

This translates to an HTTP operation calling into the module that is being tested.

PUT /modules/module_01/moduleMethod/myTestDevice/myTestModule/
{
    "methodName": "test_method",
    "payload": '"Look at me, I\'ve got payload!"',
    "responseTimeoutInSeconds": 75,
    "connectTimeoutInSeconds": 60
}

The wrapper

Inside the module being tested, there is a REST server that implements the REST API that we're using to test. This is basically the same surface as AbstractModuleApi. The wrapper is an app that was generated using swagger.io code generator tools to serve the module api as a REST surface. Each SDK uses a different HTTP server, and each SDK has different generated code, but all SDKs expose the same REST surface for each given API. It's this common REST surface that gives us the ability to script the same test cases for all languages.

The glue

Inside of the wrapper class, there are "glue" objects for each API. Right now, every language has module glue, service glue, registry glue, and some have device glue. You can usually find the glue easily (for node.js, it's in moduleGlue.js, for C, it's in ModuleGlue.cpp, etc.). The connection between the auto-generated wrapper and the glue needs to be done by hand.

For C#, the swagger.io tools generates a file called ModuleApi.cs, which has an empty implementation:

    public virtual IActionResult ModuleConnectionIdModuleMethodDeviceIdModuleIdPut([FromRoute][Required]string connectionId, [FromRoute][Required]string deviceId, [FromRoute][Required]string moduleId, [FromBody]Object methodInvokeParameters)
    {
        // Some auto-generated boilerplate
    }

A developer has manually inserted code to call the glue:

    public virtual IActionResult ModuleConnectionIdModuleMethodDeviceIdModuleIdPut([FromRoute][Required]string connectionId, [FromRoute][Required]string deviceId, [FromRoute][Required]string moduleId, [FromBody]Object methodInvokeParameters)
    {
        Task<object> t = module_glue.InvokeModuleMethodAsync(connectionId, deviceId, moduleId, methodInvokeParameters);
        t.Wait();
        return new ObjectResult(t.Result);
    }

And inside ModuleGlue.cs we have code that calls into the C# SDK and return the result.

    public async Task<object> InvokeModuleMethodAsync(string connectionId, string deviceId, string moduleId, object methodInvokeParameters)
    {
        Debug.WriteLine("InvokeModuleMethodAsync received for {0} with deviceId {1} and moduleId {2}", connectionId, deviceId, moduleId);
        Debug.WriteLine(methodInvokeParameters.ToString());
        var client = objectMap[connectionId];
        var request = _jobjectToMethodRequest(methodInvokeParameters as JObject);
        Debug.WriteLine("Invoking");
        var response = await client.InvokeMethodAsync(deviceId, moduleId, request, CancellationToken.None).ConfigureAwait(false);
        Debug.WriteLine("Response received:");
        Debug.WriteLine(JsonConvert.SerializeObject(response));
        return new JObject(
            new JProperty("status", response.Status),
            new JProperty("payload", response.ResultAsJson)
        );
    }

The most rigorous code in the wrapper is inside the glue files. Because of this, it's easy to confuse and intermix the terms "wrapper" and "glue".

The most error-prone code is in the wrapper, where the auto-generated functions call into the glue. This is because it needs to be hand-merged every time the wrapper code is re-generated.