.NET Core 3.0 REST API Template

.NET Core 3.0 has just been released.
.NET Core REST API Template could not miss such an upgrade. So, here it is :).

This new version is not only a technical upgrade. It includes:

  • DI according to .NET Core 3.0 practices
  • dependencies minimization
  • better XML comments to go deeper in OpenAPI/ Swagger specifications
  • “application/json” default setting
  • global error handling and logging with the appropriate code for implementing two different solutions:
    • middleware-based
    • lambda method-based
  • small fixes

For now, the .NET Core 3.0 version of the extension will remain in preview because of the dependencies that have not released in GA.

That said, the extension is fully functional and you can start using it.

.NET Core 3.0 REST API Template

Azure IoT Device Simulator Vs Microsoft Device Simulator (Accelerator)

IoT Development

Working on IoT global solutions is very exciting by the number of topics involved: functional valuable features, business value, devices, data, security, data analysis, AI, etc

Testing the solution with real devices is not always efficient and not even possible. So, sooner or later, IoT developers are confronted with the problem of needing devices but not having real ones.
This is where IoT developers start building their own applications or “IoT Simulators”. This is also  where I see a lot of people (developers) coding all more or less the same thing in order to test their solution …. or if not exactly the same thing, at least similar enough to start wondering: “wait a minute, wouldn’t be helpful taking the tool I created for myself, clean it a bit, make it at least ‘deliverable’ and share it?” :).

 

IoT Simulators in the “market”

Many IoT solution providers or editors have provided:

  • pieces of code simulating partial behaviors of devices
  • small solutions but not covering all the important functional flows
  • whole simulation platforms (more or less customizable)

Most of them remain focused on data generation.

However, I still felt the need to go further and build a whole IoT simulator covering most common IoT use cases. This means data generation but also other C2D and D2C flows.

This led me to build a tool for my own needs that has become Azure IoT Device Simulator now.

 

Differences with Microsoft Device Simulation Solution Accelerator

 

In the last years, I have been working with Microsoft’s ecosystem. So, the rest of the post will be related to Microsoft technologies.

Microsoft understood the need for simulating devices quite soon and published:

The simulation platform is a web solution that goes quite far and allows defining:

  • device types
  • device messages
  • emission intervals
  • a few Cloud To Device (C2D) interactions (Direct Methods, as Olivier Bloch and Troy Hopwood present here in the IoT Show)

For now, the main focus seems to be data generation though. Microsoft’s IoT Teams release updates very often. Keep posted to see how the solution evolves because I am pretty sure they will complete the solution iteration after iteration.

Azure IoT Device Simulation follows another approach and tries to cover more IoT functional features (and an important part of Azure IoT Hub SDK‘s device-side technical features; only a few of them have not been used ….. for now 🙂 ).

This solution main characteristics are:

  • Local  (not a web solution), fully .NET Core based.
  • Lightweight tool (not a whole platform)
  • Easy integration with developer tools
  • Could be used for unit tests
  • Solution configurable in many aspects (certain parts of message contents need to be improved to avoid limitations)
  • Containerizable (Docker)
  • Could be used for building a whole simulation platform (scaling out with Kubernetes or similar)
  • Functional and technical features (covers an important part of client capabilities). Implemented functional features cover:
    • Cloud To Device (C2D)
    • Device To Cloud (D2C)
    • IoT Hub Modules
    • IoT Hub Twin (Tags and Properties)
    • More details here.
  • Free (unless you decide to build a whole simulation platform with Kubernetes, etc)

 

NOTE

The solution could even be deployed in a real IoT device as a binary application (.NET Core based) as well as a Docker container.

 

The simulation logic remains quite basic (simple random logic). However, the solution is extendable with custom logic implementations.

Message structures are configurable and rely on JSON templates. A strong link exists between simulation services and JSON templates. This should be reviewed in upcoming versions and reduce that level of coupling.

Other development tools

We cannot forget the tools available for Visual Studio Code and Visual Studio :

  • extension for Visual Studio Code
  • natively integrated into Visual Studio through the Azure workload and the Cloud explorer

Even though their philosophy is different from Azure IoT Device Simulator or Microsoft Azure Device Simulator’s, these tools add IoT capabilities directly into the IDE.

 

Visual Studio Code - IoT features
Visual Studio Code – IoT features

 

Visual Studio 2019 - IoT features
Visual Studio 2019 – IoT features

 

Iteration after iteration, they are getting richer and richer in terms of features.

As mentioned earlier, these extensions are:

  • development-oriented
  • IDE-oriented

They can be used for development or testing.

They do not allow:

  • simulating whole IoT device flows (not all the IoT flows are implemented. They are not built for that either.)
  • building platforms of devices.
  • building message advanced logic or templating.
  • industrializing the simulations.

 

Conclusion

Azure IoT Device Simulator is a complementary tool for IoT developers or testers. It can also be used for building whole IoT device simulation platforms. By default, it includes three types of messages (telemetry, error, commissioning). Message schema relies on JSON templates and, except a few dependencies with the implementation of the simulation service, the schema is totally customizable. The solution implements an important set of Azure IoT SDK capabilities covering a large field of IoT scenarios.
Azure IoT Device Simulator can be used as a standalone tool or as part of a global platform (Docker + Kubernetes or binaries + Apps).

Microsoft Device Simulation Accelerator is less technical, IDE-oriented (web) and very fast to deploy. In a few minutes, a whole IoT Device simulation platform can be built. The types of messages and the metrics sent by devices can be customized totally (it seems that the current version does not allow customizing the simulation logic). The current version of the platform seems to be data generation-oriented. Costs also need to be taken into account; Microsoft Device Simulation Accelerator is a SaaS solution (web front).

Azure IoT Device Simulator Vs Microsoft Device Simulator (Accelerator)

Azure IoT Device Simulator

What is this?

Azure IoT Device Simulator was initially created as a developer tool for my personal IoT projects. Quite soon I felt the need of making it more reusable and generic.

Knowing that all IoT Developers are confronted with this same need, I thought it was maybe useful either build a monetizable solution (but with the required support, etc) or publish it as an opensource project. Great question because both alternatives were quite attractive to me.

In the end, I decided to publish the solution as an Open Source project and make it live through contributions. That will allow me to focus on other side projects and I am pretty sure I will have the chance to live new business adventures quite soon :).

The solution is available here.

I hope you will find it useful. Feel free to contribute and improve it.

Enjoy :).

 

Features

The Azure IoT Device Simulator has for purpose to help IoT developers and testers. Indirectly it should help business too, accelerating part of IoT projects and reducing costs.

The solution implements different types of Cloud To Device (C2D) / Device To Cloud (D2C) flows between Microsoft Azure IoT Hub and the simulated device.

More details here.

Example of uses:

  • development tool for developers working in Microsoft Azure IoT solutions (cloud)
  • tester tool in IoT-oriented projects
  • scalable IoT simulation platforms
  • fast and simple development of IoT devices

Azure IoT Device Simulator logs

Azure IoT Device Simulator Logs

Global features

Many of the functional features implemented are:

  • device simulation
  • module simulation
  • device simulation configuration (JSON-based)
  • oriented module oriented configuration (JSON-based)
  • no specific limitation on the number of modules (only limited by IoT Hub constraints)
  • containerizable (Docker image)
  • message templates based on JSON
  • implementation of full IoT flows (C2D, D2C, C2M, M2C)

Functional features

Device-level (C2D/D2C)

Commands

  • Request latency test
  • reboot device
  • device On/Off
  • read device Twin
  • generic command (with JSON payload)
  • generic command
  • update telemetry interval

Messages The device can send messages of different types (telemetry, error, commissioning).

Twin Any change in the Desired properties is handled by the device.

Module-level (C2M/M2C)

The features at the module level are identical to the device features except for the latency tests.

Details.

Technical core

Global technical features

Functional features are based on these generic technical features:

  • telemetry sent from a device.
  • a device can contain one or many modules.
  • each module behaves independently with its own flows (C2M/M2C) and its configuration settings.
  • telemetry sent from a module.
  • messages received by a device.
  • messages received by a module.
  • commands received by a device.
  • commands received by a module.
  • Twin Desired properties changed notification (for devices).
  • Twin Desired properties changed notification (for modules).

D2C

Device-level

  • IoT Messages
  • Twins (Reported)

Module-level (M2C)

  • IoT Messages
  • Twins (Reported)

C2D

Device-level

  • Twins (Desired)
  • Twins (Tags)
  • Direct Methods
  • Messages

Module-level (C2M)

  • Twins (Desired)
  • Twins (Tags)
  • Direct Methods
  • Messages

Technical background

Development based on:

  • .NET Core 2.x
  • Microsoft Azure IoT SDK (Device capabilities, including IoT Hub modules)

 

Delivery type:

 

 

Azure IoT Device Simulator

IoT Hub – REST API – Visual Studio 2017 / 2019 project template

IoT Hub exposes REST APIs and SDKs to interact with it.

The fact is that production projects sometimes (often) require implementing intermediate layers (logic, security or whatever) between IoT Hub used features and project’s accessible endpoints. This can be done with custom SDKs or in a more service oriented approach, with applicative services (lastly, usually REST oriented).

You will find here a Visual Studio 2017 extension that creates REST APIS (.NET Core 2.1.x) solutions exposing IoT Hub features. Indeed, the template creates a whole solution, ready to use and optimizes effort on creating business value.

Features included in the template are:

  • IoT Hub device management:
    • create and register a device (with default tags)
    • disable a device
    • enable a device
    • delete a device
    • get a given device properties
    • search devices according to given critera
    • list devices
  • Cloud to device communication:
    • call Direct Methods through REST API requests
    • update device twins (tags)
    • update device twin (desired properties)
    • create IoT Hub jobs (ex: device firmware update) and follow up status
    • get IoT Hub’s jobs details

 

All that ready to use.

You only need to set the right IoT Hub connection string (ideally, secured or stored in a KeyVault) in appsettings.json file.

The template relies on works done in a previous VS2017 template, helping to accelerate REST API .NET Core 2.1.x solutions development.

This templates includes features like:

  • Swagger/OpenAPI generation by config
  • API and Swagger versionning
  • .NET Core 2.1.x DI settings (configuration, dependency resolution, IOptions<>, exception management, etc)

 

Next steps

One of the next steps in real world projects should be to secure the REST API, either by adding embedded security management code/settings in the VS solution or using Azure API Management.

 

 

IoT Hub – REST API – Visual Studio 2017 / 2019 project template

How to measure latency between devices and Azure IoT Hub

In many of my last projects, I wanted to control latency levels among devices and IoT Hub. For many of them, I was just curious and for other, it was a real customer requirement.

How to do it?

What I ended up implementing for a very simple latency control relied on the items below:

  • new message type called “latency” and containing request timestamp
  • new IoT Hub endpoint (Event Hub type) to receive all the routed messages requesting latency control
  • new route based on “latency” message type and sending latency messages to the corresponding endpoint
  • new Azure Function connected to the Event Hub mentioned earlier. This function called an IoT Hub Direct Method. Its payload contained the original request timestamp. That way, the device could math the time spent for the round trip.

 

NOTE: in this case, latency measured device to cloud communication and its corresponding response. Single latency measure (merely device to cloud communication) could be enough in other cases.

If I get some free time in the upcoming days, I will try to package the solution and publish it (at least a kind of PoC).

Hope it helps or gives ideas.

Feel free to complete or comment.

How to measure latency between devices and Azure IoT Hub

How to use VS 2019/2017 REST API Template for .NET Core 2.1.x?

When it comes to building REST APIs we are often confronted with common and repetitive technical issues. That led me to try to industrialize and factorize the creation of REST API solutions.

Visual Studio 2019/2017 REST API Template for .NET Core is the first attempt of building a Visual Studio 2019/2017 project template (solution template actually) for creating a complete .NET Core 2.1.x solution fully usable (.NET Core 3.x available here).

Benefits

  • Industrialization
  • Fast
  • Very simple to use
  • Repeatable structure (easier to handle, understand and maintain)
  • Extendable
  • Allows to concentrate effort on creating value (business value most of the time)

 

How to install it?

Very easy.

Through a web browser

Template
VS 2017 – .NET Core REST API Template

 

Through Visual Studio

  • Open Visual Studio 2019/2017.
  • Go to Tools -> Extensions & Updates.
  • Look for the Extension (for instance, by “.Net Core”).
  • Once found, click the Download button.

 

VisualStudio-SearchExtension
Visual Studio 2017 – Tools and Extensions

 

What does the project template do?

Once the template installed, when we go to File -> New Project, we should see the screen below:

NewProjectAfterTemplateInstallation

Many inputs can be customized:

  • Name
  • Location
  • Solution name

 

After clicking on “Ok”, a Visual Studio solution is created with the structure below:

SolutionStructure_v0.2.1
Template structure

It is what it is; I guess the proposed structure can be discussed, reviewed, completed, etc. I am open to discussions :). The purpose was to propose something usable, simple, fulfilling common REST API projects technical requirements (from the production point of view) and allowing to focus most of our effort on creating business value.

Feel free to change the content, adapt it to your requirements (you should ;)) or to provide feedback in order to evolve the template and share that capitalization.

 

How to use VS 2019/2017 REST API Template for .NET Core 2.1.x?

Azure IoT Hub – Routing

In IoT projects it is quite typical that devices either are from different types or they send messages of different types. No matters the reason, IoT Hub ends up receiving different types of messages.

Before routing features existed, all the messages were treated through a single “pipeline” and eventual routing operations were coded “by hand” over that unique pipeline. Nowadays, things have changed and IoT Hub includes native routing features.

Let us see what it does and how it works.

 

Endpoints

Endpoints can be reached and defined from IoT Hub’s left-side menu (Messaging Section).

RoutesEndPointsQueries
IoT Hub’s Messaging Menu

 

Endpoints management page allows to:

  • create custom endpoints
  • configure custom endpoints
  • delete custom endpoints
  • configure built-in endpoints
    • Default events endpoint
    • Cloud to device feedback endpoint
    • File upload notifications endpoint

 

NOTE: the number of custom EndPoints is limited (as of now, 10 per IoT Hub)

 

Creating a custom endpoint is pretty straightforward. It only requires:

  • an endpoint name
  • choosing its type
    • Event Hub
    • Service Bus Queue
    • Service Bus Topic
    • Azure Storage Container

 

EndpointCreation
IoT Hub – Endpoint creation

 

Routes

Once endpoints are created, it is time create the routes. The process is pretty simple too:

RouteCreation
IoT Hub – Route creation

It requires the settings below:

  • A route name
  • Data Source type:
    • Device Messages
    • Twin Change Events
    • Device Lifecycle Events
  • Endpoint (one of the previously created or a built-in endpoints)
  • Query string the routing is built over

 

Queries are written in a T-SQL-like language and based on:

  • message properties (kind of message metadata properties, settable by IoT SDK when building messages)
  • message content properties (usually, more related to the business core of the project)

 

Ex:  status = 500 and $body.messageType = ‘error’

 

Built-in routes

IoT Hub includes built-in routes that deserve a couple of words:

  • Default route, where devices’ messages go through by default
  • File upload, that can be used by devices to upload “big” messages (ex: any kind of media, collection of messages stored locally, etc).
  • Twin notifications, could be used by IoT Hub every time a twin is updated, in order to monitor or notify changes
  • Device lifecycle, used by IoT Hub to notify device lifecycle oriented messages (ex: device created, deleted, disabled, enabled, etc)

 

Things to keep in mind

If in your project, routing queries need to be based on messages’ content, keep in mind it must be “understandable”. Some device manufacturers or MCUs may have no choice and could send hexadecimal messages (or under any other custom serialization format).

Queries can be based on message properties too.

 

Conclusion

Routing feature is really useful (literally used in all real world projects I have been involved in since the feature was rolled out). It offers the possibility to build separate and customized treatment workflows, depending on message types and project requirements on them.

 

 

Azure IoT Hub – Routing