In this article, I will describe how to publish data from a Spark Core to an Azure Event Hub for real-time processing using Azure Mobile Services as a message proxy.
Spark OS is a distributed operating system for the Internet of Things that brings the power of the cloud to low-cost connected hardware. Spark provides an online IDE for programming a Wi-Fi enabled Arduino-like device known as the Spark Core. Azure Event Hubs are a highly scalable publish-subscribe ingestor that can intake millions of events per second so that you can process and analyze the massive amounts of data produced by your connected devices and applications. Once collected into Event Hubs you can transform and store data using any real-time analytics provider or with batching/storage adapters.
To begin, I took an approach of using the Event Hubs Rest API Send Event. This seemed straightforward, simply create a request with the appropriate Request Headers over HTTP as HTTP and HTTPS are mentioned as supported in the documentation. However, when sending this request over HTTP, I received “Transport security is required to protect the security token” when including the necessary “Authorization” header. This poses a bit of a problem as the light-weight Spark device is unable to perform the computations necessary to send SSL requests.
Azure Mobile Services to the rescue!
I first created a new Azure Mobile Service with a Javscript backend:
Next, I created a new API within the service named “temp”:
Finally, create an Azure Service Bus with an Event Hub by following the instructions in Hypernephelist’s “Sending data to Azure Event Hubs from Node.JS using the REST API“.
The idea being, that I could successfully send data to the Mobile Service API as documented in Brian Sherwin’s “Wiring Up the Spark Core To Azure” then forward this data to the event hub using the information provided in Hypernephelist’s “Sending data to Azure Event Hubs from Node.JS using the REST API“. Essentially creating an Proxy via Azure Mobile Services to get data from the Spark in to an Azure Event Hub.
Let’s begin, by building the Event Hub Proxy in the “temp” API. This API will require Custom Node.JS packages that can be installed by following Redbit’s “Using Custom NodeJS Modules with Azure Mobile Services“. Follow the instructions and be sure to run an npm install for https, crypto, and moment as these are required to generate the SAS Key for sending data through the Event Hub rest Service.
The actual API code is below (with heavy reliance on Hypernephelist’s example), you will need to modify this by editing in the Azure API editor within the Azure portal, or modifying on disk after cloning per Redbit’s instructions. Be sure to edit the namespace, hubname, my_key_name, and my_key variables with the appropriate values from your Azure Event Hub.
/************************Begin AMS Code**********************
var https = require('https');
var crypto = require('crypto');
var moment = require('moment');
exports.post = function(request, response) {
sendTemperature(JSON.stringify(request.body));
console.log(request.body);
response.send(statusCodes.OK);
};
function sendTemperature(payload) {
// Event Hubs parameters
var namespace = 'EVENTHUBNAMESPACE';
var hubname ='EVENTHUBNAME';
// Shared access key (from Event Hub configuration)
var my_key_name = 'KEYNAME';
var my_key = 'KEY';
// Payload to send
//payload = "{ \"temp\": \"100\", \"hmdt\": \"78\", \"subject\": \"wthr\", \"dspl\": \"test\"," + "\"time\": " + "\"" + new Date().toISOString() + "\" }";
// Full Event Hub publisher URI
var my_uri = 'https://' + namespace + '.servicebus.windows.net' + '/' + hubname + '/messages';
// Create a SAS token
// See http://msdn.microsoft.com/library/azure/dn170477.aspx
function create_sas_token(uri, key_name, key)
{
// Token expires in one hour
var expiry = moment().add(1, 'hours').unix();
var string_to_sign = encodeURIComponent(uri) + '\n' + expiry;
var hmac = crypto.createHmac('sha256', key);
hmac.update(string_to_sign);
var signature = hmac.digest('base64');
var token = 'SharedAccessSignature sr=' + encodeURIComponent(uri) + '&sig=' + encodeURIComponent(signature) + '&se=' + expiry + '&skn=' + key_name;
return token;
}
var my_sas = create_sas_token(my_uri, my_key_name, my_key)
//console.log(my_sas);
// Send the request to the Event Hub
var options = {
hostname: namespace + '.servicebus.windows.net',
port: 443,
path: '/' + hubname + '/messages',
method: 'POST',
headers: {
'Authorization': my_sas,
'Content-Length': payload.length,
'Content-Type': 'application/atom+xml;type=entry;charset=utf-8'
}
};
var req = https.request(options, function(res) {
//console.log("statusCode: ", res.statusCode);
//console.log("headers: ", res.headers);
res.on('data', function(d) {
//process.stdout.write(d);
});
});
req.on('error', function(e) {
//console.error(e);
});
req.write(payload);
req.end();
}
/************************End AMS Code**********************
Finally, we need to set up the Spark Core with appropriate code to push data to the API in our Mobile Service. I leveraged the HttpClient as it has great logging features for debugging and is a bit easier to wield compared to Spark’s lightweight TCPClient. I also import SparkTime.h to generate the timestamp for messages from the Spark itself. Simply flash this code to your spark device, taking care to appropriately modify the AzureMobileService, AzureMobileServiceAPI, AzureMobileServiceKey, and deviceName variables. Note that the payload sent in this particular example corresponds to the expected payload in the Connect the Dots Project from MSOpenTech. This implies that there will soon be support for the Spark Core in this amazing project!
/************************Begin Spark Code******************
// This #include statement was automatically added by the Spark IDE.
#include "HttpClient/HttpClient.h"
// This #include statement was automatically added by the Spark IDE.
#include "SparkTime/SparkTime.h"
String AzureMobileService = "MOBILESERVICE.azure-mobile.net";
String AzureMobileSeriveAPI = "APINAME";
char AzureMobileServiceKey[40] = "MOBILESERVICEKEY";
char deviceName[40] = "SparkCore";
UDP UDPClient;
SparkTime rtc;
HttpClient http;
void setup()
{
rtc.begin(&UDPClient, "north-america.pool.ntp.org");
rtc.setTimeZone(-5); // gmt offset
Serial.begin(9600);
delay(10000);
}
void loop()
{
delay(5000);
unsigned long currentTime;
currentTime = rtc.now();
String timeNowString = rtc.ISODateUTCString(currentTime);
char timeNowChar[sizeof(timeNowString)];
strcpy(timeNowChar, timeNowString.c_str());
char payload[120];
snprintf(payload, sizeof(payload), "{ \"temp\": \"76\", \"hmdt\": \"32\", \"subject\": \"wthr\", \"dspl\": \"%s\", \"time\": \"%s\" }", deviceName, timeNowChar);
http_header_t headers[] = {
{ "X-ZUMO-APPLICATION", AzureMobileServiceKey },
{ "Cache-Control", "no-cache" },
{ NULL, NULL } // NOTE: Always terminate headers with NULL
};
http_request_t request;
http_response_t response;
request.hostname = AzureMobileService;
request.port = 80;
request.path = "/api/" + AzureMobileSeriveAPI;
request.body = payload;
http.post(request, response, headers);
Serial.print("Application>\tResponse status: ");
Serial.println(response.status);
Serial.print("Application>\tHTTP Response Body: ");
Serial.println(response.body);
}
/************************End Spark Code********************
Voila! I am able to verify my Spark is appropriately forwarding data to my ConnectTheDots portal!
We can also verify / debug by connecting to our Spark Core over serial and monitoring the output of HTTPClient
I absolutely love developing on the Spark device do to it’s simplicity to update and convenient online IDE. Now with the power of Azure, we can real-time analyze data coming from one of these devices!
You can find the latest code included in this project at the DXHacker/SparkEventHub repo on Github.