Building IoT applications is a complex task. It requires the integration of a variety of protocols and services, a secure and flexible backend to manage and gather huge volumes of time series data streams from ’things’ and services. Developers need a means to pre-and post process and mechanisms to alert and react to changes in both the physical and online world.
While it is possible to use traditional programming tools and techniques for the IoT, several companies including our own have begun to introduce new tools and platforms to simplify IoT integration and application development. Way back in 2008, inspired by a web mashup tool called Yahoo Pipes, we launched an early version of the WoTKit Processor tool.
Yahoo Pipes used to watch ebay prices
Yahoo Pipes allows users to create pipes that retrieve and process data from different online sources to create web data “mashups” for use in your own applications. While not designed specifically for the IoT, we recognized that as ‘things’ are connected to the internet, they will become available for tools like Yahoo Pipes to use as data sources and sinks. We developed the WoTKit Processor to use a visual data flow programming paradigm like Yahoo Pipes; users on the cloud-hosted Processor created pipes made up of input, data processing and output modules. Unlike Yahoo Pipes that processes data from web data on request, the Processor was event-based, and acted on data in real time as sensors and other data sources pushed data into the WoTKit platform.
WoTKit Processor for controlling sprinklers
Around the middle of last year (2014), we came across an open source project called Node-RED from IBM’s Emerging Technology group. The first thing we noticed about Node-RED were its similarities with our own Processor tool – it was obvious that IBM researchers and Node-RED authors Nick O’Leary and Dave Conway-Jones @ceejay understood many of the challenges around building an IoT application. The initial focus for Node-RED was to address the issues around device-level integration. It can be a real pain to deal with the huge variety of protocols and services involved in connecting to things, just getting them on the Internet. While we used python scripts to convert a variety of protocols to HTTP, Node-RED made developing these scripts much easier by providing a similar visual data flow paradigm of that used by Pipes and our Processor, only running on devices like the Raspberry Pi directly. This makes it it super easy to integrate a variety of connectivity options, protocols and services with these devices.
Node-RED example flows
We were pretty excited by this; we knew our Processor tool was a great tool for IoT developers, so we wanted to make sure Node-RED could replace our processor and hopefully make it easier. We embarked on an evaluation of the two systems (for the hard core techies out there, we published a paper comparing the two systems which you can read here. That paper also included some of our ideas on a distributed data flow system – but that’s another blog! )
While it definitely has a cool UI, what was (and still is) most impressive about Node-RED is the huge community of developers and variety of nodes they contribute for others to use in their IoT projects. They range from nodes for making it easy to connect to the GPIO pins on a Rasperry Pi to connecting to on line notification systems like PushBullet and social networks like Instagram. We decided that we should see if we could find a way to leverage the large community and growing ecosystem of nodes available to Node-RED to enhance or replace our Processor.
One of the key requirements for our platform was that it run as a multi-user/tenant cloud service. It needed to have the ability to host multiple data flows in a single server, ensuring that event processing is distributed between flows, and that each flow gets it fair share of the host resources. While Node-RED is delivered as a web service, it was obviously designed for hosting one ‘flow’ on one device/host. To potentially replace the Processor, we needed Node-RED to be deployed as a multi-user and multi-flow system. However, to take advantage of the open source community’s work on both the Node-RED platform and associated nodes, we did not want to make significant changes to the Node-RED architecture
After some email exchanges with like minded developers that we met at the IoT Conference at MIT in 2014, and exchanging messages with Nick O’Leary and others on the node-red mailing list, we came up with the idea for FRED – a front end for node-red.
FRED fronts multiple instances of node red on one or more cloud servers. It’s essentially a ‘smart’ proxy, responsible for maintaining the link between users, their login credentials, session, API keys and their node-red process in the cloud, associated configuration and flows. It makes use of node-red largely ‘out of the box’ so that as NR evolves and new service nodes are made available, FRED users can reap the benefits immediately without needing to create and maintain their own dedicated node-red servers. In addition to the free FRED community service, a version of FRED has now become a key component of the Sense Tecnic platform offering, replacing our Processor tool.
We hope you give FRED and the WoTKit community platform a try. You can sign up directly for a free account at:
We’d love to learn and share with the community how FRED and Node-RED can help build IoT applications. Send us a tweet @sensetecnic and let us know what projects you’re working on. Feel free to contact us if you think we can help you build or host your IoT project.