(Target Readers: IoT systems arcitects, Developers)
I started a new blog - on a RapberryPi, for my wife's business early 2020 - after I learned of www.ghost.org as open source blogging platform , and for my hobby - IoT, in March 2020. We are full swing covid-19 on a South African government sanctioned 21 day lockup at home, so an ideal time to relook my blogs. I brought my old blog, that I started in 2016 on blogspot accross to this blog - still some useful posts here, and shows the history of my thinking quite well. I was quite suprised when I looked at the stats of some of the posts before I deleted the site, some of them have more than 7500 reads each ! Should have monotized it in 2016!
The last blog post on that site was Feb 2017, "Reporting the Geyser Switch State". So, some of you might wonder - why do I startup a blog again, after last posting early 2017? What have we done IoT-wise since then?
Well, apart from getting really busy at my work, with lost of international travel the past three years, I changed my views of how to build and depoy IoT solutions quite a lot. Let me explain.
All the solutions I built up to this point, used Node-RED on the a RaspberryPi to craft, test, and run the solutions that controlled functions in our house. I started running into typical uptime problems with running a 'mission-critical' system (well, if the lights or geyser does not work, and I am not at home, for my wife this is mission critical). Problems such as upgrading the Node-RED, or maintanance of on of the devices that sends info to the Rapsberry Pi, ie. adding another to the solution, etc.
Furthermore, family members started asking that I also deploy a system for them - I realised that if I struggle to maintain my own system, it will be a nightmare to run other systems as well. After 3 years - of mostly only working on airplanes - my house have now been running on new solution incarnations, where the principles I chose to build the solutions with, have created a whole new set of maintainable code, which had close to zero downtime.
Redesign Principles for a more maintainable solutions.
We ended up with five principles that guided us in our top-to-bottom re-design of the solutions that runs our house. They are:
- Improve Continous Development & Integration practices of the solution.
- IoT Devices Central vs. Local Control to a mandate.
- Not configuring IoT Devices in Node-RED flows, but rather in a settings file.
- A standardised communications Protocol.
- Standardising storage structure of all IoT device events and information.
- Separate loosely coupled systems integrated trhough a message bus
Principle 1: Continous Development practices.
The Problem - Deploying solutions. To depoy combinations of Node-Red, databases, etc, in Dev and Production became too much to do by hand. I started writing bash shell scripts to do this, but was concerned about the control of the passwords, etc, in the scripts. The scripts became more complex - to the point where I had to do more on them, and over time, the maintenance became a nightmare.
The Solution - Ansible, Docker & git. After in-depth research of possible solutions, and matching this to our needs, we ended choosing Ansible - this has changed my life. All my deployments - both the development - Docker on Mac, and to Production - Docker on Raspberry Pis is done with Ansible. I also started with storing all my code, like Node-RED flows & others in github and bitbucket - and setup Ansible Playbooks, using a bash menu structure on my Mac with which to do all configuration, and deployments, from firewalls, nginx, databases, Node-RED, to each of them having their passwords stored in secret vault files, deployed with ansible-vault. I guess we now have a continous development & integration strategy deployed, ready to handle the nuances of every client's differences in Ansible settings files.
Principle 2: Central vs. Local IoT control.
The Problem. After depoying light switches, the sprinkler system, geysers and others with IoT devices, and if something went wrong with the central control via MQTT and Node-RED, our home did not have fail-back capabilities. This sometimes happened because of power failure impacting the Raspberry Pi that ran the solutions, or upgrades gone wrong.
The Solution - decentralised control with 'mandates'. The solution was to create a 'mandate' to the IoT device, served from the central system, for when they are able to decide themselves on an action, even if the central control system (MQTT --> Node-RED on RPi) was offline. An example is the sprinkler system - we configured them to receive a mandate on switching (can switch on/off on pre-set times) as a variable - via MQTT - and once they have the mandate they switch each leg on/off based on the mandate - even if the central system cannot be reached.
Principle 3: Standard code, with home config external to the code
The Problem - Node-RED is too easy to use, decreasing maintainability across systems deployed. As Node-Red is so easy to use, lots of the home configuration was in flows - for instance the name of the devices that switched on lights, that was linked back to the actuator - on a user interface, etc. The problem was -that to do maintenance after months - one forgets where in the flows the actual settings are lying. And this is easy, as I sometimes had a running system for 6 months+ before I wanted to make changes.
The Solution - a coded system with all settings for the home external to the code in settings files. We used YAML files that carries all the settings for the house. none in the code base, thus, the maintenance for adding or removing devices and 'things' to the house - is not done in the code, but in yaml. YAML? - stands for 'yet another markup language' - so so easy to write and read - with which one can build complex configurations.
Principle 4: Standardised Comms protocol
The Problem - different devices has different 'languages' they communicate in. We tried out and built several of my own IoT devices, mostly on the ESP8266 wifi micro-chips, but also Adafruit's M0 Feather, and PyCom's WiPy, and RF solutions for where wifi will not work:
- Sonoff's ESP8266-based switches, these I flash with ESP_Easy firmware (their are other ones, like Tasmota, but I fell in love with the community, and are now part of them now);
- Sonoff's RF to wifi bridge - ESP8266-based, whih we flash with open-source OpenMQTTGateway;
- Shelly ESP8266-based devices for switching, energy monitoring - their firmware is so fantastic - I leave it's firmware on, even if you can load ESP_Easy.
- Paradox Alarm, with the open source project pai;
- and, a view other R&D things...
All of the above, are able to use protocols MQTT 'message qeueuing for telemetry transport' - a lightweight 'Organization for the Advancement of Structured Information Standards' (OASIS) standard, and / or derivitives of the Machine-to-Machine service layer 'Constrained Application Protocol' CoAP, also a RESTful protocol.
In the 1st 3-years of building home control systems, we thought this will be enough, but over time we realised that the fact that devices are loosly coupled, and can speak to one another, does not mean that the 'language' they use to speak, is standardized yet, for low maintenance of end-to-end loosly coupled systems.
We needed a common 'language'... We studies SAP, Apple, and AWS's standards they were driving into their IoT platforms. We looked at opensource standards, a promissing, developing one is Web-of-Things, a w3 Consortium standard.
The Solution - homie. In 2017 we built our own protocol to solve this mess, which we called iotp_dig2, (digital twin from IoTPlay) - ean our home on this protocol for 18 months, but soon realised - as we added more devices form other vendors, that the standard cannot keep up - we needed a 'body' that helped to keep up with improving the standard to new nuances of things to be controlled and read.
Enter homie, an open-source crowd, that built a standard for different open-source home automation systems, but that only uses mqtt as its transfer protocol. We found it good enough to use, and easy enough to create protocol extentions to handle things homie standards have not considered yet.
But, we had bigger plans .. we did not want to use the open-source home controllers they catered for, but rebuilt one on Node-RED - as we think that with our understanding of standards - we can control much bigger things than houses. We are also Apple affectionados, and found the open-source product HomeBridge - with deep API's and an active community, fit in snugly with our views of loosly coupled systems integred through a message-bus.
Principle 5: Standard storage structure, device info & events
The Problem - storing states of IoT devices. In the first thre years - we ran our home with a MariaDB database, but found the pre-json storage db (it now can do this), very restictive for the multi-layered needs of IoT devices and things.
- devices, and device attributes;
- nodes, and node attributes;
- node properties, and property attributes.
Principle 6: Putting it All together
The Problem - Big Dreams = integration nightmares. Our dreams of all the solution components we wanted to have caused integration nightmares.
The Solution - loosly coupled systems with message-bus for integration. The solution was to keep systems apart - easy with containerized apps using Docker, which we integrated with a message-bus between the systems. Se diagram below. More about this in future posts - or visit the home digital twin project on github.
What can you do with the homedig2 framework?
A lot. monitor energy, do smart energy allocations & switching schemes, by a multi-protocol switch between commercial- and non-commercial IoT- and small factory or farm environments.
"Homedig2" - stands for 'home digital twin'. What is a 'digital twin'? See Wikipedia's definition here.
Keep watching this blog...