Bachelor Thesis. Evaluating IoT cloud platforms in the context of smart buildings

Transkript

1 Faculty of Technology and Society Computer Engineering Bachelor Thesis Evaluating IoT cloud platforms in the context of smart buildings Utvärdering av IoT-molnplattformar för användning inom området smarta byggnader Gustaf Bohlin Anton Hellbe Exam: Bachelor of Science in Engineering in Computer Science 180hp Area: Computer Science Date for examination: Examiner: Johan Holmberg Supervisor: Fahed Alkhabbas

2

3 Abstract Smart buildings is a common application for both Internet of Things (IoT) devices and cloud services. Recently cloud service providers such as Amazon, Google and Microsoft have started to offer IoT cloud platforms which consist of a class of services that provide a base for cloud applications utilized by IoT devices. However, there are many different providers of IoT cloud platforms and selecting one for an IoT solution for a smart building is difficult. In this thesis two IoT cloud platforms are evaluated in the context of smart buildings as part of an assignment given by Sigma Lundinova. To evaluate the IoT cloud platforms a common smart building scenario is realized by implementing a prototype using two different IoT cloud platforms. The development process makes it possible to evaluate how well the platforms support the development of the system that the scenario describes. The evaluation is based on information and experience from the process of developing the system using the IoT cloud platforms. The evaluation can be used as a guidance when selecting IoT cloud platform for an IoT solution intended for a smart building. i

4 Sammanfattning Smarta byggnader är ett vanligt användningsområde för både Internet of Things (IoT) enheter och molntjänster. På senare tid har molntjänstleverantörer som Amazon, Google och Microsoft börjat erbjuda IoT-molnplattformar. Dessa består av en klass av tjänster som utgör en bas för molnapplikationer som används av IoT-enheter. Idag finns det många olika leverantörer som tillhandahåller denna tjänsten och att välja en för en IoT-lösning är svårt. I denna rapport beskrivs utvecklingen av ett system som är vanligt förekommande i en smart byggnad. I denna rapport utvärderas IoT-molnplattformar för användning inom området smarta byggnader som en del av ett uppdrag från Sigma Lundinova. För utvärderingen implementeras ett vanligt scenario i en smart byggnad som en prototyp med hjälp av två olika IoT-molnplattformar. Syftet med detta är att utvärdera och jämföra hur väl IoT-molnplattformarna stödjer utveckling av systemet beskrivet av scenariot. Genom att implementera en prototyp insamlas underlag i form av kunskap och erfarenhet som används i utvärderingen. Utvärderingen kan användas som ett hjälpmedel för att göra det lättare att välja en IoT-molnplattform när man utvecklar IoT-lösningar för smarta byggnader. ii

5 Acknowledgements We would like to thank Sigma Lundinova for the opportunity to carry out this thesis on their behalf. Furthermore, we would like to thank Mathias Beckius at Sigma Lundinova for all the help and support with our thesis. We would also like to thank Magnus Krampell for his advice and feedback during the thesis work. iii

6 Contents 1 Introduction Background Problem statement and purpose Research questions Limitations Theoretical Background Internet of Things (IoT) Cloud services IoT cloud platforms Software Development Kit (SDK) Message Queue Telemetry Transport (MQTT) Serverless functions Smart buildings Heating, ventilation and air conditioning (HVAC) Related Work "Selecting the right IoT cloud platform" Comments "Fast-paced development of a smart campus IoT platform" Comments "IoT framework for Smart Buildings with Cloud Computing" Comments "What Does I(o)T Cost?" Comments "Design and Implementation of Intelligent HVAC System Based on IoT and Bigdata Platform" Comments Method Comparative study Constructing a scenario Comparison criteria Selecting the IoT cloud platforms to evaluate Evaluation Nunamaker and Chen s system development process Construct a conceptual framework Develop a system architecture Analyze and design the system Build the (prototype) system Observe and evaluate the system Results Constructing a scenario Comparison criteria Requirement Analysis Comparison criteria Selecting the IoT cloud platforms to evaluate iv

7 5.4 Construct a conceptual framework Problem Tree End device Edge device IoT cloud platform Web page Literature study Develop a system architecture End Device Edge Device IoT Cloud Platform Web page Analyze and design the system Hardware End device Edge Device IoT cloud platform Web page Build the (prototype) system IoT cloud platform End device Edge device Regulation Web page Observe and evaluate the system Constructing test cases Testbed Evaluation Discussion Related work Comparative study methodology Analysis of result Conclusions Contribution Future work Appendices 50 Appendix... A Appendix... B Appendix... C v

8 1 Introduction This chapter introduces the concepts of smart buildings and IoT cloud platforms, and how they can be used together. The chapter also introduces the assignment that the thesis is based upon, the research aim and the research questions. 1.1 Background Smart buildings is a common application for both Internet of Things (IoT) devices and cloud services. Usually smart buildings contain a number of IoT devices that together make the building classify as a smart building. For a smart building system to be able to perform data processing and allow for remote controlling of appliances, it can be connected to the cloud. Recently cloud service providers (CSPs) such as Amazon, Google and Microsoft have started to offer IoT cloud platforms which consist of a class of services that provide a base for cloud applications utilized by IoT devices. IoT cloud platforms integrates devices, networks, and applications [1]. These platforms hide implementation complexity from the user, because they support and enable IoT solutions by providing an ecosystem upon which IoT devices are built [1]. An example of an IoT solution that could be supported by an IoT cloud platform is "smart lights", e.g. lights in a building that are connected to an IoT cloud platform to allow for remote controlling as well as scheduling. This could be utilized to have the lights be automatically turned on in the morning, and turned off in the evening. This thesis investigates and evaluates IoT cloud platforms as a part of an assignment given by Sigma Lundinova. Sigma Lundinova is a consulting firm that specializes in electronics and embedded systems software. To carry out this assignment a common scenario for a smart building is realized and an appliance prototype is implemented. The common smart building appliance chosen for the scenario is a Heating, Ventilation and Air Conditioning (HVAC) system. In the scenario the appliance prototype is connected to an application deployed onto an IoT cloud platform. The development process of the system is meant to provide information and experience to base the evaluation upon. Sigma Lundinova requires that the evaluation covers at least two platforms and that the evaluated platforms satisfies the following requirements: PR1 The cloud service provider should offer technical support. The cloud service provider should offer technical support via both phone and . PR2 The platform should allow for direct access. No contact with the cloud service provider s support staff should be needed to create an account and start developing. PR3 The platform should have publicly available documentation. The documentation should be easy to access (e.g. publicly available) and cover basic usage. PR4 The platform should offer free trial. No costs for a certain amount of days or a certain amount of use. 1

9 PR5 The platform should not require hardware dependency. The platform should not require any hardware dependency such as special embedded hardware boards. PR6 The platform should provide example code. Example code describing how to use the platform in order to shorten time to get started should be available. 1.2 Problem statement and purpose According to Postscapes there exist 122 IoT cloud platforms today [2], and selecting one for a smart building implementation is difficult. Moreover, there appears to exists only a few papers that investigates and evaluates different IoT cloud platforms, and these papers mostly discuss the functional properties of the platforms and their performance [3][4][5]. The purpose of this thesis is to complement the existing papers with a comparative study on the application development process using an IoT cloud platform Research questions The research aim of this thesis is to evaluate IoT cloud platforms by implementing a common smart building appliance using an IoT cloud platform. The development process of the smart building appliance will provide information and experience that can be used to answer the following research questions: RQ 1 How can a heating, ventilation and air conditioning system be implemented using an IoT cloud platform? RQ 2 How well do existing IoT cloud platforms support development of a heating, ventilation and air conditioning system? 1.3 Limitations This research will be limited to only evaluate IoT cloud platform functionality needed to realize the scenario described in 5.1. The list of criteria used for the evaluation and what each criteria considers is presented in

10 2 Theoretical Background The concepts explained in this section are: IoT, Cloud services, IoT cloud platforms, SDK, MQTT, Serverless functions, Smart buildings and HVAC systems. 2.1 Internet of Things (IoT) The technical definition of Internet of Things is the network of physical things accessed through the Internet. These things contain embedded technology to measure and/or interact with the physical world [6]. Common applications for IoT are in industry, transportation, homes, as medical instruments, on animals etc. 2.2 Cloud services The capabilities of cloud services can be divided into three broad groups, Infrastructure as a Service (IaaS), Platform as a Service (PaaS) and Software as a Service (SaaS) [7]. Figure 1 shows the three services, and which part of the service the cloud provider manages. IaaS provides the resources data storage, servers and networking. The servers are usually accessed through virtualization to allow for efficient use of resources and lower costs. When purchasing IaaS you are normally granted access to a virtual machine. Data storage can be everything from databases to physical hard drive storage space. [7] PaaS is a service that makes it possible for consumers to deploy applications written in supported languages and using supported Figure 1: Cloud services groups libraries directly into the cloud. The benefit of using a PaaS is that the underlying application infrastructure (IaaS) is provided by the cloud provider. The cloud provider takes responsibility for the installation, configuration and operation of the underlying application infrastructure. This lets the consumer focus on developing the application without having to worry about scalability, hardware maintenance and hardware costs. [7] SaaS is a service that operates application software in the cloud. Users of SaaS applications do not manage the infrastructure nor the platform on which their application runs [3]. An example of SaaS is Google Docs, where Google is the cloud service provider that manages the application. 3

11 2.3 IoT cloud platforms IoT cloud platforms provide the support software that connects everything in an IoT system. An IoT cloud platform facilitates communication, data flow, device management, and the functionality of applications. What differentiates IoT cloud platforms from other platforms is that it is targeted at IoT. [8] Most CSPs also offer services to extend the platforms functionality. These services could for instance be storage, machine learning or analytics. The services are primarily tools used to customize applications after the application s needs and usually require some programming. 2.4 Software Development Kit (SDK) An SDK is a Software Development Kit used when developing software applications for e.g. devices, services, operating systems etc [9]. An SDK provides a set of tools, libraries, documentation, code samples, processes and or guides that allow for developers to create their own applications for a specific platform [9]. Examples of common SDKs are Android SDK, Java development kit (JDK) and Google App Engine SDK. 2.5 Message Queue Telemetry Transport (MQTT) Message Queue Telemetry Transport (MQTT) is a lightweight and flexible network protocol that is suitable for IoT. MQTT defines two basic entities in a network, a message broker and a number of clients. The message broker receives all the messages from the clients and routes them to the destination clients. A client is anything that can interact with a broker to receive and send messages. [10] Sending and receiving messages in MQTT is based on a publish and subscribe model. Clients that connect to the MQTT broker can subscribe to any message topic in the broker. When a client publishes a message in a topic, the MQTT broker forwards the message to all clients subscribed to that topic. [10] 2.6 Serverless functions Serverless functions, also known as Function as a Service (FaaS), is a service that runs server side functions without the customer managing their own server systems or their own server applications [11]. Instead the functions run on servers in the cloud, which means as a developer the servers are abstracted away. FaaS is event-driven which means that the functions do not start by themselves, instead they need to be called in response to a predefined event or trigger [12]. In an IoT system this event could for example be a new sensor value being published, the function can then read the value and respond back to the device if necessary. Example of different CSPs FaaS are Azure functions, Amazon Web Services s (AWS) lambda functions and Google cloud functions. 4

12 2.7 Smart buildings The concept of smart buildings has been around for a while, but the definition of what constitutes a smart building is poorly defined. Some argue that a smart building is one that is fully automated and needs no user interaction [13]. In this thesis Intertek s definition of smart home and in turn smart building was chosen since it is a clear definition. Intertek s definition is the following: A dwelling incorporating a communications network that connects the key electrical appliances and services, and allows them to be remotely controlled, monitored or accessed. [14] 2.8 Heating, ventilation and air conditioning (HVAC) A heating, ventilation and air conditioning system is one that provides thermal comfort and air quality in buildings [15]. Internal temperature in the building can be controlled by the heating and cooling. The ventilation maintains air quality inside the building. By connecting these HVAC systems to the Internet one could remotely operate the HVAC system and monitor temperature and air quality. 5

13 3 Related Work This chapter presents work that is related to this thesis. The papers presented are comparative studies, an implementation of a system using IoT cloud platforms and a framework for smart buildings with cloud computing. Their relevance to this thesis is also briefly explained and then further discussed in chapter "Selecting the right IoT cloud platform" Pankaj Ganguly [4] defines a basis set of selection and conducts a literature survey on different IoT cloud platforms. The problem stated is that there exists no list of requirements and related best practice of IoT cloud platforms. The only thing presented in related works is a framework that can be used for evaluation of IoT cloud platforms. However this framework does not include functional aspects such as device management, application protocols and analytics. Based on the functional aspects of IoT cloud platforms, a selection basis that can be used to compare IoT cloud platforms is proposed. Example of selections are application protocol, analytics, application support and security. The selection basis is used to conduct a literature survey and a comparison. The results presented is the selection basis, the literature survey and the comparison. The comparison is based upon the selection basis and shows how well the IoT cloud platforms fulfill the selection basis Comments The paper is relevant since it performs a comparative study between different IoT cloud platforms. Some of the functional aspects that the author uses as a selection basis can be reused in this thesis. 3.2 "Fast-paced development of a smart campus IoT platform" Armin et al. [16] constructed a smart campus IoT platform using Azure IoT cloud services as the backbone of the platform. The problem that the authors aim to solve is to enable open distribution of campus IoT data in association to student-life and academic activities. The authors present the development process as a three step process, design, implementation and experience. The design phase is to create a conceptual model of the platform architecture. The conceptual model includes the relationship between data contributors, cloud services and application developers. The implementation phase consists of modelling the cloud back-end services in order to connect all the parts of the system. Three related papers are presented but they focus on developing the IoT platforms from the ground up instead of using existing commercial cloud-based IoT platforms such as Azure. 6

14 The results presented is the application itself, but also the experiences gained from the development and deployment phases. The authors acknowledge issues faced in the development and deployment phases. Some of the issues faced were high billing costs, service disruption and connectivity Comments The paper contributes to this thesis by showing how an IoT cloud platform can be utilized and showing a potential architecture of the cloud back-end. Yet another contribution is the development and deployment experiences that the authors faced. This can be taken into account when developing the HVAC system on different IoT cloud platforms 3.3 "IoT framework for Smart Buildings with Cloud Computing" Enrique Carillo et al. [17] developed an IoT framework that aims to allow smart buildings to take advantage of cloud computing. They discuss the advantages of smart buildings and why they should be implemented. They also implement a smart building and test the different components in the system. In the related work section they bring up others work on communication and cloud computing. They present a paper that states how important communication standards are along with works that discuss advantages of cloud computing. The contribution of the paper is an IoT framework for smart buildings that takes advantage of cloud computing. This framework uses several end devices that are connected to a gateway through different communication protocols. The gateway in turn is connected to cloud services Comments Figure 2: Smart building architecture using cloud services This paper contribute to this thesis since they make an implementation of a smart building using cloud computing. The framework that they propose can be used for the smart building implementation. 7

15 3.4 "What Does I(o)T Cost?" Edua Eszter Kalmar et al. [18] conducted an investigation of different IoT cloud service providers pricing models. The paper focuses on four providers; Microsoft, IBM, Amazon and Oracle. A literature survey is performed on the different providers pricing models. To evaluate the different pricing models, two scenarios are constructed. The scenarios are ones that are commonly found in IoT applications. The survey is complemented by calculating an estimation of what the scenarios would cost if implemented with each provider. The paper presents that based on the number of devices, amount of messages and data used some providers cost less than others. Finally an actual implementation is done with one of the scenarios using IBM Bluemix as IoT cloud platform to find out if the calculations were correct. The result of the implementation was that it was slightly more expensive with a real implementation due to their calculation using too small message sizes Comments This paper is relevant to this thesis by showing how an evaluation can be performed on IoT cloud providers pricing models. The methodology used in this paper can be reused in this thesis to achieve a scenario-based comparative study. 3.5 "Design and Implementation of Intelligent HVAC System Based on IoT and Bigdata Platform" Tacklim Lee et al. [19] present a design and implementation of an intelligent HVAC system. The HVAC system implemented is a suggested solution to the problem that HVAC systems are not usually accessible other than from the control device, as well as having different profiles for different parts of a building. The HVAC system they implement is connected to the Internet and provides personalized settings for the users. By connecting the HVAC system to the Internet it is possible for users to monitor the state of the system and remotely control it. The system uses a Raspberry PI3 that collects the sensor data and communicates with a server to regulate the environmental variables by turning on/off fan, air conditioner and heating element. The contribution of the paper is an HVAC system that is connected to a user customized control center and real-time monitoring service. By adding these features to the system it makes it easier for users to interact and view the status of the system Comments The paper is relevant for this thesis by showing an implementation of an IoT HVAC system. It also proposes features such as remote controlling, monitoring and a desired temperature range for a HVAC system that can be reused in this thesis. 8

16 4 Method The method of choice is a mixed-method approach which consist of a comparative study and Nunamaker and Chen s system development method [20]. The comparative study is conducted with criteria which can be evaluated by developing and deploying an application onto the IoT cloud platforms. Nunamaker and Chen s system development method is used to develop the systems for the comparative study. 4.1 Comparative study A comparative study is the method of choice for the main contribution of this thesis. The comparative study compares a number of IoT cloud platforms based on predefined criteria. According to the research aim, an evaluation and comparison of IoT cloud platforms will be conducted. The comparative study method is a structured way of doing a comparison with the steps comparison criteria definition, platform selection and evaluation. The comparative study method is derived from two different papers to combine a scenariobased comparative study as well as a criteria based comparison. In the paper What Does I(o)T Cost [18] a method is used that implements two scenarios as a way to perform the comparative study of pricing models. In the paper Cloud Storage Providers: A Comparison Review and Evaluation [21], a step-by-step method is used to perform a comparative study between different providers. In the paper the steps taken are comparison criteria definition, selection of providers and performance bench-marking. The comparative study method used in this thesis is a combination of the step-by-step method and the scenario-based comparison. First a scenario is proposed, then the comparison criteria are defined and then a selection of providers is established. In the next step the scenario is implemented and an application deployed on the IoT cloud platform. After the implementation phase, enough information and experience have been gathered to perform an evaluation of the providers based on the predefined criteria. The difference between this thesis and the work done in Cloud Storage Providers: A Comparison Review and Evaluation [21] is what the evaluation is based upon. The work done in [21] is based upon performance benchmarking whilst this thesis focuses on application development. By combining the comparative study method used in [21] with the scenariobased comparative study method used in [18], they provide a method to answer the research questions stated in this thesis Constructing a scenario In this step of the comparative study a scenario for the system is constructed. The system is one that can be commonly found in a smart building and makes use of an IoT cloud platform. The result of this part of the comparative study can be found in

17 4.1.2 Comparison criteria In this step of the comparative study the criteria for the evaluation is defined. Using the scenario constructed in 5.1, a list of requirements is constructed. A literature survey is then conducted to find existing criteria that can be evaluated by satisfying the requirements. The existing criteria are then complemented by adding new criteria that can also be evaluated by satisfying the requirement. The result of this part of the comparative study can be found in Selecting the IoT cloud platforms to evaluate This step of the comparative study is to select the IoT cloud platforms to evaluate. This is done by first conducting a literature survey to find existing IoT cloud platforms. To reduce the list of IoT cloud platforms to evaluate the set of requirements defined by Sigma Lundinova is used to construct a smaller subset. The result of this part of the comparative study can be found in 5.3. The next step is Nunamaker and Chen s system development process to implement the scenario, in order to gather information and experience needed to grade the different criteria Evaluation The last step of the comparative study is to perform the evaluation and comparison of the IoT cloud platforms. The evaluation and comparison is done by using the criteria constructed in and the information gathered when developing the system. The criteria are graded on a scale from 1-5 based on information and experience from the development process. A short explanation and argumentation of the grade is also provided for each criteria. The result of this step in the comparative study can be found in Nunamaker and Chen s system development process The method of choice for the system development is Nunamaker and Chen s method for information systems development. This method was chosen because of its systematical and iterative approach for developing a system from a concept to a prototype. The method consists of a five step process which is visualized in figure 3. 10

18 Figure 3: Nunamaker and Chen s System Development Research Process. Based on [20] Construct a conceptual framework The first step of the process is to identify the problems and investigate what needs to be researched in order to solve them. The systematical approach used to identify the problems is constructing a problem tree. This tree gives a deeper understanding in exactly how the problems can be solved by solving smaller and easier problems. A literature study is also conducted with the aim to gain sufficient knowledge about the selected IoT cloud platforms, and their functionality in order to be able to solve the problems stated in the problem tree. The results from this step in the process are presented in Develop a system architecture This step of the process is to create an architectural view of the system and the different components in the system. This includes the components used, their relation to each other as well as their responsibility. All the relationships are visualized using a diagram. The results from this step in the process are presented in

19 4.2.3 Analyze and design the system This step of the process is to construct a conceptual model of the execution of the application. A systematic approach towards software design is conducted and UML documents are created. The UML documents created are sequence diagrams for the end device, the edge device, the IoT cloud platform and the web page. The electronic hardware is designed by selecting components and drawing a circuit diagram. The results from this step in the process are presented in Build the (prototype) system This step of the process is to build the system. The system is built by solving the problems in the problem tree constructed in By utilizing the diagrams drawn in the previous step, a systematical approach towards software development is achieved. The hardware in the prototype follows the diagram drawn in The results from this step in the process are presented in Observe and evaluate the system This step of the process is to verify the system. To do this the whole system is tested to confirm that all the functionality works as intended without complications. This is done by constructing test cases that verify that the requirements defined in are met. The test cases are then run on the systems. The results from this step in the process are presented in

20 5 Results 5.1 Constructing a scenario The scenario presented is one that can be used to collect the requirements for the system. Furthermore, the scenario also describes the functionality of the system. By implementing the system described by the scenario using an IoT cloud platform, information and experience can be gathered for the comparative study. "Simon and his colleagues enter a room in a smart building to have a meeting. In the room the temperature and CO 2 levels are monitored by a temperature and CO 2 sensor. If the temperature in the room is not in the range predefined by the building owner, the heating or the air conditioner is started automatically. If the CO 2 levels rise above normal level (which is 600 ppm [22]), the ventilation is started automatically in order to clean the air in the room. At any given time the building owner can also remotely control the heating, ventilation, and air conditioner." With this scenario many aspects of an IoT cloud platform s functionality such as sending sensor data to the cloud, controlling the appliance remotely and decision making are used. 5.2 Comparison criteria This section describes the process of constructing the comparison criteria. The comparison criteria was formulated using existing criteria found by conducting a literature study and by doing a requirement analysis. The requirements constructed in the requirement analysis are used to choose which existing criteria can be reused in the evaluation Requirement Analysis The requirements of the system were constructed by using the scenario in 5.1 to get a clear view of what the system needs to do. They also provide information about what functionality the system will need from the IoT cloud platform. The requirements of the system are the following: R1 The system should regulate temperature to be inside the desired range defined by the building owner. R2 The system should regulate CO 2 level to be less than 600 ppm. R3 The system s heating, ventilation and air conditioning should be remote controllable via a web page. R4 The system should display CO 2 and temperature sensor values on a web page. R5 The system should allow new devices to be added into the system. R6 The system should only allow authorized devices access to the system. R7 The building owner should be able to change the desired temperature range via a web page. 13

21 5.2.2 Comparison criteria The criteria were defined by using existing criteria and the requirements constructed in The existing criteria were found by searching IEEE, ACM and Google Scholar with the search string: "IoT application" "IoT platform" "comparison". From the search the two articles chosen are Selecting the right IoT Cloud Platform [4] and A survey of IoT cloud platforms [5] as they presented relevant criteria. The following criteria are partially from [4][5] and partially constructed using the requirements from C1 Communication management C1.1 Device to cloud communication This criteria considers message data formatting, sample code, documentation and how it is implemented. C1.2 Cloud to device communication This criteria considers message data formatting, sample code, documentation and how it is implemented. C1.3 Message routing The criteria message routing means the routing of messages and events from the IoT cloud platform to other services that the CSP offer such as storage, serverless functions, etc. This criteria considers the comprehensibility and documentation of message routing. It also considers the filtering of messages and events. C1.4 Communication protocols [4] This criteria considers the communication protocols supported to communicate with the IoT cloud platform and if they are suitable for an IoT solution. C2 Device management [4][5] C2.1 Device monitoring The criteria device monitoring means the monitoring of a device s state, e.g. the state of actuators and sensors. This criteria considers documentation, sample code and implementation of device monitoring. C2.2 Remote controlling The criteria remote controlling means controlling the state of for example an actuator remotely via a web page. This criteria considers documentation, sample code and implementation of remote controlling. C2.3 Device provisioning, taken from [4] The criteria device provisioning means adding new devices into the application on the IoT cloud platform and creating the necessary authorization. This criteria considers ease of use for a developer, documentation and interfacing options such as web portal, command line interface, etc. C3 Data processing C3.1 Run-time decision making The criteria run-time decision making means making decisions based on data received during run-time. This criteria considers the creation of a function, the 14

22 C4 Development [5] triggering of said function and documentation. It also considers the development environment provided by the CSP to develop the function. C4.1 Software Development Kit (SDK) This criteria considers the SDKs ease of use for a developer, installation of the SDKs and the SDKs documentation. C4.2 SDK languages [4] This criteria considers which programming languages the SDKs are available in. C4.3 Integration support for external application The criteria integration support for external application means how well the platform supports the process of developing an application that is not hosted on the platform nor is a provisioned device. The external application could for example be a web page. This criteria consider sample code, documentation and authorization implementation. In table 1 the criteria are matched against the requirements in to provide an overview on how the criteria can be evaluated based on implementation of functionality. R1 R2 R3 R4 R5 R6 R7 C1 Communication management x x x x x C2 Device Management x x x x C3 Data Processing x x C4 Development x x x x x Table 1: Criteria matching 5.3 Selecting the IoT cloud platforms to evaluate The literature survey identified the site Postscapes [2] that lists 122 IoT cloud platforms. The list of all can be found in Appendix A. The platforms in the list were matched against the platform requirements from Sigma Lundinova seen in 1.1. The filtering yielded 19 platforms listed in table 2. Amazon AWS IoT AT&T M2X Cloudplugs Cloudthing.io ForgeRock GE Predix Google Cloud IoT GroveStreams IBM Watson IoT Platform Intel R IoT Platform Microsoft Azure IoT Oracle Internet of Things Cloud PTC ThingWorx PubNub scriptr.io Splunk thethings.io Thing+ Bosch IoT Cloud Table 2: IoT cloud platforms eligible for comparison (listed alphabetically row-wise) 15

23 In the assignment given by Sigma Lundinova, it was stated that at least two platforms were to be compared. The platforms chosen for the final comparison are Microsoft s Azure IoT hub [23] and Amazon s Amazon Web Services (AWS) IoT [24], which are both PaaS. Microsoft and Amazon are according to Synergy Research Group [25] the leading cloud providers. A Survey of IoT Cloud Providers [3] appears to be the one article that cover both these two platforms, however their work focuses on the provider rather than the platform. Azure IoT hub and AWS IoT are chosen since Microsoft and Amazon are the two largest cloud providers and there appears to exist no articles that evaluate their IoT cloud platforms. 5.4 Construct a conceptual framework Problem Tree Figure 4: Problem tree for the prototype A problem tree was created to get an overview of the problems for each part of the prototype. The problem tree consists of four branches: end device, edge device, IoT cloud platform, and web page. The individual branches are broken down further in the following sections End device Figure 5: Problem tree for the end device The end device branch has three sub-branches: serial communication, hardware, and sensor and actuator interfacing. The serial communication is for the end device to send sensor values and receive commands. The hardware problem is the choice of components and construction of the actual circuit. The final problem, sensor and actuator interfacing, 16

24 is how the end device can change the actuator states as well as being able to read the sensors Edge device Figure 6: Problem tree for the edge device The edge device is acting as a gateway between the end device and the IoT cloud platform, explained further in The edge device s sub-branches are communication with the end device and IoT cloud platform. Sensor values are sent from the end device to the IoT cloud platform via the edge device. Commands are sent to the end device from the IoT cloud platform via the edge device IoT cloud platform Figure 7: Problem tree for the IoT cloud platform The IoT cloud platform branch has three sub-branches: message routing, regulation and device provisioning. Message routing involves routing messages between the IoT cloud platform and other services provided by the CSP. Regulation is broken down into three sub-branches: decision making, cloud to device communication and device to cloud communication. The regulation is processing data and making decisions regarding which commands to send to the edge device to regulate CO 2 and temperature. Device to cloud 17

25 communication is receiving the current temperature and CO 2. Cloud to device communication is sending commands to the edge device. The final sub-branch is device provisioning which means adding new devices into the system Web page Figure 8: Problem tree for the web page The web page branch has two sub-branches: monitor and remote controlling. The monitoring branch involves displaying the sensor values and the actuator states that the edge device sends to the IoT cloud platform. The remote controlling branch involves being able to remote control the actuators from the web page. By clicking e.g. a button the state of the actuator should toggle between on and off Literature study The literature study was conducted to gain knowledge on which functionality on Azure IoT hub and AWS IoT can be used to solve the problems shown in the problem tree. The problems discussed in this section are device provisioning, regulation and decision making, message routing, cloud to device and device to cloud communication, monitoring and remote controlling. The literature study is presented in a certain order because they are based on each other, e.g. message routing and regulation. Device provisioning The following functionality on Azure IoT hub and AWS IoT can be used to provision devices. Azure IoT hub - Identity registry Devices that want to connect to Azure IoT hub needs to be provisioned. To register a new device a device id must provided. In addition to the device Id, authorization also needs to be added for the device. Azure IoT hub uses both certificates and token based authorization. Furthermore it can be specified which resources the device will be able to access which is done through permissions. 18

26 AWS IoT - Thing registry Devices that want to connect to AWS IoT also needs to be provisioned. AWS IoT requires a device id, authorization and a policy when provisioning a new device. AWS IoT uses certificates for authorization and a policy to specify what resources the device will be able to access. Monitoring and remote controlling The following functionality on Azure IoT hub and AWS IoT can be used to monitor and remote control devices. Azure IoT hub - Device twin Device twin is an implementation of the concept digital twin [26]. Digital twin refers to a digital replica of a physical asset, e.g. an IoT device. The digital replica is represented as a JSON document that contains two types of properties, reported and desired. The reported properties represent the device s current state, e.g. status of one or more actuators. The desired properties are used to request the device to update its current state. The device will be notified when the desired properties of its digital twin are updated and can take action to match the desired and reported properties. The JSON document is called Device twin on Azure IoT hub. AWS IoT - Shadow service Shadow service is also an implementation of the concept digital twin [26]. It is also represented as a JSON document containing reported and desired properties, which are used in the same way as the Device twin. On AWS IoT the JSON document is called Thing shadow. Device to cloud and cloud to device communication Communication between device and cloud can be implemented using the Device twin on Azure IoT hub and Shadow service on AWS IoT. Azure IoT hub - Communication using the Device twin Azure IoT hub provides bi-directional communication between device and cloud. To send and receive messages Azure IoT hub uses endpoints which are exposed for the protocols HTTPS, Advanced Message Queueing Protocol (AMQP) and MQTT, where each device has endpoints that they receive and send messages on. 19

27 Communication between a device and the cloud can be achieved by keeping the desired and reported properties in the Device twin document updated. Figure 9 shows the endpoints used to interact with the Device twin. Figure 9: Interaction with the Device twin on Azure IoT hub AWS IoT - Communication using the Shadow service AWS IoT provides bi-directional communication between device and cloud through MQTT topics that are exposed for the protocols HTTPS and MQTT. Devices that are provisioned can subscribe and publish to topics. In the same way as the Device twin, the Thing shadow document can be used to achieve communication. The following MQTT topics on AWS IoT can be used to interact with a device s Thing shadow: $aws/ t h i n g s / device_id /shadow/ update $aws/ t h i n g s / device_id /shadow/ update /documents $aws/ t h i n g s / device_id /shadow/ get $aws/ t h i n g s / device_id /shadow/ d e l e t e Message routing The following functionality on Azure IoT hub and AWS IoT can be used to route messages between the platform and other services provided by Azure and AWS. Azure IoT hub - Routing rules Azure uses Routing rules for message routing. Routing rules allow for device messages, device lifecycle events and device twin change events to be routed to a custom endpoint. The custom endpoint is used for other services to be able to receive the messages and events. Other services could for instance be database storage, Azure functions etc. Routing rules also allows for filtering by applying an SQL statement to the routing rule. Only messages and events containing the fields queried in the statement will be routed. AWS IoT - Rules Message routing on AWS IoT is done through rules. Rules are applied to MQTT topics and by adding an action to a rule, messages and events can be routed to other AWS services such as Lambda functions, database storage, etc. Rules also allow for filtering in the same way as Azure IoT hub s Routing rules, by applying an SQL statement. 20

28 Regulation The following service on Azure IoT hub and AWS IoT can be used for regulation of CO 2 and temperature: Azure IoT hub - Azure functions Azure functions can be integrated with custom endpoints as mentioned in Message routing. When integrated the function will be triggered each time a new message or event is routed to the custom endpoint. When the function is invoked it has access to the message or event and can then process the data inside the message or event. The Azure function can be used for simple data processing and run-time decision making. AWS IoT - Lambda functions Lambda functions can be integrated with AWS IoT by adding a Lambda function as an action to a rule. In the Lambda function configuration, services can be added which the function will be able to access during run-time. The Lambda function can be used for simple data processing and run-time decision making. 21

29 5.5 Develop a system architecture The systems consist of four parts: end device, edge device, IoT cloud platform and a web page. Figure 10 shows an overview of the systems and how the different parts relate to each other End Device The end device is an Arduino Uno that interfaces the sensors and actuators. It controls the actuators based on commands that it receives from the edge device. The microcontroller also takes measurements from sensors, and sends it to the edge device when asked to Edge Device The edge device is a Raspberry Pi B+ running Linux. The edge device s purpose is to act as a gateway between the IoT cloud platform and the end device. This is a design choice taken from the paper IoT framework for Smart Buildings with Cloud Computing [17]. Since the end device s processor is not powerful enough to implement security and authorization, the edge device is necessary to offload these duties. Furthermore, in the context of smart buildings it is common that there exists more than one end device. The edge device allows for more end devices to be easily integrated into the system, which can be seen in figure 2. Figure 10: Architecture of the systems IoT Cloud Platform The IoT cloud platform is the central piece in the system. All the logic of the system is placed in the cloud and the IoT cloud platform is responsible for the regulation, message routing and integration of the different parts of the system Web page The purpose of the web page in the system is to visualize sensor data and actuator states, and to allow for remote controlling. The web page is the only part that the end user interacts with in the system and thus is also responsible for allowing the user to change settings such as the desired temperature range. 22

30 5.6 Analyze and design the system This section describes all the preparations before developing the system, that is making the sequence diagrams, flowcharts and design choices Hardware Figure 11: Circuit diagram The hardware consists of a high wattage resistor to generate heat, a peltier element for cooling and a fan for the ventilation. The temperature sensor chosen was an LM35 [27]. Figure 11 shows the circuit diagram. 23

31 5.6.2 End device Figure 12: Sequence diagram for the end device As seen in the sequence diagram in figure 12, the end device is designed to read the sensor values every other second. The sensor values are stored to allow for the edge device to read them. The CO 2 value is simulated on the end device to allow for better control and less volatile data. The end device listens for incoming commands from the edge device to change the actuator states. When the actuator states are changed, the end device responds whether it was a success or not Edge Device Figure 13: Sequence diagram for the edge device 24

32 The edge device is acting as a gateway and is communicating with both the IoT cloud platform and the end device. Whenever a desired property changes in the Device twin or Thing shadow, it invokes a callback on the edge device. The edge device communicates the new desired properties to the end device. The end device responds and the reported properties are updated and sent to the IoT cloud platform. The edge device is also reading the sensor values with an interval of 30 seconds. When the sensor values are read they are sent to the IoT cloud platform. By using two threads the edge device is able to communicate with both the IoT cloud platform and the end device at the same time, as seen in figure IoT cloud platform Figure 14: Sequence diagram for the IoT cloud platform To regulate the temperature and CO 2 the IoT cloud platform is designed to trigger an Azure or Lambda function. The function is invoked every time the edge device updates its Device twin or Thing shadow. When the function is invoked the Device twin/thing shadow document is passed as parameter to the function. The function can read the current sensor values from the document and compare them against the desired temperature range and CO 2 level, located in the desired properties. By comparing the values the function can regulate the temperature and CO 2 by responding back with commands to the edge device. 25

33 5.6.5 Web page Figure 15: Sequence diagram for monitoring on web page Figure 15 shows the sequence diagram for device monitoring on the web page. The web page shows the temperature, CO 2 level and actuator states from the reported properties. Figure 16: Sequence diagram for remote controlling Figure 16 shows the sequence diagram of remote controlling via the web page. When the user clicks a button or changes the desired temperature range, the desired state in the Device twin/thing shadow is updated. 5.7 Build the (prototype) system In this section the process of building the prototype is explained. The sections included first presents the parts that Azure IoT hub and AWS IoT have in common. Then the specifics regarding Azure IoT hub and AWS IoT is explained IoT cloud platform Common None. 26

34 Azure IoT hub Azure IoT hub provides a portal in which the user can create and manage the application. By providing a name for the application a new IoT Hub application is created. The next step was to provision the edge device. From the portal the edge device was provisioned by providing a device id and selecting the type of security. The security used is an autogenerated token which provides the authentication for the edge device. The permission required for the edge device to connect were preconfigured. The last step was to configure the message routing on Azure. This was done by creating a Routing rule that routes the edge device s device twin events to the endpoint messages/events. AWS IoT AWS IoT provides a portal in which the user can create and manage the application. In AWS IoT the first step was to provision the edge device by providing a device id. After the edge device was provisioned, the security certificates were generated and downloaded to be used on the edge device. Furthermore a policy was created to specify which resources the edge device should be able to access. The policy for the edge device is that it is allowed to connect to the AWS IoT MQTT broker as well as being able to access its Thing shadow. When the edge device was provisioned and prepared, the rule for triggering the AWS Lambda function was created. The rule listens on the following MQTT topic: $aws/ t h i n g s / edge_device /shadow/ update /documents When the edge device s Thing shadow is updated, the new Thing shadow document is published on this MQTT topic End device Common The end device software is independent of which IoT cloud platform is used. The software was implemented in C++ using the Arduino IDE according to the sequence diagram seen in figure 12. The communication with the edge device seen in the sequence diagram was implemented using the Arduino library Serial [28]. Accompanying the software is the hardware, which was constructed according to the circuit diagram seen in figure 11. In the circuit diagram the Arduino s digital pins are shown to which the transistors are connected. The transistors are switched on/off to control the actuators. The temperature sensor used is an LM35. The CO 2 however is simulated as mentioned in The simulation for CO 2 was implemented according to the following pseudo code: i f 200 ms s i n c e l a s t update co2 += persons co2_per_person_constant i f fan i s on co2 = fan_constant 27

35 Azure IoT hub None. AWS IoT None Edge device Common The software on the edge device was implemented using Python 3.5 and follows the sequence diagram seen in figure 13. The serial communication with the end device was implemented using the library PySerial [29] which allows for communication on the USB ports. The following JSON document is the Device Twin/Thing shadow representing the edge device on both Azure IoT hub and AWS IoT: { } " d e s i r e d " : { " v e n t i l a t i o n " : 0, " heating " : 0, " a i r C o n d i t i o n " : 0, " c l i m a t e C o n t r o l " : { "mintemp " : 20, "maxtemp" : 23, " maxco2level " : 600 } }, " reported " : { " temperature " : , " v e n t i l a t i o n " : 0, " a i r C o n d i t i o n " : 0, " heating " : 0, "CO2" : 234 } Azure IoT hub The SDK provided by Azure allows for the creation of an IoTHubClient object. This object was used to initialize the connection to the IoT cloud platform and later used to send data. In the initialization of the communication, a callback was attached to the IoTHubClient object which allows for receiving Device twin updates from the cloud. set_ device_ method_ callback ( device_ method_ callback, 0) When a Device twin update occurred the device_method_callback is invoked. 28

36 For the device to cloud communication the same instance of the object IoTHubClient is used. The IoTHubClient contains a method for updating the reported state in the Device twin. send_reported_state ( reported_state, l e n ( reported_state ), send_reported_state_callback, send_reported_state_context ) The reported_state is a JSON string which contains the new updated state and the send_reported_state_callback is a callback which is invoked when the update is received by the platform and contains information such as whether it was a success. AWS IoT The SDK provided by AWS allows for the creation of an object called AWSIoTMQTT- ShadowClient. Upon creation of the object, the security credentials are provided and then a callback method is configured for receiving delta callbacks. shadowregisterdeltacallback ( d e l t a _ c a l l b a c k ) The delta_callback method is invoked whenever an update to the Thing shadow creates a difference between the desired and reported properties of the Thing shadow. When a reported property needs to be updated, the method shadowupdate is called. shadowupdate ( payload, report_callback, 10) The shadowupdate method requires the parameter payload, which is a JSON string of the updated reported properties. The second parameter report_callback is a method which receives the status response, and the third is a timeout for the request specified in seconds Regulation Common The regulation was implemented using serverless functions for both Azure IoT hub and AWS IoT. The functions consist mainly of conditional statements meant to regulate the temperature and CO 2 levels. The conditional statements for the functions were implemented according to the sequence diagram, seen in figure 14. Azure IoT hub The first step was to create and attach the Azure function to the messages/events endpoint, on to which the Device twin change events are routed. The Azure function was implemented using Node.js and is invoked every time a new event is routed to the endpoint. The Device twin is passed into the function and the desired and reported properties are retrieved from it. 29

37 If the function needs to change the state of any actuator, a JSON string is created. The JSON string contains the new desired properties of the Device twin. The JSON string is then passed into to update function: twin. update ( updatedproperties, errorhandler ) ; Where updatedproperties is the JSON string containing the new desired properties and errorhandler is a function that is invoked if an error occurs when updating the Device twin. AWS IoT The Lambda function was added as an action to the rule created earlier and is invoked when a message is published on the edge device s Thing shadow update topic. The Lambda function was implemented using Python 2.7 and retrieves information on the edge device through its Thing shadow. When the Lambda function is invoked it has access to messages published on the MQTT topic, and can retrieve the desired and reported properties. If the function needs to update any desired properties a JSON string is created. The JSON string contains the new desired properties of the Thing shadow. To update the Thing shadow the following method is used: update_thing_shadow ( thingname= edge_ device, payload=payload_desired_properties ) Web page Common Displaying the sensor values and current status of the actuators is done by reading the reported properties each time a new message is received, as seen in the sequence diagram in figure 15. With each message a timestamp is provided by both Azure IoT hub and AWS IoT. The sensor values along with the timestamps are stored in arrays so that each pair of sensor value is associated with a timestamp. The sensor values are visualized using a line chart. The left Y-axis is temperature, the right Y-axis is CO 2 and the X-axis is time. Figure 17 shows the graph displayed on the web page. 30

38 Figure 17: Web page displaying run-time sensor values The current state of the actuators are visualized as buttons displaying green if on, red if off. When clicking a button, a request to update the edge device s Device twin/thing shadow is triggered. Figure 18 shows the actuator buttons and the desired temperature range input displayed on the web page. The functionality of the buttons were implemented according to the sequence diagram seen in figure 16. Figure 18: Web page remote controlling and actuator states Azure IoT hub Azure provides a platform SDK for Node.js, which was used for developing the web page for the Azure IoT hub application. The Node.js SDK provides capabilities to set up an AMQP connection over websocket to the application deployed on Azure IoT hub. The websocket connection was established towards the endpoint messages/events which was configured when setting up the Azure IoT hub application. Remote controlling on Azure IoT hub was implemented using the twin.update method. By passing a JSON string containing the new desired properties to the twin.update method, the actuators are remote controllable from the web page. 31

39 AWS IoT AWS provides an SDK for Node.js, which was utilized when building the web page for the AWS IoT application. The Node.js SDK for AWS IoT provides capability to set up a MQTT connection over websocket. In order to be able to establish the connection, security permissions were configured to give the web page full access to the AWS IoT application. The next step was to retrieve updates of the edge device s Thing shadow in the web page, which was done using the following method provided by the SDK: shadowclient. r e g i s t e r ( edge_device ) ; The register method registers interest in the Thing shadow of the device Id provided as parameter, in this case the edge device s device Id was provided. The last step was to listen to foreignstatechange events. ForeignStateChange events are emitted when different clients update or delete operations are accepted on the edge device s Thing shadow, i.e. the Lambda function or the edge device itself. Remote controlling was implemented similarly to how it was implemented on Azure. The platform SDK for AWS IoT also provides an update function which is used to pass the new desired properties. By passing the actuator and the new state of the actuator as the new desired properties. It was implemented with the following method: shadowclient. update ( edge_device, updatedproperties ) ; 5.8 Observe and evaluate the system Constructing test cases The test cases for the system were constructed to reflect the requirements specified in When the test cases were created they were reviewed to make sure that if they all pass, the system satisfies the requirements. The test cases can be found in Appendix B.1. 32

40 5.8.2 Testbed Figure 19: The testbed used to test the system. The final hardware for testing the system can be seen in figure 19. As can be seen in the picture buttons were added for testing purposes. The buttons allows for increasing and decreasing the CO2 values by increasing and decreasing the persons variable seen in the pseudo code in Furthermore the buttons also allow for manual control of the heating and cooling functionality. By being able to manually control the heating and cooling, aspects of the system that was dependent on what the temperature sensor reported could be tested in a controlled way. The final test results can be seen in Appendix C. As seen in the figure, the systems for Azure IoT hub and AWS IoT passed all the test cases. 33

41 5.9 Evaluation In this section the evaluation of the criteria is presented. For each criteria the basis is first presented for Azure IoT hub and AWS IoT, followed by a comparison between them and finally a short argumentation for the score that Azure IoT hub and AWS IoT received. The final results are summarized in table 3. Communication management Device management Data processing Development Device to cloud communication Cloud to device communication Message routing Communication protocols Device monitoring Remote controlling Device provisioning Run-time decision making Software Development Kit SDK languages Integration support for external application Azure IoT hub 5/5 5/5 4/5 5/5 5/5 5/5 5/5 4/5 2/5 5/5 5/5 AWS IoT 5/5 5/5 5/5 5/5 5/5 5/5 4/5 4/5 5/5 5/5 3/5 Table 3: Summary of the evaluated criteria 34

42 Azure IoT hub The communication from the device to the cloud was implemented using the Device twin. The implementation was easy since Azure provides good samples on the Python SDK Github page that show how to implement communication using the Device twin. The formatting of the message data uses the JSON format which is a well established format. Moreover the JSON string only needs to contain the reported properties that are intended to be changed, and not the complete document. The Azure documentation provides examples of JSON strings intended to update the reported properties. When updating the reported properties, the report method is implemented in a non-blocking way, such that the program does not get stuck when sending the reported state. Upon completion of the update, a callback is invoked that provides information whether the update was a success. Furthermore Azure IoT hub also allows to update a device s Device twin document directly through the Azure portal, which can be used for testing without having the device online. C1.1 Device to cloud communication AWS IoT Communication from device to cloud was implemented using the Shadow service. It was easy getting started with the Shadow service for communication as AWS provides good samples and documentation for the Shadow service. AWS IoT also uses JSON formatting, nonblocking updates, callbacks and allows for editing of the Thing shadow document through the portal. These implementations are done in the same way as Azure IoT hub. There are no mentionable differences between Azure IoT hub and AWS IoT regarding the message data formatting, sample code, documentation and implementation for the criteria device to cloud communication. Azure IoT hub scored 5/5 on the criteria device to cloud because the implementation process was easy with sufficient code samples, they use good standards and the Device twin was a good way of sending device to cloud messages. AWS IoT scored 5/5 on the criteria device to cloud for the same reasons as Azure IoT hub. Azure SDK samples: AWS SDK samples: 35

43 Azure IoT hub Cloud to device communication on Azure IoT hub was implemented in the Azure function by using the platform SDK. The platform SDK allows the Azure function to connect to the IoT Hub and retrieve the Device twin. The Device twin s desired properties can then be updated. Receiving the cloud to device messages on the edge device was implemented using the Device twin callbacks. The callbacks made the development easy due to all the threading work being abstracted away by the SDK. The messages received uses the data format JSON which made it easy to parse the messages. The Github page for the Python SDK also provide samples on how to implement the Device twin callbacks. Moreover the documentation on the Device twin also includes a guide on how to implement device twin callbacks. When receiving a Device twin update on Azure IoT hub it only contains the desired properties that has been updated and not the entire device twin document which makes parsing complicated. This can be counteracted by always sending all desired properties, whether they actually changed value or not. C1.2 Cloud to device communication AWS IoT Communication from Cloud to device was implemented using the desired properties on the Thing shadow for AWS IoT. AWS provides Boto which is the AWS SDK for Python. Boto is used to access other services on the AWS platform and allows for the Lambda function to update a Thing shadow s desired properties. It was simple to use and the documentation is extensive. To receive the Thing shadow updates, a callback is provided by the SDK that is called delta callback. The delta callback sends the difference between the desired and reported properties. However it is not possible to receive the all the desired properties, which can make parsing difficult. The implementation of the delta callback was easy due to extensive documentation, good samples and because the messages uses the message data format JSON. In Azure IoT hub it is possible to predetermine what will be received in the Device twin callback whilst in the delta callback it could be any or all desired properties. In comparison Azure IoT hub gives more freedom to choose what will be sent and AWS IoT limits the freedom by restricting the data sent to be as little as possible. Both implementations have good reasoning, however for the implemented scenario we favour Azure IoT hubs implementation. Azure IoT hub scored 5/5 on the criteria cloud to device communication because the implementation was easy, and the device twin callback worked well with the implemented scenario. AWS IoT scored 5/5 on the criteria cloud to device communication. AWS IoT s implementation with delta callbacks worked well and their design choice is justified. SDK documentation: Boto documentation: 36

44 Azure IoT hub Message routing on Azure IoT hub was implemented using a Routing rule. The routing rule specified that Device twin change events should be routed to the endpoint messages/events. The routing rule was easily set up because Azure s documentation provided step-by-step instructions that included pictures on how to set up a routing rule. However the service (Azure function, Blob storage, etc.) attached to the messages/events endpoint had to be configured on the endpoint, not in the Routing rule. A setting in the Routing rule that we consider useful is the SQL statement which allows for simple filtering of messages or events. C1.3 Message routing AWS IoT Message routing on AWS IoT was implemented using a rule. The rule specified an MQTT topic to listen to and an action. When creating a rule AWS IoT shows a list of possible actions to add to the rule. The actions are the different services that can be assigned to the rule, such as a Lambda function, database storage, etc. AWS provides extensive documentation on how to create and configure rules. AWS s rules also has support for simple filtering of messages or events using an SQL statement. Compared to Azure IoT hub s routing rules, AWS s rules were more intuitive to create and configure. AWS s integration with the service is done directly in the rule while Azure IoT hub s routing rules are more complex in the way that the endpoint needs to be configured and integrated with the Azure service separately. Furthermore the endpoint being routed to needs to be compatible with the Azure service. Azure IoT hub scored 4/5 on the criteria message routing. This is due to the integration with other Azure services being complex and difficult to understand. AWS IoT scored 5/5 on the criteria message routing. The reason for the score is that message routing on AWS IoT is easy to understand and provides an intuitive interface for routing between AWS IoT and other AWS services. Endpoints and Azure services: Rules for AWS IoT: Azure IoT hub Azure IoT hub provides native support for the following communication protocols: MQTT, MQTT over WebSockets, AMQP, AMQP over WebSockets, HTTPS. C1.4 Communication protocols AWS IoT AWS IoT provides native support for the following communication protocols: MQTT, HTTP, MQTT over WebSocket. Both Azure and AWS scored 5/5 on the criteria communication protocols. This is because both have support for the most common protocols with regard to IoT. Azure protocols: AWS protocols: 37

45 Azure IoT hub Device monitoring on Azure IoT hub was implemented on the web page. Device twin documents provides an easy way to monitor attributes of devices by adding the attributes to the Device twin document s reported properties. A Device twin document can be seen in section The Device twin service is well documented and Azure provides images similar to figure 9 which helps understanding the Device twin. Azure also provides comprehensive samples on how to observe changes to a device s Device twin document. The device twin document is maintained by Azure IoT hub and is therefore always available, even if the edge device is offline in which case it contains the latest known state of the edge device. The Device monitoring using the Device twin on Azure is good because it offers more functionality than ordinary message passing would. C2.1 Device monitoring AWS IoT Device monitoring on AWS IoT was implemented on the web page through the Shadow service. The shadow service provides a good way to implement device monitoring since the reported properties will show the current state of the device. Overall the Shadow service has the same functionality as Azure. AWS provides a sample on how to implement monitoring operations using the Shadow service which made implementation easy. They also have good documentation covering how the MQTT topics work and which one to listen to to observe changes to the Thing shadow document. The only mentionable difference between Azure IoT Hub and AWS IoT on the criteria device monitoring is that AWS s samples are not as comprehensive as Azure s. Azure IoT hub scored 5/5 on the criteria device monitoring because the Device twin allowed for easy implementation of device monitoring. They also provided good samples with extensive documentation. AWS IoT scored 5/5 for the same reasons as Azure IoT hub. Even though the samples were not as extensive, they provided enough information for the implemented system. Monitoring sample Azure: Monitoring samples AWS IoT: 38

46 Azure IoT hub Remote controlling was implemented on Azure IoT hub using the desired properties on the edge device s Device twin. The desired properties on the Device twin were used to request the edge device to turn on/off actuators. This implementation was good since the web page gets feedback on the remote controlling operation. The device will update its current state according to the desired properties and then report its new state. When the device reports its new state the web page receives a notification that the state has been updated. Azure provides sample code that shows how to update the desired properties for a device s Device twin document. Azure s documentation also provide information and images on how to update the Device twin document. C2.2 Remote controlling AWS IoT Remote controlling was implemented on AWS IoT using the desired properties on the edge device s Thing shadow. In the same way as Azure the desired properties were used to request the device to turn on/off the actuators. The implementation with AWS also received feedback when updating a state. AWS IoT provides sample code that shows how to update the desired properties in a device s Thing shadow document. AWS also provide extensive documentation on how to update the Thing shadow document. There are no mentionable differences between Azure IoT hub and AWS IoT how remote controlling is done on Azure IoT hub and AWS IoT. Azure IoT hub scored 5/5 on the criteria remote controlling because Device twin provides an easy way to achieve remote controlling through the desired properties. AWS IoT also scored 5/5 on the criteria remote controlling for the same reasons as Azure IoT hub. Azure Device twin sample: Azure Device twin get startediot-hub-python-twin-getstarted AWS Thing shadow sample: 39

47 Azure IoT hub The Azure portal provides an easy to use web user interface through which devices can be created. Upon creation of the device by entering a device ID, a connection string is automatically generated which is all information needed for the device to be able to connect to the platform. All in all the device provisioning process is done with only a few key presses when using the portal. In the documentation there are guides on how to set up devices through the portal. The guides use screenshots of the portal to show the steps needed to connect a device. Azure IoT hub also create the basic permissions for a device automatically when it is provisioned. Other ways of provisioning devices is using either the Command Line Interface (CLI) or the iothub-explorer tool. Azure provides step-by-step instructions to provision device through both the Azure CLI and iothub-explorer. C2.3 Device provisioning AWS IoT The AWS portal also provides an easy to use web user interface through which devices can be created. The provisioning process is done by entering a device ID which is followed by generating, downloading and placing the certificates on the device that wants to connect. The last step of the device provisioning on AWS IoT is to attach a policy to the device. AWS s documentation provides a step-bystep guide for the device provisioning process. Apart from the AWS IoT portal, AWS IoT also provides a CLI to provision new devices. Device provisioning is more complicated on AWS IoT compared to Azure IoT hub. This is because AWS requires security certificates to be generated, downloaded and put on the device while Azure only require the connection string. Moreover AWS requires a user defined policy while Azure s permissions are preconfigured for basic use. The security implications for either implementation are not considered. Azure IoT hub scored 5/5 on the criteria device provisioning. The reason is that the device provisioning process is very simple, and Azure provides step-by-step instructions how to do it. Furthermore it is good that Azure also provides more than one way (CLI and iothub-explorer) to interface the IoT hub. AWS IoT scored 4/5 on the criteria device provisioning. The reason it lost one star is because provisioning process included many steps such as generating, downloading and moving the certificates to the device that wants to connect. Other than that, it is well documented and easy to use. Azure device provisioning: Azure iothub explorer tool: Azure CLI documentation: AWS device provisioning: AWS CLI documentation: 40

48 C3.1 Run-time decision making Azure IoT hub The run-time decision making on Azure IoT hub was implemented using an Azure function. The Azure function was created through the Azure portal which was a simple process. However selecting the predefined event that the function was supposed to trigger on was complicated since there are multiple triggers to select from. Moreover the trigger also had to be compatible with the endpoint to which the messages was routed to. Although it was difficult to set up the function, Azure provided an online development environment to write the function which was easy to use. In the development environment the output logs from the function are located in the same view as the development environment which made it is easy to troubleshoot the function if any errors occurred. A screenshot of the development environment can be seen in figure 20. Furthermore Azure provides extensive documentation and code samples for Azure functions. AWS IoT The run-time decision making on AWS IoT was implemented using AWS Lambda. The AWS Lambda function was created in the AWS portal. During the creation the web user interface was helpful by visually showing the trigger of the Lambda function, but also which services it is allowed to interact with. The configuration was very intuitive, as seen in figure 21. The web user interface also provided an online development environment for writing the Lambda function. The online development environment did not contain the output logs of the Lambda function. Instead they were displayed in another service called Cloudwatch. This made development and troubleshooting in AWS a more tedious process due to being required to use another service to troubleshoot the function. The documentation for the Lambda function was extensive and provided code examples for interactions between different AWS services. Figure 20: Azure function and output log Figure 21: Configuration of the Lambda function 41

49 Troubleshooting of the Azure function was easier compared to the Lambda function due to the output logs being located in the development environment instead of in another service. However AWS provided a more intuitive interface when creating and configuring the Lambda function, compared to when creating the Azure function. Azure scored 4/5 on the criteria run-time decision making. The reason for the score is that Azure provides an online development environment and displays the output logs in the same view which make it easy to develop. Azure lost a star due to the configuration of the trigger being complicated. AWS IoT scored 4/5 on the criteria run-time decision making. The reason for the score is that AWS provides a good web user interface and an online development environment which make it easy to develop the Lambda functions. However, AWS IoT lost one star since the development and troubleshooting slowed down because of dependency of the Cloudwatch service. Azure function samples: AWS Lambda: Boto documentation: 42

50 C4.1 Software Development Kit (SDK) Azure IoT hub The SDKs used were the Node.js SDK and the Python SDK. Both the Node.js SDK and the Python SDK have a sample folder on their Github page. The code in the samples were not commented properly, meaning there were no comments on methods describing them and their parameters and there were no inline comments where some extra explanation could be necessary. Although the code was not documented, Azure provided API reference for the Node.js SDK, however none for the Python SDK. On the Github page for the Python SDK there is a roadmap describing what it supports, what it does not support and their plans for adding support. The SDKs for other languages, excluding Node.js, does also have roadmaps and do not have support for all features. The Node.js SDK does not have a roadmap since it supports all features. The github pages for the SDKs all provide the installation instructions for the SDK. The Python SDK was easy to install using Python Package Index (PyPI) and the Node.js SDK using Node Package Manager (NPM). AWS IoT The SDKs used were the Node.js and Python SDK. The SDKs have an extensive documentation with an API reference and samples provided on their Github page. The sample code for the Python SDK could have used some more comments since it is usually aimed towards people using it for the first time. The Node.js samples had plenty of comments describing the samples. In the same way as Azure, AWS s Github pages for the SDKs also include installation instructions for the SDKs. The Python SDK was installed using PyPI and the Node.js using NPM. AWS SDKs are extensively documented and provides well documented samples in comparison to Azure s SDKs. As mentioned, Azure s Python SDK does not have an API reference and the samples are poorly commented, which makes it difficult to understand. AWS provides an API reference for all their SDKs and the samples included in the SDKs are properly commented and documented. Azure IoT hub scored 2/5 on the criteria Software Development Kit (SDK). This is because their SDK documentation is poor and their samples are not properly commented. The reason they got two stars is because they provide samples which are useful. AWS IoT scored 5/5 on the criteria Software Development Kit (SDK) because they have extensive documentation for all their different SDKs. They also provide samples for each SDK. Azure Python SDK: Azure Node.js SDK: AWS Node.js SDK: AWS Python SDK: 43

51 C4.2 SDK languages Azure IoT hub Azure IoT hub provides platform SDKs for the following programming languages: C#, Java, JavaScript, Python and C. AWS IoT AWS IoT provides platform SDKs for the following programming languages: Java, JavaScript, Python, C++ and C. Both Azure IoT hub and AWS IoT scored 5/5 on the criteria SDK languages. The reasons for the score is that they provide SDKs available in a wide range of programming languages. Azure SDKs: AWS SDKs: C4.3 Integration support for external application Azure IoT hub The external application integrated with Azure IoT hub is the web page. The web page was built using the Node.js SDK that Azure provides. Connecting the web page to the Azure IoT Hub application was a simple process when following the sample that Azure provides. The sample provides the essential security steps that needs to be configured to establish the connection, but also how to receive and parse messages from an Azure IoT hub application which made the integration process simple. However Azure s documentation could be better, there are few inline comments explaining the code in the sample and the Azure docs describes how to get the sample running but not how the sample works. AWS IoT The external application integrated with AWS IoT is the web page. The web page for the AWS IoT application was developed using the Node.js that AWS IoT provides. AWS provides extensive samples for external web page applications, however there were a lot of security related issues when connecting the web page to AWS IoT. The samples did not include any information which security permissions that had to be configured when connecting the web page to AWS IoT. Furthermore, AWS s documentation did not provide any instructions how to configure the required security permissions either. Connecting the web page to AWS IoT was more complicated compared to Azure. There were more security permissions on AWS IoT that needed to be configured. Furthermore, the documentation and samples that AWS provides for external browser applications does not include any instructions on how to configure them. Azure scored 5/5 on the criteria integration support for external application. The reason for the score is that Azure provides a Node.js SDK which is well documented and an extensive web page example. AWS IoT scored 3/5 on the criteria integration support for external application. The reason for the score is that AWS provides also provide a well documented Node.js SDK and extensive samples. However AWS IoT lost two stars because the documentation and the browser samples did not provide any information at all regarding the necessary security permissions needed to connect the web page to the AWS IoT application. Azure browser sample: AWS browser samples: 44

52 6 Discussion 6.1 Related work The paper that is closest related to the work done in this thesis is Selecting the right IoT cloud platform [4]. The results in the paper is based upon a literature survey whilst this thesis is a subjective evaluation based on experience from the development process. Furthermore, the paper provides an overview whether certain functionality is supported or not by the evaluated platforms. In this thesis we selected two platforms and implemented the same functionality on both platforms. Through the implementation process we gathered experience of platform documentation, how well platform functionality actually work, ease of use and more. This experience was then used for a more in depth comparison of the two platforms. The paper What Does I(o)T Cost? [18] is also closely related to the work done in this thesis. Their contribution is also an evaluation performed in a similar manner as ours, except that their work only evaluates the cost and pricing plans of different platforms. The two papers Design and Implementation of Intelligent HVAC System Based on IoT and Big data Platform [19] and IoT framework for Smart Buildings with Cloud Computing [17] provided inspiration and design choices for the prototype developed. The work in [19] provided insight on how remote controlling and monitoring solutions can be used in an HVAC system and is the inspiration for the remote controlling and monitoring implemented in our prototype. The paper [17] presented a suitable architecture for smart buildings which is utilized in the prototype. The architecture worked well since it made it easy to connect the smart building to the IoT cloud platform and it allows for additional end devices to be added into the system with ease. The paper Fast-paced development of a smart campus IoT platform [16] presented some potential issues when using IoT cloud platforms such as high billing costs, service disruption and connectivity. When developing the HVAC system prototype we faced none of these issues, we believe that the reason for this is that they implemented their system on a larger scale. 6.2 Comparative study methodology The comparative study used was a derived from two different papers, Cloud storage providers: A Comparison Review and Evaluation [21] and What Does I(o)T Cost? [18]. Together they created a method that provided a systematical approach to an evaluation based on implementation experience. The step construct a scenario from [18] gave something definite to base the criteria on. Since the criteria are based on the scenario, it was assured that the criteria could be evaluated when the scenario had been implemented. The platforms could then be selected after the criteria had been defined to prevent biased opinions on which criteria to evaluate. With the platforms selected the scenario could be implemented and by doing so the experience needed to evaluate the platforms was collected. 45

53 This combined methodology worked well because the two comparative methods complemented each other. The systematic approach for a comparative study was achieved by following [21] and the scenario-based comparative study was achieved by including the steps from [18]. 6.3 Analysis of result The aim of this thesis was to evaluate IoT cloud platforms in the context of smart buildings. The result shows that both Azure and AWS has scored high on most criteria which could have been expected since they are the market leaders. Furthermore, the results also shows that much of the functionality that Azure IoT Hub and AWS IoT provide are implemented in a similar manner, e.g. Azure s Routing rules and AWS s Rules. The most notable differences between the two platforms are documentation, ease of use and comprehensibility. But there are also differences such as how the cloud to device communication using Device twin/shadow service is implemented, which is discussed in the evaluation. The differences between the two platforms can be advantageous or disadvantageous depending on the system to be implemented. For example, for systems that make use of other CSP services such as storage or data processing we believe that AWS IoT is a better choice. AWS IoT provides a more intuitive way of routing messages between the different services that AWS offers. However for a system that is supposed to be integrated with external applications such as a web page, we believe that Azure IoT hub is a more suitable choice. These recommendations are based on the evaluation of the functionality that was used when implementing the prototype on Azure IoT Hub and AWS IoT. The prototype developed uses the Device twin/thing shadow for all communication between the different components, which is a design choice based upon the literature study. Design choices like this one have affected the outcome of the evaluation, not necessarily in a bad way but other design choices would probably have resulted in a different outcome. Another aspect to consider with our results are the criteria. The criteria used in this thesis only evaluate parts of the platform and not all existing functionality. Criteria that could have been used for a more comprehensive evaluation are bulk provisioning, machine learning, etc. These criteria were not considered because they were not covered by the requirements constructed in

54 7 Conclusions The IoT cloud platforms Azure IoT hub and AWS IoT are both suitable for usage in a smart building application. Although it is still difficult choosing an IoT cloud platform for smart buildings, we have shown that for an HVAC system there are two candidates that can be used. The research questions stated in are as follows: RQ 1 How can a heating, ventilation and air conditioning system be implemented using an IoT cloud platform? RQ 2 How well do existing IoT cloud platforms support development of a heating, ventilation and air conditioning system? In section an implementation of an HVAC system using an IoT cloud platform has been described which answers the first research question, RQ 1. The second question, RQ 2, has been answered in section 5.9 by performing an evaluation of Azure IoT hub and AWS IoT using the experience gained from development process of the prototype. 7.1 Contribution As stated in 1.2 there appears to exist few articles that evaluate IoT cloud platforms. This thesis complement the existing articles on IoT cloud platforms by describing how a smart building appliance using an IoT cloud platform can be developed. This thesis also complement existing articles with an evaluation and comparison on the leading cloud providers IoT cloud platforms. The results of this thesis can be used by Sigma Lundinova when developing cloud based IoT solutions. 7.2 Future work This thesis covered a set of criteria and a set of functionality offered by Azure IoT Hub and AWS IoT. For future work there are many more criteria that can be evaluated, such as bulk provisioning, privacy, security, etc. CSPs also offers more functionality and services that was not evaluated, such as machine learning, storage, analytics, etc. Moreover, as can be seen in section 5.3 there are many IoT cloud platforms that can be evaluated. 47

55 References [1] Mauro A. A. da Cruz et. al. A Reference Model for Internet of Things Middleware. In: IEEE Internet of Things Journal (2018), pp [2] Postscapes. Postscapes. url: (visited on 27/02/2018). [3] T. Pflanzner, A. Kertesz. A survey of IoT cloud providers. In: Information and Communication Technology, Electronics and Microelectronics (MIPRO), th International Convention on (2016), pp [4] Pankaj Ganguly. Selecting the right IoT Cloud Platform. In: 2016 International Conference on Internet of Things and Applications (IOTA) (2016), pp [5] Partha Pratim Ray. A survey of IoT cloud platforms. In: Future Computing and Informatics Journal (2016), pp [6] Ahmed Banafa. How the Internet of Things is shaping modern business url: (visited on 29/03/2018). [7] Claude Baudoin et al. Practical Guide to Platform-as-a-Service. Tech. rep. Version 1.0. Cloud standards customer council, url: org/deliverables/cscc-practical-guide-to-paas.pdf. [8] Calum McClelland. What is an IoT platform? url: com/what-is-an-iot-platform/ (visited on 05/03/2018). [9] Kristopher Sandoval. What is the Difference Between an API and an SDK? url: (visited on 25/04/2018). [10] Michael Yuan. Getting to know MQTT url: library/iot-mqtt-why-good-for-iot/index.html (visited on 12/04/2018). [11] Mike Roberts. Serverless Architectures url: articles/serverless.html (visited on 05/03/2018). [12] Rhyan Solomon. What s All the FaaS About? url: blog/what-is-functions-as-a-service-faas/ (visited on 09/03/2018). [13] A.H. Buckman et al. What is a Smart Building? In: Smart and Sustainable Built Environment 3 (2014), pp [14] Nicola King. SMART HOME - A DEFINITION url: org. uk / _assets / Resources / Housing / Housing _ advice / Smart _ Home_ - _A _ definition_september_2003.pdf (visited on 07/02/2018). [15] Mike Aguilar. HVAC Design Principles and Major System Components url: design- principlesand-major-system-components-for-hvac/ (visited on 28/03/2018). [16] Armin Haghi et al. Fast-paced Development of a Smart Campus IoT Platform. In: Global Internet of Things Summit (GIoTS), 2017 (2017), pp [17] Enrique Carillo et al. IoT framework for Smart Buildings with Cloud Computing. In: Smart Cities Conference (ISC2), 2015 IEEE First International (2015), pp [18] Edua Eszter Kalmar, Attila Kertesz. What Does I(o)T Cost. In: International Conference on Performance Engineering Companion (2017), pp [19] Tacklim Lee et al. Design and Implementation of Intelligent HVAC System Based on IoT and Bigdata Platform. In: 2017 IEEE International Conference on Consumer Electronics (ICCE) (2017), pp

56 [20] Jay F. Nunamaker, Minder Chen. Systems Development in Information Systems Research. In: System Sciences, 1990., Proceedings of the Twenty-Third Annual Hawaii International Conference on 3 (1990), pp [21] Xhemal Zenuni et al. Cloud storage providers: a comparison review and evaluation. In: CompSysTech 14 Proceedings of the 15th International Conference on Computer Systems and Technologies (2014), pp [22] Engineering ToolBox. Carbon Dioxide Concentration - Comfort Levels url: https : / / www. engineeringtoolbox. com / co2 - comfort - level - d _ html (visited on 10/04/2018). [23] Microsoft. IoT Hub url: iot-hub/ (visited on 16/04/2018). [24] Amazon. AWS IoT url: (visited on 16/04/2018). [25] Synergy Research Group. The Leading Cloud Providers Continue to Run Away with the Market url: (visited on 02/04/2018). [26] Ian Skerrett. The Reality of Digital Twins for IoT url: (visited on 09/04/2018). [27] Texas Instruments. LM35 Precision Centigrade Temperature Sensors url: (visited on 13/04/2018). [28] Arduino. Serial url: functions/communication/serial/ (visited on 13/04/2018). [29] PySerial. PySerial url: (visited on 16/04/2018). 49

57 Appendices Appendix A Afero AirVantage M2M Cloud Amazon AWS IoT Ardic ARCLOUD Arduino Cloud Arrayent AT&T M2X Augury Autodesk SeeControl Platform Axonize Ayla IoT Platform Bit Stew MIx Core Bosch IoT Cloud Bright Wolf Strandz Buddy Built.io Flow C2M C3 IoT Platform Carriots Cisco Jasper Control Center Clearblade Novi IoT Platform Cloudplugs Cloudthing.io Cumulocity data.sparkfun DeviceHive DevicePilot ElasticM2M EMnifiy Eurotech Device Cloud EVRYTHNG Platform Exosite flowthings FogHorn Cloud ForgeRock Fybr Platform GE Predix Golgi Google Cloud IoT Greenwave Systems AXON GroveStreams Helium Hitachi Lumada IBM Watson IoT Platform Infiswift Initial State Intel R IoT Platform iobeam IoTIFY Kaa Keen Home Kii Lelylan Lhings Linkafy Litmus Loop Lord MicroStrain SensorCloud LOSANT MachineShop Meshify Microsoft Azure IoT Microsoft Soliar mnubu Mode Mongoose IoT Platform mydevices platform Nabto NewAer nio Novelti Numerex nxfast Octoblu Open Picus Iomote Opensensors Oracle Internet of Things Cloud Particle Cloud Pega 7 Platform Proximetry PTC ThingWorx PubNub Reekoh Resin RhythmOS Samsung ARTIK Cloud SAP HANA Cloud Platform scriptr.io Seluxit metaflow IoT Platform SenseIoT SensorUp Siemens Mindsphere SiteWhere SkkyHub Splunk Stream IoT-X TM Platform SUPLA-CLOUD Supra IoT Telit IoT Portal Temboo TempoIQ thethings.io Thing+ ThingSpeak Tiboo AggreGate Tulip Ubidots Undercontrol UnificationEngine Uptake Valarm Vertical M2M: CommonSense Waylay Wia Wind River Helix Cloud Xaptum Xively Zapier Zebra Zatar Zetta Zonoff Table A.1: All IoT cloud platforms found on Postscapes [2] on February 27th,

58 Appendix B Figure B.1: Test cases used to validate that the system fulfills the requirements. 51

59 Appendix C Figure C.1: Test results for the systems developed with Azure IoT Hub and AWS IoT 52