class: title, smokescreen, shelf, bottom, no-footer background-image: url(images/pub-sub.png) # 181U Spring 2020 ### IoT Protocols (part 1) --- layout: true .footer[ - Geoffrey Brown, 2020 - 181U ] <style> h1 { border-bottom: 8px solid rgb(32,67,143); border-radius: 2px; width: 90%; } .smokescreen h1 { border-bottom: none; } .small.remark-slide-content.compact {font-size:1.2rem} .smaller.remark-slide-content.compact {font-size:1.1rem} .small-code.remark-slide-content.compact code {font-size:1.0rem} .very-small-code.remark-slide-content.compact code {font-size:0.9rem} .line-numbers{ /* Set "line-numbers-counter" to 0 */ counter-reset: line-numbers-counter; } .line-numbers .remark-code-line::before { /* Increment "line-numbers-counter" by 1 */ counter-increment: line-numbers-counter; content: counter(line-numbers-counter); text-align: right; width: 20px; border-right: 1px solid #aaa; display: inline-block; margin-right: 10px; padding: 0 5px; } </style> --- class: compact # Agenda * MQTT * Publish-Subscribe Model * MQTT Example * Protocol * Brokers * JSON * COAP --- class: compact # The problem  * "Traditional" applications based upon reliable byte streams between endpoints * Not a good fit for many-to many communication (e.g. Twitter, Snapchat, etc) Image from Wikimedia commons --- class: compact # Ways to exchange messages   --- class:compact # Request/Response * **REST** : Representational State Transfer * Widely used; often based upon HTTP * Later we'll discuss a lighter approach **CoAP** for IoT   --- class: compact # Why isn't HTTP enough ? * The HTTP standard revolutionized how *people* consume data * A single simple model: Send a request, read the response * Available via any tablet, laptop, phone, etc. * The Internet of Things has fundamentally different challenges * HTTP remains a good way to request data from a known source * But we need an event-oriented paradigm: * Emitting information: *one to many* * Listening for events *whenever they happen* * Distrubute huge volumes of small data packets * *Push* information over *unreliable networks* --- class: compact # Pub/Sub Approach  * Publish/Subscribe * producer/consumer * Various protocols * MQTT, AMQP, XMPP * Pub/sub separates **publisher** who sends messages about specific **topics** from **subscriber** who receives messages about specific **topics** * The protocol depends upon a **broker** known by both **publishers** and **subscribers** which filters incoming messages and distributes them. --- class: compact # MQTT > MQTT is a machine-to-machine (M2M)/Internet of Things connectivity protocol. > It was designed as an extremely lightweight publish/subscribe messaging transport. It is useful for connections with remote > locations where a small code footprint is required and/or network bandwidth is at a premium. (http://mqtt.org)   Figure:http://istsos.org/en/trunk/doc/ws_mqtt.html --- class: compact # The Realm of MQTT   --- class: compact # Background/Goals of MQTT * Make it simple to connect physical world to traditional IT world * Expect and cater to frequent network disruption -- built for *low bandwidth*, *high-latency*, *unreliable*, *high-cost* networks (per byte) * Expect client applications may have very limited resources (8-bit, small memory) * Provide loose coupling to support dynamic system environments * Provide multiple deterministic message delivery quialities * Capable of supporting very large numbers of devices * Simple for developers to implement protocol --- class: compact,small # MQTT works on top of ... * Mainly TCP * There is a closely related protocol MQTT-SN (MQTT for sensor networks) that runs on UDP, but COAP is a UDP protocol that is used more which we will look at. * Websockets can also be used. * Enables embedding in web pages.   * Both TCP and websockets can operate on top of "transport layer security" (TLS) and SSL. --- class: compact # MQTT Example  * Clients publish to a topic based upon their location **sensors/temperature/{country}/{city}/{street}** * Sensor on Baker Street in London would publish to **sensors/temperature/uk/london/baker_street** with a message containing the current temperature * A subscriber might collect all temperature data in london by subscribing to **sensors/temperature/uk/london/#** * Note the use of '/' to build topic hierarchy. * **#** is a wildcard matching any number of levels, **+** is a wildcard matching a single level. https://zoetrope.io/tech-blog/brief-practical-introduction-mqtt-protocol-and-its-application-iot/ --- class: compact # MQTT Protocol  MQTT is connection based. Publishers and Subscribers connect to a broker (using TCP protocol). * In this example, a publisher (client B) regularly updates the current temperature. A subscriber (client A) connects and receives the temperature. * The example illustrates the use of topics -- **temperature/roof** and an optional feature -- retained messages. Simon A. Eugster [<a href="https://creativecommons.org/licenses/by-sa/4.0">CC BY-SA 4.0</a>], <a href="https://commons.wikimedia.org/wiki/File:MQTT_protocol_example_without_QoS.svg">via Wikimedia Commons</a> --- class:compact # Publish/Subscribe Protocol Sequence   --- class: compact # Topics * MQTT Topics are structured in a hierarchy similar to folders and files in a files system (and URLs) using forward slash (/) as a delimiter  * Allow creation of user friendly and self-sescriptive naming structure * Topic names are: * Case sensitive * use UTF-8 strings * Must have at least one character  * $SYS is the only **default** or **standard** topic --- class: compact,small # Topics -- wildcards * Topic subscriptions can have wildcards. These enable nodes to subscribe to groups of topics (that may not yet exist !) * '+' matches anything at a given tree level * '#' matches a while sub-tree * Examples: * Subscribing to topic **house/\#** covers: * house/room1/main-light * house/room1/alarm * house/garage/main-light * house/main-door * Subscribing to **house/+/main-light** covers: * house/room1/main-light * house/room2/main-light * house/garage/main-light * but doesn't cover * house/room1/alarm --- class:compact # Quality of Service (QoS) * Messages are published with a **Quality of Service** (QoS) * QoS 0 ("at most once") -- messages are fire-and-forget. * For example, a doorbell notification might only matter if delivered immediately * QoS 1 ("at least once") -- stored by the broker until clients acknowledge delivery * May be delivered with duplicates * QoS 2 ("exactly once") -- messages have a second acknowledgement round-trip to ensure that **non-idempotent** messages are not duplicated. - --- class:compact #MQTT Publish/Subscribe Messaging * A subscription can be durable or non-durable * Durable - Once a subscription is in place a broker will forward matching messages to a subscriber: * Immediately if the subscriber is connects * Store them until the subscriber connects * Non-durable - The subscription lifetime is the period the subscriber is connected to the broker * A publication may be retained * A publisher can mark a publication as retained - The broker remembers the last known good message for a topic - When a subscriber connects, the broker gives the retained message for the topic. --- class: compact # Use on Constrained Devices * Designed for constrained networks: * Protocol compressed into bit-wise headers and variable length fields * Smallest possible packet is 2 bytes * Asynchronous bidirectional **push** delivery of messages to applications * Supports always-connected and sometimes-connected models * Provides session awareness - "last will and testament" enable applications to know when a client goes offline abnormally * Typically uses TCP * Provides multiple deterministic message deliveries quality of service (QOS) * 0 : message delivered at most once * 1 : messsage will be delivered but may be duplicated * 2 : once and only once delivery --- class: compact # Constrained Device * Designed for constrained devices * Suited for applications/devices with limited resources * 8-bit controllers * battery power * Multiple MQTT client implementations in different langauges * Tiny MQTT client -- 30KB C library, 64KB Java library --- class: compact # Use-Cases: Facebook Messenger   --- class: compact # Use-Cases: Home pacemaker monitor  --- class: compact # Use-Cases: Physical to Enterprise   --- class: compact # Areas where MQTT has been used   --- class:compact # MQTT Brokers * Widely Used * http://mosquitto.org (we'll use this in lab) * man page: man page: https://mosquitto.org/man/mosquitto-8.html * http://www.hivemq.com   --- class:compact,small # Cloud Based MQTT Brokers * https://www.cloudmqtt.com (has a free tier) * https://flespi.com/mqtt-broker (has a free tier) * AWS IoT Core : https://aws.amazon.com/iot-core/ * AZURE IoT Hub : https://azure.microsoft.com/en-us/services/iot-hub/ * Google MQTT-bridge : https://cloud.google.com/iot/docs/how-tos/mqtt-bridge   --- class:compact # flespi.com  --- class:compact,small # Open Brokers (sandboxes) * TCP Based * https://iot.eclipse.org/getting-started/#sandboxes * Hostname: iot.eclipse.org * http://test.mosquitto.org/ * Hostname: test.mosquitto.org * https://www.hivemq.com/public-mqtt-broker/ * Hostname: broker.hivemq.com * http://www.mqtt-dashboard.com/ * Ports: * standard: 1883 * encrypted: 8883 (TLS v1.2, v1.1 or v1.0 with x509 certificates) * Websockets based: * broker.mqttdashboard.com * test.mosquitto.org * broker.hivemq.com --- # Clients for testing * Mosquitto broker comes with command-line tools for pub/sub * `mosquitto_sub -h HOSTNAME -t TOPIC [-v]` * `mosquitto_pub -h HOSTNAME -t TOPIC -m MESSAGE` * Paho C client library also includes * `paho_c_sub` * `paho_c_pub` --- class: compact,small # MQTT Protocol Example   An example using the tools **mosquitto-sub** and **mosquitto-pub** to send/receive topics from a local broker (localhost). * -h broker name * -t topic string (on subscriber side this can include wildcards) * -m message string (publish only) * -v (verbose subscription) --- class: compact # MQTT Clients : IOS   --- class: compact # MQTT Clients : Android   --- class: compact # MQTT Client: Node-Red   --- class: compact,very-small-code,hljs-tomorrow-night-eighties,col-2 # MQTT Client Library : Mosquitto (C only) ```c void on_message(struct mosquitto *m, void *user_data, const struct mosquitto_message *msg) { fprintf(stderr, "lights at %s are %s\n", msg->topic, (char*)msg->payload); } int main(int argc, char **argv) { struct mosquitto *client; int ret; mosquitto_lib_init(); client = mosquitto_new("client-id", true, NULL); if (!client) err(1, "mosquitto failed"); ret = mosquitto_connect(client, "127.0.0.1", 1883, 60); if (ret != MOSQ_ERR_SUCCESS) err(1, "mosquitto failed"); ret = mosquitto_subscribe(client, NULL, "switches/+/status", 0); if (ret != MOSQ_ERR_SUCCESS) err(1, "mosquitto failed"); mosquitto_message_callback_set(client, on_message); while (MOSQ_ERR_SUCCESS == mosquitto_loop(client, -1, 1)); return 0; } ``` --- class: compact # MQTT Client Library : Paho * Open source MQTT client library https://www.eclipse.org/paho/   --- class: compact,very-small-code,hljs-tomorrow-night-eighties,col-2 # Paho Python Example ```Python import paho.mqtt.client as mqtt # The callback for when the client receives a CONNACK # response from the server. def on_connect(client, userdata, flags, rc): print("Connected with result code "+str(rc)) # Subscribing in on_connect() means that if we # lose the connection and # reconnect then subscriptions will be renewed. client.subscribe("$SYS/#") # The callback for when a PUBLISH message is # received from the server. def on_message(client, userdata, msg): print(msg.topic+" "+str(msg.payload)) client = mqtt.Client() client.on_connect = on_connect client.on_message = on_message client.connect("localhost", 1883, 60) # Blocking call that processes network traffic, # dispatches callbacks and handles reconnecting. # Other loop*() functions are available that give a # threaded interface and a manual interface. client.loop_forever() ``` --- class: compact # Comments on Paho Python * Note the use of "callbacks" -- this is a common structuring mechanism for servers * Also supported in Paho asynchronous C library * There are many more options * SSL/TLS support for security * Protocol version (we're using 3.11 in this class) * Support for error logging * Threaded architecture for main loop -- enables main loop to do other things https://www.eclipse.org/paho/clients/python/docs/ --- class: compact # MQTT Protocol Specification ([link](http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.pdf))   --- class: compact # MQTT Protocol (from specification) * Data representation * Bits in a byte numbered 7 down to 0. 7 is most significant * Integers 16-bit, "big endian" * UTF-8 encoded strings with example (A𪛔) (41 F0 AA 9B 94)    --- class: compact # MQTT Malformed Data   --- class: compact,small # MQTT Protocol (control packet)     | Name | Value | Direction | Description | | ---- | ----- | ---------- | ---------- | | CONNECT | 1 | Client to Server | Connect | | CONNACT | 2 | Server to Client | Ack Conn| | PUBLISH | 3 | Client to Server | Publish message | | PUBACK | 4 | Both | Ack Pub | | ... | ... | ... | ... | | SUBSCRIBE | 8 | Client to Server | Subscribe | | SUBACK | 9 | Server to Client | Ack Sub | | ... | ... | ... | ... | | DISCONNECT | 14 | Client to Server | Disconnect | --- class: compact # MQTT Protocol Publish Message   * QoS, retain encoded in flags bits * Remaining length -- length of a variable (topic) header + payload * Topic -- utf-8 encoded string (length + utf-8), 16-bit packet identifier * Payload -- uninterpreted bytes. * Topic string limited to 65536 bytes (16-bit length) * Payload is limited to 268,535,456 bytes --- class: compact # MQTT Publish Packet Variable Header (Topic)   --- class:compact,small-code,hljs-tomorrow-night-eighties, # MQTT Remaining Length Field Remaining length field is variable length encoded - bit 7 used a "continuation" bit  ```c do { encodedByte = X MOD 128 X = X DIV 128 // if there are more data to encode, //set the top bit of this byte if ( X > 0 ) { encodedByte = encodedByte OR 128 } 'output' encodedByte } while ( X > 0 ) ``` --- class: compact # MQTT Operational Behavior * Storing State -- client and server must store session state for duration of session * QoS 1 (At least once delivery) -- receiver must respond with PUBACK packet, packet identifier ignored * QoS 2 (Exactly once) -- receiver responds and treats new packets with same identifier as duplicates --- class: compact # MQTT Topics (partial)   --- class: compact # Designing Topics The MQTT protocol leaves important issues undefined * Topics -- The protocol defines topics as a hierarchy of names, but gives no guidance about choosing topics. For a small network this might not matter, but it doesn't scale well * Message Bodies -- these are just byte sequences with *no* syntax or semantics. * syntax -- json ?, protobuf ? + schemas * semantics -- left as exercise to user https://d1.awsstatic.com/whitepapers/Designing_MQTT_Topics_for_AWS_IoT_Core.pdf --- class: compact # MQTT Message Patterns : Point-to-Point   * Basic building block -- two things use a single topic as "point-to-point" channel --- class: compact # MQTT Message Patterns: One to many with point-to-point messages Point-to-point messages be used a one-to-many by using a (slightly) different topic for each endpoint   --- class: compact # MQTT Message Patterns: broadcast   --- class: compact # MQTT Message Patterns: Many to one   --- class: compact # MQTT Topic Best Practices * Use only lowercase letters, numbers, dashes: MQTT topics are case sensitive -- mixing cases can lead to issues * Topic level structures should follow general to specific pattern * Platform/building/location/thing   * Include relevant routing information in MQTT topic * Prefix topics to distinguish data and control --- class: compact # MQTT Topics for data Data * prefix (dt) * IoT application * context (one or more layers) * name of thing transmitting data * type of data dt/<application>/<context>/<thing-name>/<dt-type> --- class: compact # MQTT Topics for Control Control * prefix (cmd) * IoT application * context * destination thing * request type; request parameters should go in message cmd/<application>/<context>/<destination-id>/<req-type> Typically, there will be a corresponding response cmd/<application>/<context>/<destination-id>/<res-type> --- class: compact # MQTT Documentation * MQTT Community Wiki * https://github.com/mqtt/mqtt.github.io/wiki * MQTT Tutorial * https://www.hivemq.com/mqtt-essentials/ * MQTT 3.1.1 Specification * http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/mqtt-v3.1.1.html * MQTT Standards body * https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=mqtt --- class: compact # Summary * Cover Photo: Mentor Automotive * Acknowledgements -- slide content and images drawn from: * https://github.com/moleike/mqtt-slides * https://www.oasis-open.org/committees/download.php/49205/MQTT-OASIS-Webinar.pdf * http://wireless.ictp.it/school_2019/slides/MQTT_v2.pdf Pietro Manzoni Intro * https://d1.awsstatic.com/whitepapers/Designing_MQTT_Topics_for_AWS_IoT_Core.pdf