How vital is MQTT Keep Alive & Client Takeover?

How vital is MQTT Keep Alive & Client Takeover?

Do you know what Keep Alive in MQTT is and what it does? We’ll learn about MQTT’s Keep Alive implementation and why it’s essential in this blog. The keep alive feature, as its name suggests, is used to keep a network connection alive. This term isn’t just used in MQTT; it’s also used in a variety of other protocols. Keep alive ensures that the connection between a server and a client stays available and that both users are aware that they are connected. By guaranteeing that only one connection is used per client, network congestion and latency are reduced. This is especially useful in Machine-to-Machine interactions, because the nodes have very limited resources. This is why the Keep Alive feature is necessary in MQTT Broker client connection & communication.

 Keep Alive in MQTT

We already know that MQTT is built on the TCP/IP model, which assures that packets are exchanged in a reliable, orderly, and error-free manner. In fact, MQTT’s transport layer employs Transmission Control Protocol, which includes its own Keep Alive capability. Then you might wonder why there is a requirement for an additional maintain alive implementation on the Application layer. Even with the TCP stay alive implementation, in real-world applications, particularly in the case of IoT devices with so many limitations on the network, the transfer between communicating clients can become out of sync due to clients crashing, unreliable networks, or other issues. Furthermore, because MQTT is client-decoupled, if a connection is only half-open, the side that is still connected continues to transmit messages and waits for acknowledgements. 

How MQTT Keep Alive works

When the client connects to the broker, the client sends the broker a keep alive time interval in seconds. This interval specifies the maximum amount of time that the broker and the client may be unable to communicate with one another. There is no need to send an extra message to check if the connection is still open as long as messages are sent within the keep-alive period. If the client does not send a message during the keep-alive period, the broker must send a ping request packet to indicate that it is still reachable. The broker will respond to the client with a ping response packet, and the connection will be maintained. The keep alive timer is reset when the ping replies are received, and the cycle restarts. A client that does not send a message or a ping request packet in one and a half times the keep alive interval shall be disconnected by the broker. If the client does not get a ping response packet from the broker, the connection is expected to be closed. If the client has established a Last Will and Testament message, the LWT message will be forwarded to all subscribed clients during this disconnect.

Example

Here’s a simple example of how MQTT’s keep alive feature helps to keep network communication running smoothly. When particular circumstances, possibly related to temperature, are fulfilled, a sensor connected to the network is expected to send a message. The exact condition that causes the message to be sent cannot be measured, and it can happen at anytime. This sensor is beneficial for ensuring the safety of network-connected automated processes. Setting the keep alive interval to a low estimate ensures that the broker is always informed of the status of the client’s connection. A condition-driven message will be sent, or a steady flow of PINGREQ messages will verify that the client is connected. If the time interval is exceeded, the broker will terminate the connection and send the last will and testament message. This should contain warnings or alarms in the case of an essential network component like this sensor, so that preventative measures can be done.

Client Take-Over

Clients who have lost contact with their broker will, in most cases, try to reconnect. MQTT allows the broker to do a client take-over, which allows the broker to cancel a half-open connection and let the same client to establish a new connection. It’s a fail-safe to prevent devices from disconnecting from the network due to whatever condition triggered the half-open connection.  Allowing brokers to participate in client takeovers can be essential to maintaining communication in a MQTT network. The takeover ensures that the client’s ability to restore communication with the broker is unaffected by the difficulty that resulted in a half-open connection. It eliminates a potential point of failure in the network when clients are unable to terminate connections on their own.

Another reason MQTT is a great solution for current distributed networks and inter-device communication is its keep alive feature. You can design a robust network solution that can endure instances of unstable connectivity with adequate configuration.

MQTT Retained Messages: What you need to know

MQTT Retained Messages: What you need to know

When a publisher publishes a message to a subject to which no one has subscribed, the message is simply discarded by the broker. By setting the retained message flag, the publisher can inform the broker to keep the last message on that topic. This feature allows you to store a single message per MQTT topic and send it to all current and future subscribers. To save a retained message, simply set a retained flag when publishing it to the broker. At that point, the message is delivered to all current subscribers as usual, as well as any future devices that subscribe to that topic. In this post, we will provide an overview of this feature and explain it using an example. Let’s take a look at this right now. 

Introduction

When we hear the phrase retained messages, we immediately think of messages that have been stored. These MQTT Retained messages are defined in the MQTT 3.1.1 specification and are created by publishing a message with the retained flag set to ‘True.’ The broker stores the most recently retained message as well as the MQTT QoS for that topic. When compared to MQTT v3.1.1, it provides a Retain Handling field in the subscription option and makes a more in-depth division of whether to send a retained message when the subscription is established in MQTT v5.0. As previously stated, a retained message is delivered to any currently subscribed clients in the same way that a normal message is, and it is also automatically sent to new clients who subscribe to that topic. Because only one retained message can be saved per topic, any future retained messages will replace any existing retained messages. 

For example,

In this example, the sensor sends the broker its current value along with a true retained message. This message is immediately received by the client who is online. The broker stores the message and delivers it to the second client once it comes online because the retained flag is set to true. As a result of this feature, when the offline client comes online, it is updated with the most recent sensor value.

Retaining messages with MQTT is an example of how it can be done.

As an initial message of configuration 

After a client subscribes to a topic, MQTT retain messages are sent to the client. You can publish a configuration message with the RETAIN flag set if you want all clients who subscribe to a topic to receive the MQTT retained message immediately after their subscription. When a new configuration message is published, subscribing clients receive updates to that configuration as well.

As a message of the last-known state

Devices can save messages by setting the RETAIN flag on current-state messages. When applications connect or reconnect, they can subscribe to this topic and receive the most recent reported state immediately after doing so. They can avoid having to wait for the device’s next message to see the current state this way.

We’ll go over how to set  and delete the MQTT retained messages feature in the sections below.

Enable retained message

A retain message is a standard MQTT publish message with the retained flag enabled. When a publisher sends data to the broker, it also sends some other information in addition to the topic and the message itself, as you are probably aware. The PUBLISH packet is the name of this packet. They’ve provided a field on the fixed header in this packet to set the retain flag to 1 or 0. It will be set to 0 by default, indicating that the message will not be saved. However, if we set it to 1, the broker will store the message and its corresponding Quality of Service level for that topic when it receives the packet.

Disable retained message

Can you figure out how to disable the retain feature for a specific topic? It’s not as simple as sending a message with the retain flag set to 0, because that’s the default condition. Sending an empty message to the same topic with the retain flag set to 1 is the correct way to do it. Yes, this may appear counter intuitive, but it was done on purpose to avoid conflicts with any real time publishing on the same topic that did not have the retain flag set to 1.

The MQTT retained messages feature is useful for preserving an object’s most recent state, which is especially useful when the state does not change frequently. The quality of the service settings has no impact on the messages that are retained.

Developing Scalable IoT Application with MQTT Broker

Developing Scalable IoT Application with MQTT Broker

Do you want to quickly and easily create an IoT implementation that meets your requirements? This is the right blog for you. Let’s see what would be the best choice to develop an IoT application easily. Building an IoT application comes with several challenges that are security, connectivity, data collection & processing etc. You already know that building an IoT application from scratch is one of the most difficult tasks. Because you must develop code for each one individually. Can you believe that a single framework contains everything you need to build any IoT application? Yes, Bevywise MQTT Broker is an IoT application framework that is suitable for the implementation of any IoT application. It provides a solution to all of the challenges mentioned above. Now let’s look at how we can configure an IoT application using that framework. 

Components required

Creating an application usually necessitates the use of components. It is essential that we have to ready with the components for developing an IoT implementation. Let’s take a look at what they are. MQTT Protocol, MQTT Clients, and MQTT Broker. When it comes to developing a large-scale IoT application, the MQTT protocol is the best choice. MQTT was found to be a flexible and scalable network protocol that is well-suited for developing an IoT application for greater implementation after taking into account a variety of developing and mature IoT protocols. MQTT Clients, which are the devices used for your implementation, can be used to develop your IoT application based on your requirements. MQTT Broker is the one that act as a middle ware to connect and communicate the MQTT clients. In that case, our Bevywise MQTT Broker also comes with Python extensions, making it perfect for building a full-fledged IoT application with high-level IoT implementation. 

MQTTRoute, a Bevywise MQTT Broker offers a free trial that allows you to experiment with the features and Python extensions to create IoT applications. You can get the free trial pack link by here –https://www.bevywise.com/mqtt-broker/download.html 

The MQTTRoute framework comprises all of the necessary options required to develop an IoT application. You can readily construct a large-scale IoT application with this and benefit from it. If you want to get in-depth detail about the installation and basic configuration of MQTTRoute. You can get it right here –https://www.bevywise.com/mqtt-broker/help.html

Now, Let us move on to the example to know more about the MQTT protocol. Assume you have two devices and various weather sensors (a humidity sensor and a temperature sensor). You’d like to send data on the humidity level and the temperature level to two different devices.

This task can be simply managed with MQTT. You must first set up a MQTT broker service. Then, on the broker, you may connect two sensors as clients and configure them to deliver data on the topics “Humid” and “Temp.” 

After that, you can connect the devices to the broker and subscribe the first one to the topic “Temp” and the second to the topic “Humid.” As a result, whenever the sensors publish the appropriate information to the broker, two connected devices will receive messages regarding humidity and temperature.

In the above example, the client sends the message to the MQTT broker, because as previously states that the clients cannot communicate each other directly. The messages sent by the clients are stored to the default database in the broker then it sends the messages to the appropriate subscribed client. The MQTT broker’s interaction with the devices was indicated in this example. Depending on your needs, developing an IoT application will require a specific UI and database. They can be found in the Python Extensions of the Bevywise MQTT Broker. Let’s look at how the Python Extensions for MQTT Broker allows you to customise an IoT implementation to your specific requirements. Here’s how these extensions can be used to create an IoT application.

Custom data storage based on IoT implementation

First and foremost, it is important to store data for an application. As previously stated, data for an application will be automatically stored in the MQTT Broker’s default database. By default, the Bevywise MQTT broker has the ability to store data in the following databases. MYSQL, MSSQL, PostgreSQL, and SQLite  However, we have the custom storage extension for you if you need advanced storage for developing your IoT implementation. If your IoT application necessitates the use of other advanced databases, you can specify the name and port of the database and use it in your implementation. For my IoT implementation, I’m using an elastic database through the use of the custom storage option. Because I want to analyse large amounts of data quickly and in near real time, which the Elastic Search database allows me to do. 

Let’s see how to configure the elastic search database here:

  • Make the necessary changes in the data_store.conf folder first. Set the DATASTORE option to CUSTOM after enabling the CUSTOMSTORAGE option.

  • The database’s PORT number and the INDEX NAME, which is the name by which you want the data to be stored and is specified here based on your needs.

  • We have commited the configuration files and the python extensions file in Git. Please refer to the data_store.conf folder and perform the above procedure using the link provided here. https://github.com/bevywise-networks/mqtt-broker

  • After that, set up the custom_store.py folder. The elastic search connector was used to set this up. Refer to the code below for a better understanding.

  • The websocket process is used to display stored data in the user interface. This will be useful when implementing a custom UI.

  • You can refer the custom_store.py extension files here –https://github.com/bevywise-networks/mqtt-broker/blob/main/extensions/custom_store.py

Crafting your own UI

The data is usually stored in a MQTT broker before being sent to the appropriate subscriber. But how can we be certain that the data is arriving and that the process is proceeding? That’s why we have a user interface for viewing messages between clients. By default, the Bevywise MQTT broker’s UI is useful for extracting data from devices, but you can also customise your dashboard and widgets here. However, if you want a completely different UI for your IoT implementation, use our Custom UI functionality for your high-level IoT application, which is one of the python extensions.

You can refer the custom_ui_server.py extension files here –https://github.com/bevywise-networks/mqtt-broker/blob/main/extensions/custom_ui_server.py

If you want to develop your own UI code based on your requirements for any IoT implementation, then insert it in a custom_ui_server.py folder. The code below is an example of how to see data in a line graph format for my IoT application; kindly refer this too.

Reference code for line graph

xaxis: {
type: ‘date’,
range: [olderTime, futureTime]
}
};
Plotly.relayout (graph_id, minuteView);
Plotly.extendTraces (graph_id, update, d)

Crafting_Custom_UI_reference_line_graph

 

This UI reference code is for live temperature monitoring application. Plotly provided this reference code. It takes the temperature of a room and displays it in a user interface. We can view it through the line graph’s output in this case.

Configure scheduler to generate alerts

We’ve already seen how to create your own database and customise the UI to view the data the way you want it. Taken the example given below, the room temperature monitor, and we’ll look at how to create conditions, alarms, alerts, and warnings in it. To do the above, you’ll need to use a custom Scheduler. This is also one of the Python extensions in the MQTT Broker.

You can refer the custom_scheduler.py extension files here – https://github.com/bevywise-networks/mqtt-broker/blob/main/extensions/custom_scheduler.py

Create your code based on your IoT implementation and insert it into the #write your code here section. A small example of the scheduler option is provided below, and you can write your own code to suit your needs.

Reference code for Alerting high temperature in room

def schedule_conf( ):

schedules={}

schedules={

‘STATUS’ : ‘DISABLE’,

‘SCHEDULES’ : [

{‘OnceIn’ : 5,‘methodtocall’ : fiveminschedule },

{‘OnceIn’ : 10,‘methodtocall’ : tenminschedule } ] }

if ( p_avg < data3 [‘msg’] [‘message’] [0] ){

document.getElementById (“Alarm”).innerHTML=data3 [‘msg’] [‘message’] [0] + “ALERT!” + “Temperature High”;

}

return schedules

The above code represents the condition applied which aggregate the temperature between every 5 minutes and 10 minutes schedule data and also refers to the alert setting when the temperature is high in the room. The data from the scheduler is pushed into the UI by the conditions listed above. If the temperature in the room is too high, we will receive an alert message via the UI. This way, you can easily configure it based on your needs to suit your IoT implementation. 

Enabling Advanced Authentication

Let’s take a look at MQTT broker security now. The data security in MQTT Broker is excellent. It is dependent on the security of TLS / SSL. However, your data still requires security, and we offer Custom authentication if you think security is essential for your IoT application. Custom authentication functionality in Python extensions will prove to be beneficial. Because, Bevywise MQTT Broker provides custom authentication functionality, allowing users to integrate any central IAM and SSO for your big level IoT implementation & for managing lot of users in your application based on your requirements.

Let’s see how to configure authentication in broker.conf folder

  • Set the REMOTEAUTH_ENABLED =YES

You can refer the broker.conf and custom_auth.py extension files here –https://github.com/bevywise-networks/mqtt-broker

# Request Retries Count

requests.adapters.DEFAULT_RETRIES = 3

# Request URL

url = “https://www.bevywise.com/auth”

# Request Timeout

request_timeout = 0.1

# Request Method

request_auth_method = “POST”

# POST | GET | PUT

When we manage a large number of users, as previously stated, if one of them forgets or mistypes his password, you can specify how many times you can try again in the requests.adapters.DEFAULT RETRIES field. This field’s default value is 3. You can change the count based on your IoT implementation. You can enter the URL of your authentication landing page in this field, “url = “https://www.bevywise.com/auth”. This validates your attempt to connect using with your login credentials.

All of the functionality described above is provided by the Python extensions in  Bevywise MQTT Broker. With these, you can easily set up a large-scale, flexible, and scalable implementation of your IoT application.

In our MQTTRoute, all python extensions were pushed to Git. Use this link to refer to all of the folders – https://github.com/bevywise-networks/mqtt-broker

 

To develop a scalable IoT application, start by downloading the free Bevywise MQTT Broker.

 

For the detail information about python extensions to develop any scalable IoT implementation, refer this link provided here – https://www.bevywise.com/mqtt-broker/python-mqtt-broker-interface.html

 

Persistent Sessions and Message Queuing – MQTT Fundamentals

Persistent Sessions and Message Queuing – MQTT Fundamentals

Did you know that after a client disconnects, the MQTT broker can store new messages for that client? The messages will be sent as rapidly as the client reconnects. MQTT Persistent sessions & message queuing is the name for this feature in MQTT. In this post, I’ll describe how it performs. Let’s look at this more closely.

MQTT Persistent Sessions or Clean Session

MQTT Persistent sessions as previously stated, the expiry interval specifies how long the session should be kept after a disconnect. Consider the following scenario: Assume you have two MQTT clients who are both connected to a MQTT broker. One MQTT client is publishing to a topic, while the other is subscribed to it. If a subscribing client disconnects from the broker unexpectedly, it will not receive any messages published to that topic until it reconnects. 

When connecting to the broker, we use the Persistent Session or Clean Session flags to solve this problem. When a client connects to the broker, the client has the option of requesting a persistent session. The broker will keep track of the client’s information as well as the topics to which it has subscribed. The broker will transmit all undelivered messages to the client when the client reconnects after a disconnection.

Setting a clean session

When a client connects to the broker, the client can create a persistent session by setting the cleanSession flag to False. This setting is set to True by default. Let’s glance at what a persistent session stores.

  • A session’s  existence (even without subscriptions).

  • The client’s whole subscriptions.

  • The client has not yet confirmed all messages in 1 or 2 flow on Quality of Service levels (QoS).

  • All new QoS 1 or 2 messages that the client missed while disconnection.

  • All QoS 2 messages received from clients are not yet fully accepted.

When should a clean session be used?

When a client must receive all (or and almost all) of the messages from a topic to which it has subscribed and the internet is unreliable, it is essential to use a Persistent Session. 

How are client-side actions handled by persistent sessions?

Let’s look at handling client-side persistent sessions. If you want a MQTT persistent session and tell the broker I want a persistent session in the CONNACK packet, there is a small flag called session persistent and the broker tells you as a client if the broker remembers you as a client or if the broker believes you are connecting for the first time. And this provides you with a convenient way on the client side to determine whether you need to resubscribe to your MQTT topics, or if the broker already has information about you. So this is something to look into, and most libraries allow you to check clean sessions in order to create MQTTclients that are resilient.

Message Queuing

Then there’s message queuing, which is unique to MQTT. Because messages are queued per client. This means that if you have, but only for persistent session clients, you can use message queuing. If you have an active subscription and a persistent session client that is disconnected from the broker, the broker will queue all quality of service (QoS 1 or 2) messages that you missed while disconnected. This means that if you’re disconnected for a while and then reconnect, the broker will receive messages you missed, so even if you’re connected on the client side, you’ll get the most up-to-date information and won’t miss a message. So this is really great, but you need a persistent session here and you need to use the quality of service one or two messages here. Keep in mind that messages with a quality of service of zero are never queued.

We’ve already discussed persistent sessions and message queuing. It’s possible that if your MQTT broker doesn’t have expire sessions, persistent sessions, or queues, you’re susceptible to denial of service attacks. So, if you’re working on a professional project, make sure you have a MQTT that supports all of the important operations and configurations.

Getting to know MQTT Topics & its Basics

Getting to know MQTT Topics & its Basics

MQTT messages aren’t sent directly from one thing to another. Rather, they are assigned to “MQTT topics.” The MQTT broker then distributes the messages to any clients who have subscribed to them. The term topic in protocol refers to a UTF-8 string used by the broker to filter messages for each subscribed client. When someone subscribes to a topic and then publishes a message to that topic with the retained message set to True, the topics are established. When the last subscriber to that broker disconnects, a clean session is established. Clean session is set to True when a MQTT client connects. 

An MQTT topic’s structure

sensors/water_level/water_status – sensors, water_level & water_status are topic levels.

Forward slash ( / ) –   a topic level separator.

Topic level Separator

The topic is made up of one or more topic levels. A forward slash separates each topic level (topic level separator). The broker does not require that topic be pre-registered.

MQTT topics are very lightweight in compared to a message queue. Before publishing or subscribing to a topic, the client does not need to create topics. Without any prior initialization, the broker accepts any valid topic.

Here are a few examples to look at: 

A look at MQTT Topics used in Bevywise Broker.

Topic Semantic and Usage

Except for the length of a UTF-8 encoded string, there is no restriction to the number of levels in a Topic Name or Topic Filter. Topic Names and Topic Filters are subject to the following guidelines:

  • It is important to note that each topic must contain at least one character and that the topic string allows for empty spaces.

  • The null character cannot be used in topic names or topic filters.

  • The MQTT topics are case sensitive.

  • For example, sensor/temperature and Sensor/Temperature, are two distinct topics.

Topic WildCards

The Topic Filter of a subscription can contain special wildcard characters that allow you to subscribe to multiple topics at once. The wildcard characters can be used in Topic Filters but not in Topic Names. Wildcards are classified into two types:

  • Single-level wildcard and

  • Multi-level wildcard.

When one of the two wildcard characters is encountered in Topic Filters given by subscribing Clients, the use of the topic level separator is significant. Separators for topic levels can exist anywhere in a Topic Filter or a Topic Name. A zero length topic level is indicated by adjacent topic level separators.

Single-level wildcard

A single-level wildcard character in a topic is represented by the plus symbol “+.” The Topic Filter’s single-level wildcard can be used at any level, including the first and last. It must fill the entire level of the filter wherever it is utilized. It can be used in the Topic Filter at multiple levels and in conjunction with the multilevel wildcard.

For example,

  • “+”

  • “tank/+/water_level” – These are valid topics with single level wildcard.

Any topic like “tank/water_level/+” matches “tank/water_level/flow_rate1” and “tank/water_level/flow_rate2,” but not “tank/water_level/flow_rate1/increase.” Also, because the single-level wildcard matches only one level, “tank/+” does not match “tank,” but “tank/” does.

Multi-level wildcard

The wildcard character “#” is a multilevel wildcard that can be used to match any number of levels within a topic. The parent and any number of child levels are represented by the multi-level wildcard. The multi-level wildcard character must either be given alone or after a topic level separator. It must be the last character supplied in the Topic Filter in either scenario.

For example,

  • “tank/water_level/#” is a valid topic with multilevel wildcard.

If a Client subscribes to “tank/water_level/flow_rate1/#,” for example, it will get messages published under the following topic names:

  • “tank/water_level/flow_rate1”

  • “tank/water_level/flow_rate1/increasing”

  • “tank/water_level/flow_rate1/increasing/decreasing”

Reserved Topics 

Topics beginning with a dollar sign ($) are reserved topics. The $-symbol topics are for the broker’s internal statistics. Topic Filters that begin with a wildcard character (# or +) must not be matched with Topic Names that begin with a $ character. Clients should not be able to send messages to other Clients with such Topic Names, according to the Server. Topic Names that begin with a leading $ character may be used for additional purposes by server implementations. $SYS/ has become a prominent prefix for topics containing Server-specific data or control APIs. A topic containing a leading $ character cannot be used by applications for their own reasons. 

All of the following information is usually stored in $SYS/, however broker implementations differ. A few instances are shown below.

  • Any messages published on a topic beginning with a $ will not be received if you subscribe to “#.”

  • Any messages sent to “$SYS/broker/Clients” will not be received by a subscription to “+/broker/Clients.”

  • Messages published to subjects beginning with “$SYS/” will be received if you subscribe to “$SYS/#.”

  • Messages sent to “$SYS/monitor/Clients” will be received if you subscribe to “$SYS/broker/+.”

  • A Client must subscribe to both “#” and “$SYS/#” in order to receive messages from topics that begin with $SYS/ and subjects that do not begin with a $.

The above are the basics of message topics. MQTT topics, as you can see, are dynamic and offer a lot of flexibility. Keep using its extensibility without changing the whole topic hierarchy for the subsequent inclusion of new sensors by adding these to your topic tree.