Do you know what Keep Alive in MQTT is and what it does? We’ll learn about MQTT’s Keep Alive implementation and why it’s essential in this blog. The keep alive feature, as its name suggests, is used to keep a network connection alive. This term isn’t just used in MQTT; it’s also used in a variety of other protocols. Keep alive ensures that the connection between a server and a client stays available and that both users are aware that they are connected. By guaranteeing that only one connection is used per client, network congestion and latency are reduced. This is especially useful in Machine-to-Machine interactions, because the nodes have very limited resources. This is why the Keep Alive feature is necessary in MQTT Broker client connection & communication.

 Keep Alive in MQTT

We already know that MQTT is built on the TCP/IP model, which assures that packets are exchanged in a reliable, orderly, and error-free manner. In fact, MQTT’s transport layer employs Transmission Control Protocol, which includes its own Keep Alive capability. Then you might wonder why there is a requirement for an additional maintain alive implementation on the Application layer. Even with the TCP stay alive implementation, in real-world applications, particularly in the case of IoT devices with so many limitations on the network, the transfer between communicating clients can become out of sync due to clients crashing, unreliable networks, or other issues. Furthermore, as MQTT is client-decoupled, if a connection is only half-open, the side that is still connected continues to transmit messages and waits for acknowledgements. 

How MQTT Keep Alive works

When the client connects to the broker, the client sends the broker a keep alive time interval in seconds. This interval specifies the maximum amount of time that the broker and the client may be unable to communicate with one another. There is no need to send an extra message to check if the connection is still open as long as messages are sent within the keep-alive period. If the client does not send a message during the keep-alive period, the broker must send a ping request packet to indicate that it is still reachable. The broker will respond to the client with a ping response packet, and the connection will be maintained. The keep alive timer is reset when the ping replies are received, and the cycle restarts. A client that does not send a message or a ping request packet in one and a half times the keep alive interval shall be disconnected by the broker. If the client does not get a ping response packet from the broker, the connection is expected to be closed. If the client has established a Last Will and Testament message, the LWT message will be forwarded to all subscribed clients during this disconnect.

Example

Here’s a simple example of how MQTT’s keep alive feature helps to keep network communication running smoothly. When particular circumstances, possibly related to temperature, are fulfilled, a sensor connected to the network is expected to send a message. The exact condition that causes the message to be sent cannot be measured, and it can happen at anytime. This sensor is beneficial for ensuring the safety of network-connected automated processes. Setting the keep alive interval to a low estimate ensures that the broker is always informed of the status of the client’s connection. A condition-driven message will be sent, or a steady flow of PINGREQ messages will verify that the client is connected. If the time interval is exceeded, the broker will terminate the connection and send the last will and testament message. This should contain warnings or alarms in the case of an essential network component like this sensor, so that preventative measures can be done.

Client Take-Over

Clients who have lost contact with their broker will, in most cases, try to reconnect. MQTT allows the broker to do a client take-over, which allows the broker to cancel a half-open connection and let the same client to establish a new connection. It’s a fail-safe to prevent devices from disconnecting from the network due to whatever condition triggered the half-open connection.  Allowing brokers to participate in client takeovers can be essential to maintain the communication in a MQTT network. The takeover ensures that the client’s ability to restore communication with the broker is unaffected by the difficulty that resulted in a half-open connection. It eliminates a potential point of failure in the network when the clients are unable to terminate connections on their own.

Another reason for saying MQTT is a great solution for current distributed networks and inter-device communication is its keep alive feature. You can design a robust network solution that can endure instances of unstable connectivity with adequate configuration.

When a publisher publishes a message to a subject to which no one has subscribed, the message is simply discarded by the broker. By setting the retained message flag, the publisher can inform the broker to keep the last message on that topic. This feature allows you to store a single message per MQTT topic and send it to all current and future subscribers. To save a retained message, simply set a retained flag when publishing it to the broker. At that point, the message is delivered to all current subscribers as usual, as well as any future devices that subscribe to that topic. In this post, we will provide an overview of this feature and explain it using an example. Let’s take a look at this right now. 

Introduction

When we hear the phrase retained messages, we immediately think of messages that have been stored. These MQTT Retained messages are defined in the MQTT 3.1.1 specification and are created by publishing a message with the retained flag set to ‘True.’ The broker stores the most recently retained message as well as the MQTT QoS for that topic. When compared to MQTT v3.1.1, it provides a Retain Handling field in the subscription option and makes a more in-depth division of whether to send a retained message when the subscription is established in MQTT v5.0. As previously stated, a retained message is delivered to any currently subscribed clients in the same way that a normal message is, and it is also automatically sent to new clients who subscribe to that topic. As only one retained message can be saved per topic, any future retained messages will replace the existing retained messages. 

For example,

In this example, the sensor sends the broker its current value along with a true retained message. This message is immediately received by the MQTT client who is online. The broker stores the message and delivers it to the second client once it comes online as the retained flag is set to true. As a result of this feature, when the offline client comes online, it is updated with the most recent sensor value.

Retaining messages with MQTT is an example of how it can be done.

As an initial message of configuration 

After a client subscribes to a topic, MQTT retain messages are sent to the client. You can publish a configuration message with the RETAIN flag set, if you want all clients who subscribe to a topic to receive the MQTT retained message immediately after their subscription. When a new configuration message is published, subscribing clients receive updates to that configuration as well.

As a message of the last-known state

Devices can save messages by setting the RETAIN flag on current-state messages. When applications connect or reconnect, they can subscribe to this topic and receive the most recent reported state immediately after doing so. They can avoid waiting for the device’s next message to see the current state this way.

We’ll go over how to set  and delete the MQTT retained messages feature in the sections below.

Enable retained message

A retain message is a standard MQTT publish message with the retained flag enabled. When a publisher sends data to the broker, it also sends some extra information in addition to the topic and the message, as you are probably aware. The PUBLISH packet is the name of this packet. They’ve provided a field on the fixed header in this packet to set the retain flag to 1 or 0. It will be set to 0 by default, indicating that the message will not be saved. However, if we set it to 1, the broker will store the message and its corresponding Quality of Service level for that topic, when it receives the packet.

Disable retained message

Can you figure out how to disable the retain feature for a specific topic? It’s not as simple as sending a message with the retain flag set to 0, as that’s the default condition. Sending an empty message to the same topic with the retain flag set to 1 is the correct way to do it. Yes, this may appear counter intuitive, but it was done on purpose to avoid conflicts with any real time publishing on the same topic that did not have the retain flag set to 1.

The MQTT retained message feature is useful for preserving an object’s most recent state, which is especially useful when the state does not change frequently. The quality of the service settings has no impact on the messages that are retained.

Did you know that after a client disconnects, the MQTT broker can store new messages for that client? The messages will be sent as rapidly as the client reconnects. MQTT Persistent sessions & message queuing is the name for this feature in MQTT. In this post, I’ll describe how it performs. Let’s look at this more closely.

MQTT Persistent Sessions or Clean Session

MQTT Persistent sessions as previously stated, the expiry interval specifies how long the session should be kept after a disconnect. Consider the following scenario: Assume you have two MQTT clients that are both connected to a MQTT broker. One MQTT client is publishing to a topic, while the other subscribes it. If the subscribing client disconnects from the broker unexpectedly, it will not receive any messages published to that topic until it reconnects. 

When connecting to the broker, we use the Persistent Session or Clean Session flags to solve this problem. When a client connects to the broker, the client has the option of requesting a persistent session. The broker will keep track of the client’s information as well as the topics to which it is subscribed. The broker will transmit all the undelivered messages to the client when the client reconnects after a disconnection.

Setting a clean session

When a client connects to the broker, the client can create a persistent session by setting the cleanSession flag to False. This setting is set to True by default. Let’s take a glance at what a persistent session stores.

• A session’s  existence (even without subscriptions).

• The client’s whole subscriptions.

• The client has not yet confirmed all messages in 1 or 2 flow on Quality of Service levels (QoS).

• All new QoS 1 or 2 messages that the client missed while disconnection.

• All QoS 2 messages received from clients are not yet fully accepted.

When should a clean session be used?

When a client needs to receive all (or / and almost all) of the messages from a topic to which it has subscribed and the internet is unreliable, it is essential to use a Persistent Session. 

How are client-side actions handled by persistent sessions?

Let’s look at handling client-side persistent sessions. If you want a MQTT persistent session and tell the broker I want a persistent session in the CONNACK packet, there is a small flag called session persistent and the broker tells you as a client if the broker remembers you as a client or if the broker believes you’re connecting for the first time. And this provides you with a convenient way on the client side to determine whether you need to resubscribe to your MQTT topics, or if the broker already has information about you. So this is something to look into, and most libraries allow you to check clean sessions in order to create MQTTclients that are resilient.

Message Queuing

Then there’s message queuing, which is unique to MQTT. Using this messages can be queued per client. Only for persistent session clients, you can use message queuing. If you have an active subscription and a persistent session client that is disconnected from the broker, the broker will queue all quality of service (QoS 1 or 2) messages that you missed while disconnected. This means that if you’re disconnected for a while and then reconnect, the broker will receive messages you missed, so even if you’re connected on the client side, you’ll get the most up-to-date information and won’t miss a message. So this is really great, but you need a persistent session here and you need to use the quality of service one or two messages here. Keep in mind that messages with a quality of service of zero are never queued.

We’ve already discussed persistent sessions and message queuing. It’s possible that if your MQTT broker doesn’t have expire sessions, persistent sessions, or queues, you’re susceptible to denial of service attacks. So, if you’re working on a professional project, make sure you have a MQTT that supports all of the important operations and configurations.

MQTT messages aren’t sent directly from one thing to another. Rather, they are assigned to “MQTT topics.” The MQTT broker then distributes the messages to any clients who have subscribed to them. The term topic in protocol refers to a UTF-8 string used by the broker to filter messages for each subscribed client. When someone subscribes to a topic and then publishes a message to that topic with the retained message set to True, the topics are established. When the last subscriber to that broker disconnects, a clean session is established. Clean session is set to True when a MQTT client connects. 

An MQTT topic’s structure

sensors/water_level/water_status – sensors, water_level & water_status are topic levels.

Forward slash ( / ) –   a topic level separator.

Topic level Separator

The topic is made up of one or more topic levels. A forward slash separates each topic level (topic level separator). The broker does not require that topic to be pre-registered.

MQTT topics are very lightweight in compared to a message queue. Before publishing or subscribing to a topic, the client does not need to create topics. Without any prior initialization, the broker accepts any valid topic.

Here are a few examples to look at: 

A look at MQTT Topics used in Bevywise Broker.

Topic Semantic and Usage

Except for the length of a UTF-8 encoded string, there is no restriction to the number of levels in a Topic Name or Topic Filter. Topic Names and Topic Filters are subject to the following guidelines:

• It is important to note that each topic must contain at least one character and that the topic string allows for empty spaces.

• The null character cannot be used in topic names or topic filters.

• The MQTT topics are case sensitive.

• For example, sensor/temperature and Sensor/Temperature, are two distinct topics.

Topic WildCards

The Topic Filter of a subscription can contain special wildcard characters that allow you to subscribe to multiple topics at once. The wildcard characters can be used in Topic Filters but not in Topic Names. Wildcards are classified into two types:

• Single-level wildcard and

• Multi-level wildcard.

When one of the two wildcard characters is encountered in Topic Filters given by subscribing Clients, the use of the topic level separator is significant. Separators for topic levels can exist anywhere in a Topic Filter or a Topic Name. A zero length topic level is indicated by adjacent topic level separators.

Single-level wildcard

A single-level wildcard character in a topic is represented by the plus symbol “+.” The Topic Filter’s single-level wildcard can be used at any level, including the first and last. It must fill the entire level of the filter wherever it is utilized. It can be used in the Topic Filter at multiple levels and in conjunction with the multilevel wildcard.

For example,

• “+”

• “tank/+/water_level” – These are valid topics with single level wildcard.

Any topic like “tank/water_level/+” matches “tank/water_level/flow_rate1” and “tank/water_level/flow_rate2,” but not “tank/water_level/flow_rate1/increase.” Also, because the single-level wildcard matches only one level, “tank/+” does not match “tank,” but “tank/” does.

Multi-level wildcard

The wildcard character “#” is a multilevel wildcard that can be used to match any number of levels within a topic. The parent and any number of child levels are represented by the multi-level wildcard. The multi-level wildcard character must either be given alone or after a topic level separator. It must be the last character supplied in the Topic Filter in either scenario.

For example,

• “tank/water_level/#” is a valid topic with multilevel wildcard.

If a Client subscribes to “tank/water_level/flow_rate1/#,” for example, it will get messages published under the following topic names:

• “tank/water_level/flow_rate1”

• “tank/water_level/flow_rate1/increasing”

• “tank/water_level/flow_rate1/increasing/decreasing”

Reserved Topics 

Topics beginning with a dollar sign ($) are reserved topics. The $-symbol topics are for the broker’s internal statistics. Topic Filters that begin with a wildcard character (# or +) must not be matched with Topic Names that begin with a $ character. Clients should not be able to send messages to other Clients with such Topic Names, according to the Server. Topic Names that begin with a leading $ character may be used for additional purposes by server implementations. $SYS/ has become a prominent prefix for topics containing Server-specific data or control APIs. A topic containing a leading $ character cannot be used by applications for their own reasons. 

All of the following information is usually stored in $SYS/, however broker implementations differ. A few instances are shown below.

• Any messages published on a topic beginning with a $ will not be received if you subscribe to “#.”

• Any messages sent to “$SYS/broker/Clients” will not be received by a subscription to “+/broker/Clients.”

• Messages published to subjects beginning with “$SYS/” will be received if you subscribe to “$SYS/#.”

• Messages sent to “$SYS/monitor/Clients” will be received if you subscribe to “$SYS/broker/+.”

• A Client must subscribe to both “#” and “$SYS/#” in order to receive messages from topics that begin with $SYS/ and subjects that do not begin with a $.

The above are the basics of message topics. MQTT topics, as you can see, are dynamic and offer a lot of flexibility. Keep using its extensibility without changing the whole topic hierarchy for the subsequent inclusion of new sensors by adding these to your topic tree.

Today, the MQTT protocol is the most widely used and well-received TCP/IP IoT protocol in the world. It’s no surprise that the protocol’s widespread adoption has resulted in a high demand for further development of the MQTT specification. MQTT 5 attempts to meet this demand. In Manufacturing systems & logistics fields, and the smart home application, as well as enterprise IoT applications and mobile applications, are all popular MQTT v5 use cases.

MQTT 5 features overview

The new features in MQTT version 5.0 are intended to accomplish the following objectives:

Large-scale system performance:  The communication between thousands, if not millions, of devices is now more streamlined. There are no protocol constraints, but MQTT 5 an appropriate architecture is required to organize these many devices.

Reporting of errors: The return code in the MQTTv5.0 protocol has been renamed to a reason code, which can indicate a wider range of failures.

Common interactions are implemented: The current version has standardised the many ways devices interact with one another. The system now includes the ability to define the capabilities of participating devices and how they respond to queries. 

Extensibility mechanisms have been included: Custom properties can now be specified, as well as the content type or payload format.

Better Support: Particularly for smaller users who want to use this latest protocol version to increase their productivity.

 

How MQTT 5 is different from MQTT 3?

MQTT v5.0 is MQTT 3.1‘s successor. Most importantly, MQTTv5.0 does not support backward compatibility (like v3.1.1). As there are so many new items introduced, current implementations must be revisited.

• The Session Expiry Interval property can be used to achieve session expiration. The subscription time, for example, might be included in the Topic, as well as the amount of time the message will be held.

• Restrictions on both the MQTT client and the server Limitations such as maximum packet size (number of bytes to broadcast) and maximum to receive can be built-in (either a client or server, the number of messages to be sent at the same time).

• Will Delay Interval is a property that allows you to send messages on a timer.

• Server Redirect or Server Reference are attributes that can help in the transfer of packets between brokers or servers.

• Message Expiry Interval is the amount of time that messages are stored before they expire.

• Payload format indicator and content type – The sort of payload flags that can be utilised is defined by this attribute: bytes (binary), UTF-8, or MIME types.

• Topic Aliases – For example, the alias 1 is assigned to the topic. Then this function can help you by reducing the amount of data packages you need.

• Response Topics – The protocol can work similarly to an HTTP protocol with a response-request scheme when Response Topics are used.

• Non-Local Publishing- Users can choose whether they want to receive messages from the client.

• Retained Message Control – The ordering of messages is controlled by this option.

• Subscription identifier, which is used to identify the subscription on the server.

• Subscriptions that can be shared to allow for more flexible subscriptions with extra symbols and filtering options.

• Reason codes on all ACK messages –  Errors can happen at any point in the process. In MQTTv3.1.1, there was little information from the server about what went wrong in various phases, such as establishing communication, sending a message, or subscribing to topics.

• Server Disconnection –  Unlike 3.1, MQTT 5.0 allows packets about disconnections to be sent from both the client and server sides.

• User properties – Keys can be used to store values for a variety of properties.

• Enhanced Authentication – The mechanism of query / response style authentication method which executes two-way authentication between the client and the server. 

A Technical Overview of MQTT 5’s New Features

Session Expiry

Detach the Clean Session flag into a Clean Start flag, which indicates that the session should begin without using an existing session, and a Session Expiry interval, which specifies how long the session should be retained after a disconnect. At disconnect, the session expiry interval can be changed. In MQTT v3.1, setting Clean Start to 1 and Session Expiry Interval to 0 is equivalent to setting Clean Session to 1. 

Message Expiry 

Message expiry is an optional component of the PUBLISH control packet. The Message Expiry Interval in the PUBLISH packet delivered by the Server to a Client must be set to the received value excluding the time the message has been waiting in the Server. The Publish Expiry Interval, which applies to both online and queued messages, is the duration of the publication in seconds. Allow for the setting of an expiry interval when a message is published.

Reason Code and Reason String

Include a reason code in all response packets. CONNACK, PUBACK, PUBREC, PUBREL, PUBCOMP, SUBACK, UNSUBACK, DISCONNECT, and AUTH are some examples. This allows the invoker to check whether the requested function was completed successfully. Most packets containing a reason code should now allow for an optional reason string as well. This is intended to be used for problem identification rather than parsing by the receiver. 

Topic Alias 

Topic alias in MQTT v5 allows you to use the pub-sub messaging model in a more flexible way. Topic aliases can effectively save both network and processing resources for messages repeatedly published to a finite collection of topics, especially at big volumes. It allows the topic name to be abbreviated to a small integer to reduce the size of the MQTT packet overhead. The Client and Server each specify the maximum number of topic aliases they will accept.

Server Disconnect

The server can send a disconnect message to the client along with a reason code in MQTT 5. Unlike MQTT 3.1, MQTT 5.0 allows messages about disconnections to be sent from both the client and the server. Allow the Server to send DISCONNECT to specify why the connection is being closed. If there were any issues with the server, it would just terminate the session. 

Payload Format & Content type

When sending a PUBLISH message, another identifier/value pair is available. This is the indicator for the Payload Format. When a message is published, allow the payload format (binary, text) and a MIME style content type to be specified. These are forwarded to the message’s recipient.

Request / Response Configuration 

Request/Response pattern provides the Response Topic and Correlation Data features in MQTT to allow response messages to be routed back to the request’s publisher. Also, give the Client the ability to get configuration information from the Server about how to build the response topics. This mechanism enables the implementation of a “business acknowledgement” functionality that is extensible, dynamic, and transparent.

Shared Subscriptions

With standard MQTT protocols, shared subscriptions are a wonderful way to distribute messages among several MQTT subscribers. In version 5, support for shared subscriptions was added, allowing for load-balanced subscription consumers. A reserved topic root named $share and a share name are used to group subscribers in shared subscriptions.

Subscription Identifiers

When subscribing, the client might specify a subscription identifier. When you successfully create or edit a subscription, the broker will generate and store the mapping relationship between this subscription and the subscription identifier. Allows a numeric subscription identifier to be given on a SUBSCRIBE and returned when the message is delivered. This enables the Client to determine which subscription or subscriptions were responsible for the delivery of the message.

Subscription Options

Subscription options should be defined primarily for message gateway applications. There’s also a noLocal option for not sending messages from this Client, as well as options for how to handle retained messages on subscription. To customize the server behaviour, you can use more subscription options in MQTT v5.

Flow Control

Allow the Client and Server to select the amount of outstanding reliable messages (QoS>0) that they will accept separately. To stay within this quota, the sender pauses sending such messages. This is used to limit the rate at which reliable communications are sent and the number of messages in flight at any given time.

User Properties

Most packets should have User Properties. Client applications define the user properties on PUBLISH, which are included with the message. The Server sends the user properties on PUBLISH and Will Properties to the message receiver. The Server implementation defines the user properties on the CONNECT, SUBSCRIBE, and UNSUBSCRIBE packets. The sender defines the user characteristics on CONNACK, PUBACK, PUBREC, PUBREL, PUBCOMP, SUBACK, UNSUBACK, and AUTH packets, and each sender implementation is unique. MQTT does not define the meaning of user properties.

Maximum Packet Size

Allow the Client and Server to declare the maximum packet size they can support independently. Sending a larger packet by the session partner is an error. This property must not be sent if it will raise the size of the PUBACK packet over the receiver’s Maximum Packet Size.

Optional server feature availability

Define a set of features that the Server does not allow, and give the Server a reason to determine the Client about them. Maximum QoS, Retain Available, Wild card Subscription Available, Subscription Identifier Available, and Shared Subscription Available are some of the characteristics that can be specified this way. Using features that the Server has indicated are not available is an error for the Client. 

In previous versions , a Server could refuse to implement a feature by declaring that the Client is not authorised to use it. When the Client utilises one of these features anyhow, this feature allows such optional behaviour to be declared and adds corresponding Reason Codes.

Will Delay

Allow for a time delay between the end of the connection and the transmission of the will message. This is done so that the will message is not sent if the connection to the session is re-established. This allows for minor interruptions of the connection to occur without others being notified. 

Server Keep Alive

Keep alive ensures that the broker-client connection is still active and that both the broker and the client are aware that they are linked. It allows the Server to provide the keep alive value the Client should use. This allows the Server to define a maximum keep alive time while still having the Client respect it.

Server Reference

The Client can utilise the Server reference to find another server to use. It allows  the Server to specify a different server for CONNACK and DISCONNECT. This can be used to redirect or to perform provisioning.

Assigned Client Identifiers 

Allows the Broker to assign Client IDs to clients in a consistent manner. Return the assigned ClientID in situations where the ClientID is assigned by the Server. The restriction that Server-assigned ClientIDs could only be used with Clean Session=1 connections is also lifted.

Why You need MQTT 5? 

For most IoT use cases, the MQTT 5 specification has become the logical choice. MQTT 5’s new features successfully solves version 3’s limitations while also opening the path for future development. The protocol adoption is expected to sky-rocket in the coming years across all sectors, including industrial, automotive, critical infrastructure, logistics, smart cities, and so on. MQTT is on the edge of becoming the industry standard for all things connected to the Internet of Things.