MQTT For Web Developers

What is this about?

During this year's unKonf, an unconference taking place in the German city of Mannheim, I prepared a quick talk about MQTT and how it could be used in web applications. This article is a written, improved version of this talk, taking into account all the feedback that I received.

The Message Queuing Telemetry Transport protocol, or MQTT for short, is a protocol that delivers messages between clients via a central broker. It was initially designed for low-bandwidth, high-latency networks (like cellular) to be used directly on top of TCP, and optimized for clients with low storage and computing capabilities (like embedded devices) but can also be used on top of Websockets. This enables us to use MQTT in traditional web applications.

This article will focus on exactly that by designing a chat application using MQTT 3.1.1 as a practical example. I've also implemented this app using mqtt.js, react, shadcn, tailwind and the redux-toolkit to build the frontend and mosquitto as the MQTT broker. You can check out the repository here if you want to play along, but I have also included screen recordings of the app at the end of every chapter.

Why should you care?

MQTT comes with many features that allow its users to build robust and scalable messaging systems without writing any backend code. Features provided by the protocol include the ability for clients to selectively choose messages relevant to them at any given time and the option to define a measure of importance for each message, controlling how hard all participants should try to deliver it. The websocket operation mode allows us to integrate it directly into web applications, making it ideal for building systems that display various types of real-time information from multiple sources to many clients simultaneously.

MQTT Basics

MQTT governs the communication between message senders, a central broker that distributes these messages, and message receivers. There are no direct connections between MQTT clients: Every message is proxied via the broker.

Every MQTT connection starts with a connect handshake between a client and the broker. The simplest form of this handshake does not contain much besides a session identifier to identify the running session.

After a connection is established, the server periodically sends pings to the client to determine whether the client is still alive. The interval in which these pings are sent can be configured by the client with the connect package.

Sending And Receiving Messages

When clients want to send a message, they can send a publish package to the broker. This package contains the message payload and is addressed to a topic. Topics are hierarchical strings and, similar to a file path, separated by slashes, e.g. room/hallway/temperature.

The message is then forwarded by the broker to all clients that are subscribed to this topic. A client subscribes to a topic by sending a subscribe message to the broker that either contains a topic or a topic pattern. A topic pattern can contain two different kinds of wildcard characters: + and #. The + character matches exactly one level of the topic hierarchy and can be used multiple times, while the # character matches any number of levels and must be the last character of a pattern string. Both wildcards can only match a complete level of the topic hierarchy, which means that room/hall+ would be an invalid pattern. The following subscription patterns would match the message sent to the topic room/hallway/temperature:

// Examples of VALID patterns that match room/hallway/temperature
// Complete topic
room/hallway/temperature
// One single level wildcard
room/hallway/+
// One single level wildcard that matches every temperature topic
room/+/temperature
// Two single level wildcards that match every message in every room
room/+/+
// Three single level wildcards that match every message sent on the broker
// with three hierarchy levels
+/+/+
// Multilevel wildcard for all topics that start with room/
room/#
// The single multilevel wildcard matches all topics on a broker
#

// Examples of INVALID patterns
// A multilevel wildcard is not last character of the pattern
room/#/temperature
// Multiple multilevel wildcards are not allowed
room/#/#
// Wildcards can only be used to match complete levels
room/hall+/temperature

With these three building blocks (a session identifier, the publish and subscribe commands) we can build a simple chat application. This application would work as follows:

A user can choose any username, which the application will use as the MQTT session identifier (this will be important later)
By convention, a user can send messages to any other user by publishing a message to the topic chat/messages/[receiver]/[sender]
To receive messages, a user subscribes to the topic chat/messages/[username]/+ to get all the messages that are addressed to themselves.

If you are running the demo application, you can try out this first implementation of our chat app by navigating to http://localhost:5173/?stage=10 and entering any name to login and start chatting.

Quality of Service (QoS)

When a message is published to a MQTT topic, the sender can specify a Quality of Service (QoS) level. When a client subscribes to a topic, a QoS value can also be specified which determines the maximum QoS level that is delivered to this subscription. Colloquially speaking, the QoS value determines how hard the MQTT broker should try to deliver the message: the higher the QoS, the harder the broker tries. If the QoS level of a subscription is lower than the QoS level of a message that should be sent to a subscription, the broker downgrades the QoS of the message to match the one of the subscription.

Before we take a look at the different QoS levels, it is important to talk about the definition of "delivering" a message. In the MQTT-sense, a message is considered to be delivered when the sender has tried its best to send the message (in the case of QoS0) or after the package has been acknowledged via some kind of ack packet (in the case of QoS1/2). As both of these actions occur before the data is passed to the receiving application, MQTT cannot manage message processing. If the client crashes immediately after receiving the message, the message is still considered delivered. This is different from other message queuing systems, such as RabbitMQ, where messages can be redelivered if the processing of the message failed.

QoS 0: Deliver At Most Once

QoS 0 is the default value for many MQTT clients when publishing a message. On this level, the message is delivered in a best effort fashion, but both the client and broker can to drop the message if they cannot deliver it on the first try for any reason. Typical cases when messages are dropped are when the client is offline, the network is congested or the broker is overloaded.

In our chat application we could use QoS 0 for informational messages, like the information about when a user comes online or goes offline. We won't implement this right away, but keep this in mind for later.

QoS 1: Deliver At Least Once

At QoS 1, the published message is delivered at least once, but may be delivered multiple times. At this level, many client libraries implement a message queue to keep track of the messages that have not yet been acknowledged by the broker. Brokers now require a client to send an puback package to confirm that it has successfully received the package. They also start to keep track of which messages have been delivered to which sessions based on the session identifier, so that they can resend the message if a client is not online when a new message is published.

It is important to note that, even though it is widely supported, this feature of storing messages for offline clients (called session persistence) is not required for a MQTT-compliant broker. The section 4.1 of the standard states that

The Client and Server MUST store Session state for the entire duration of the Session. A Session MUST last at least as long it has an active Network Connection.

Until now, our chat application had the problem that messages can only be delivered when the receiver is online. By using session persistence and QoS 1 messages we can make sure that every chat message arrives at the receiver at least once. On the other hand we don't care about redeliveries, because we can easily deduplicate the message on the client side, so we don't need QoS 2 (see below).

If you are running the demo application, you can try out this improved implementation of our chat app by navigating to http://localhost:5173/?stage=20 or check the Use QoS 1 checkbox. You can still enter any name to login, but whenever you close a chat window and login again, you will receive all the messages that have been sent to you while you were offline.

QoS 2: Deliver Exactly Once

QoS 2 is the strictest of the three QoS levels. At this level, each message is delivered exactly once to every receiver. This is asserted by a two-way handshake that is performed whenever a message is delivered.

This quality level is not implemented by many brokers because it is the most expensive in terms of state management. For many MQTT use cases, it also not that useful because the use case can often be modeled with QoS 1 messages and client-side deduplication.

Retained Messages

While publishing a message, a client can specify an additional parameter besides the topic, payload and QoS: the retain flag. If this feature is enabled on the broker and the retain flag is set, the message is stored on the given topic. This means that every client that subscribes to this topic will receive the last retained message that has been sent to this topic.

It is important to note that retained messages are kept independently of the sessions state of the sender, meaning that retained messages are kept even is session state is discarded.

In our chat application, we can use retained messages to display the online/offline status of a user. Whenever our client application connects to the broker successfully, we send a message of the following form on the topic chat/status/[username] using QoS 0 and the retain flag set:

{
    status: "online",
    lastOnlineTimestamp: DateTime.now().toISO(),
}

Whenever a user wants to disconnect by clicking the logout button, we switch our status to offline by publishing:

{
    status: "offline",
    lastOnlineTimestamp: DateTime.now().toISO(),
}

Now, whenever a client subscribes to the topic chat/status/+, they will receive the last status update of every client that is registered with the server. As we only care for the newest status information, we choose QoS 0 for the status messages. With a higher QoS the broker would queue the message for delivery for clients that are offline, which would then receive a boatload of outdated status updates when they reconnect. QoS 0 guarantees that older status updates are discarded when a subscriber is offline, and only the newest status update is delivered when they resubscribe.

Last Will And Testament (LWT)

A MQTT client can specify a last will as a part of the connect packet when connecting to the broker. This feature allows a client to request the broker to send one final message should the connection between the two be interrupted unexpectedly, for example by a client that crashes or a network outage. A last will consists of everything that is needed to send a normal message: a topic, a payload, a QoS level and the retain flag.

However, it is important to note that the LWT only triggers during abnormal disconnects. If a client disconnects in a graceful manner, the LWT will not be sent.

In our chat application we can use the LWT to make our status feature more robust. Currently, we cannot send a status update to set the status of a user to offline when a user just closes the browser tab. With LWT, we can fix this issue: during connection to the broker, we specify that an offline status update should be sent to the status topic, with the same parameters (QoS, retain) as we would use for a normal status update:

let will: IClientOptions["will"] | undefined = undefined
// ...
const willMessage: WireStatusMessage = {
  status: "offline",
  lastOnlineTimestamp: DateTime.now().toISO(),
}

will = {
  topic: `chat/status/${action.payload.credentials.username}`,
  payload: Buffer.from(JSON.stringify(willMessage)),
  qos: 0,
  retain: true,
}

The downside of this approach is that we cannot update the timestamp in the status message, as the LWT cannot be updated after it was initially set without disconnecting and reconnecting to the broker.

The demo application provides this feature set with the toggle use status messages, available via http://localhost:5173/?stage=30. Every chat should now display a status badge next to the name of the chat participant that updates whenever a user logs in or out of the app.

Authentication

In the MQTT ecosystem, two kinds of authentication are in widespread use: Password-based and certificate-based authentication.

The first one is defined in the MQTT standard: In the initial connect package, the client can include a password and a username. The problem with this approach is that both username and password are sent in plaintext, which becomes a security liability if as the transport layer is unencrypted too.

The second option, which is not standardized but de-facto supported on almost every broker, is to use client-certificates. This approach uses a feature of the TLS protocol (so it is only available if MQTT is running on top of TLS, in which case it is often called MQTTS), that requires the TLS client to present a certificate during the TLS handshake. Client certificates are more difficult to set up initially, but can save a lot of headache further down the line for two reasons. First, the TLS handshake uses a challenge mechanism to verify the client certificate, which means that no confidential data is exchanged in plaintext. Second, using client certificates relieves the broker operator from the burden of maintaining the list of username/password pairs at the MQTT broker or, especially in the web application context where user information might be kept in an external database, from synchronizing it with external data sources. Instead of checking the identity of each certificate individually, the broker can use the certificate's chain of trust to verify the validity of the certificate by checking it against the public key of a root certificate that was used to sign the client certificate.

Authorization

Until now, a client could just send messages to any MQTT topic, which means that they could easily impersonate another user. With authentication in place, we can now ensure that users can only send messages with themselves as the sender by defining authorization rules.

Unfortunately, authorization is not part of the MQTT standard. As stated by section 5.4.2 of the standard

An implementation may restrict access to Server resources based on information provided by the Client

That's all, which is why the following section is mosquitto specific. If you need authorization, check with the broker that you want to use before you start using it.

Mosquitto supports reading authorization rules from a file called acl_file. This file contains a list of rules of the following form

[pattern|topic] [read|write|readwrite|deny] [topic]

topic rules can use the same wildcard characters as subscription patterns. Rules on top of the file will be applied to anonymous clients. Rules can also be applied to single username by prefixing a block of rules with a user <username> line. patterns on the other hand can contain two additional wildcards %s and %u. %s is replaced by the session identifier of the client, while %u is replaced with the username that is used in the connect package. Patterns always apply to all clients.

To make sure that a user can only send messages with themselves as the sender, and subscribe only to topics with themselves as the target, we need two rules

# Patterns affect all clients
# Allow clients to read their own messages
pattern read chat/messages/%u/+
# Allow clients to send messages to other
# clients, as long as they use themselves as
# the sender
pattern write chat/messages/+/%u

The topics for the status feature can be protected in the same way:

# Patterns affect all clients
# Allow clients to write their own status
pattern write chat/status/%u
# Allow all clients to read other people's status
pattern read chat/status/+

To illustrate how to use special authorization rules for a single user, I've added a rule that allows the admin user to read all messages for compliance reasons:

# Allow the admin user to read all messages
user admin
topic read chat/messages/+/+

In the demo application, authorization and authentication can be enabled via the use authentication checkbox: http://localhost:5173/?stage=40. To be able to enforce the authorization rules correctly, this switch actually redirects all traffic to a second broker that has authentication enabled.

If we log in as the admin user (password: admin) we can now read the messages of all users, while they can only read the messages which are addressed to them:

With this feature, our simple chat application is completed.

Limitations and how to overcome them

If you are now considering using MQTT in your own application, you should be aware of a few common problems that arise in real-world applications that I did not include in my demo application.

Ad Hoc authorization

Mosquitto's static, topic-based authorization works well if the topics are known in advance, but falls short when the topics are created dynamically.

In our chat application, this problem arises when we want to implement chat groups consisting of multiple participants that can be added and removed dynamically. Let's assume that we assign an id to every chat room and use a topic of the form chat/room/[room_id] to distribute messages to the group.

As Mosquitto keeps the authorization rules in a file, we would need to update this file and reload the configuration whenever the membership of a chat room changes.

The Mosquitto ecosystem offers two plugins that solve this problem: The dynamic security plugin offers an API via a magic MQTT topic to update the authorization rules at runtime and introduces additional concepts such as roles and groups. These rules, however, are stored in a JSON file and require other applications to use MQTT as well or use the included cli application to update the rules.

Another option that is better suited for more complex applications is mosquitto-go-auth which offers the option to store the authorization rules in a database like postgres, redis, or mongoDB, which can be accessed and maintained by external applications.

Data Validation

One of the most common issues that arise when implementing MQTT in a real-life, user-facing application is the need to validate the messages that are sent to a specific topic. This is something that a typical MQTT broker cannot do out of the box, but there are common patterns to overcome this limitation.

Let's stick with our chat example: Assume that we maintain a blacklist of forbidden words that may not be sent via our application. Checking against this list only on the frontend side would allow a user to connect to our broker via a different MQTT client and send messages containing forbidden words anyways, so this solution won't cut it.

One Pattern that is, for example, employed by the AWS IoT Core, is to split the single chat topic into multiple smaller ones: chat/[receiver]/[sender]/request for message send requests, chat/[receiver]/[sender]/accepted for messages that pass the blacklist check and chat/[receiver]/[sender]/rejected for messages that don't. We now need to change the authorization rules as follows to allow message senders to publish new chat messages only to the /request topic and subscribe to the /rejected topic to receive feedback on their messages. Message receivers can subscribe to the /accepted topic to receive only those messages that have passed the blacklist check:

# Permissions for chat users
pattern read chat/messages/%u/+/accepted
pattern write chat/messages/+/%u/request
pattern write chat/messages/+/%u/rejected

# Permissions for the backend app
# that has to perform the blacklist check
user backend
topic read chat/messages/+/+/request
topic write chat/messages/+/+/accepted
topic write chat/messages/+/+/rejected

The check against the blacklist has to be performed by a custom-built worker, authorized by the username backend, running on a server somewhere. This worker subscribes to the /request topic and publishes the messages to the /accepted or /rejected topic, depending on the result of the check against the blacklist. It is important to note, however, that every message that is sent on any chat topic now has to pass through this worker. This might not be a problem when checking against a static blacklist, but depending on the complexity of the check (which might contain calls to a datastore), might create a bottleneck in our application.

Use Case Examples

To conclude this article I wanted to present a few more applications that could be implemented using MQTT, but didn't make the cut to be selected for my demo application:

Newsticker App

Publishes new articles to a hierarchy of topics that represent the different categories of the news, e.g. news/politics/us/utah,news/politics/europe, news/sport/football, news/sport/formula-1. Clients can subscribe to the categories that are most interesting to them either directly, or by using wildcards like news/sport/# to get all sport news.

Progress Reporting

Useful for long-running, asynchronous tasks that are performed by a worker. The worker publishes progress updates as retained messages to a topic that is unique to the task. This way, whenever a client subscribes to the progress topic, they would get the newest progress update immediately as well as all future updates.

Real Time Online Auctions

Auctioneers can publish new items to a topic that is unique to the auction. Clients can subscribe to the wildcard topic auctions/# to get all new items that are up for auction. Bids could be placed by addressing the item topic directly, e.g. auctions/1234/bid, but would have to be verified by using a backend worker as described in the data validation chapter to make sure that the bid is higher than the current highest bid. Successful bids would be published on the original auction topic as a retained message, so that all clients that are interested in the item would get the newest bid immediately.

About the header Image

Created with OpenAIs GPT-4o on 2024-11-15. Prompt:

Create an image of a cellular tower in a happy mood. Use a comic style. Do not show the faces of people.