Menu

Expand
Rate this page:

Thanks for rating this page!

We are always striving to improve our documentation quality, and your feedback is valuable to us. How could this documentation serve you better?

Programmable Wireless: How to Run a Test MQTT Broker

In this tutorial, we show you how to create a local MQTT test server using Eclipse Mosquitto for testing or use with our tutorials and TwilioQuest missions. You can run mosquitto in many environments, but launching it via Docker is one of the quickest paths.

After you complete this tutorial, you can run our broadband or Narrowbnad MQTT Quickstart substituting your server credentials.

What is MQTT?

MQTT is a publish/subscribe messaging protocol built on top of TCP/IP. It stands for Message Queuing Telemetry Transport.

Lightweight protocols like MQTT are perfect for both low bandwidth Narrowband and Broadband IoT connectivity. They help you mix higher powered and low powered devices in the same communications channels.

Sounds good, let's broker things.

Use a Docker Mosquitto Image

We'll get a local mosquitto instance up and running quickly using Docker on your machine or virtual host.

If you prefer, mosquitto is also available via apt, homebrew, or an installer package - see the mosquitto download page for more details on alternate installation methods.

The service we stand up here is unencrypted and NOT not suitable for production.

A production ready implementation of mosquitto includes:

  • TLS
  • Strong username/password or PKI certificate credentials
  • Perhaps an ACL customized to your needs

To make future tweaks to our Docker image config easier, we are going to create a directory to hold our mosquitto configuration:

mkdir config

To start up a very public, very lightly protected, mosquitto broker that you (soon anyone on the internet) can access using the default configuration on your local machine we will first set up a basic configuration file with a name of config/mosquitto.conf

connection_messages true

allow_anonymous false
password_file /mosquitto/config/mosquitto.passwd

listener 1883
protocol mqtt
listener 8000
protocol websockets

Next, create a simple plaintext password file (our connectivity is in plaintext – again, note this is not suitable for production).

Name this file config/mosquitto.passwd

username:password

Please change both username and password to something unique to you.

In a real-world deployment, we recommend running your password file through mosquitto's hashing function:

mosquitto_passwd -U mosquitto.passwd

to update the passwords to hashes. Note this command will not differentiate between cleartext and previously hashed passwords... so don't run it twice or the resulting string will be unusable.

To start the eclipse-mosquitto docker image using this modified configuration file, follow the instructions for your operating system below.

Mac OSX/*NIX

On *NIX or OSX, you should run this command from the command line:

docker run -it -p 1883:1883 -p 8000:8000 -v $(pwd)/config:/mosquitto/config eclipse-mosquitto

Let's dig into the parameters we're passing to docker here:

-it runs docker interactively and allocates a pseudo-tty so you can see output. You can use -d instead here to run the process completely in the background.

--name mosquitto specifies a name for our docker container to make referencing it easier later.

-p 1883:1883 and -p 8000:8000 forward both 1883 (MQTT) and 8000 (MQTT over websockets) from the container's environment to the host computer. If you wish to use TLS to protect the communications you would replace the 1883 forward with -p 8883:8883 and supply certificate information - see the mosquitto-tls man page or further in this tutorial.

-v $(pwd)/config:/mosquitto/config forwards our local config directory to the container for the mosquitto.conf and mosquitto.password files.

This initial configuration does not enable mosquitto's ability to either write a log file or a persistent data storage location for cross-server-run topic subscriptions. As-is, if the server restarts any un-received messages sent to subscribed topics will be lost, as will any retained or will messages.

The final parameter, eclipse-mosquitto tells docker we wish to use the eclipse-mosquitto image for the container.

If you are running this on a host with a publicly accessible IP address, your server will be up and available upon running this command.

If you are running it on your local network, you are probably behind a NAT gateway and will need port forwarding or a tool like ngrok before your address is publicly accessible.

If you have a local firewall running, you may have to explicitly allow connections to the port of your choosing (today we're using 1883 for cleartext MQTT and 8000 for cleartext Websocket MQTT first. Later in the tutorial we will use 8883.).

Windows 10 (Pro, Enterprise, or Educational)

Docker requires Windows Hyper-V which is not available on Windows 10 Home Edition. You can check out mosquitto's website for a Windows installer instead.

This section is coming soon.

Let's Connect!

Connect Locally to the Broker

Before talking to the broker from your IoT device, let's check everything is working properly first.

You can subscribe or publish to your new MQTT broker using:

Let's look at both now.

Using the HiveMQT Websocket Broker

In this step, you can test your server using the HiveMQ websocket client even if you are only running the service locally. This is because the MQTT client connection comes from your browser, not HiveMQ.

Visit HiveMQ's web based client

  1. Change the Host to localhost.
  2. Leave the port at 8000 for websocket cleartext connection
  3. We recommend leaving the ClientID at the default random value. (ClientIDs must be unique in MQTT... collisions result in prior clients sharing the ClientID being disconnected.)
  4. Set Username and Password to the values you assigned in the mosquitto.passwd file
  5. Click Connect

Now that we are connected, subscribe to a topic:

  1. Click on Add New Topic Subscription under Subscriptions
  2. Subscribe to YOURRANDOMSTRING/# where YOURRANDOMSTRING is any alphanumeric string (with hyphens), for example twilio-loves-mqtt/#. the # indicates a wildcard for any child channels, for example twilio-loves-mqtt, twilio-loves-mqtt/1, twilio-loves-mqtt/sensors/basement/temperature, and so on.
  3. Click on the Subscribe button. Do not press Enter/Return on your keyboard, it will reload the page. The default QOS of 2 is ok to leave.

Finally, we will publish a message to our topic:

  • Under Publish, ensure the Topic has a topic name that matches the pattern you subscribed to such as YOURRANDOMSTRING/data
  • Type a nice message to yourself in Message
  • Click Publish

You should see your message appear in the Messages section!

Use a Command Line Client

Mosquitto distributes its own client applications as packages for many operating systems (details can be found on the mosquitto download page.)

The commands below assume you are running this in a docker image local to your computer. If you are running on a different host, substitute the correct IP address for 127.0.0.1.

MQTT works by clients subscribing to "topics" and messages being published to those topics. We will first subscribe to a topic by running this command in a dedicated shell window:

mosquitto_sub -h localhost -p 1883 -u username -P password -t 'testtopic/#'

Then, in another window:

mosquitto_pub -h localhost -p 1883 -u username -P password -t 'testtopic/hello' -m 'Hello, world'

... and you should see your message!

Next, let's connect an IoT device to the broker.

Enable remote connections

Allow remote connections to your MQTT Broker

If you are setting up this test server to experiment with one of Twilio's IoT Development Kits, you probably need to take one additional step. You need to make your broker publically accessible to allow the device to reach the MQTT broker over the internet.

If you are running this code on an internet connected host such as a VPS, you may already have access.

But, if you are running this locally, you probably need to use a proxy to tunnel TCP traffic to your local port 1883. One popular option is to use ngrok.

ngrok is typically used to forward web traffic, but we wish to forward a raw TCP port. Lucky for us, ngrok supports this. Simply run:

ngrok tcp 1883

ngrok will provide you a new hostname and port number.

It's important to note that this port number will not be 1883. The hostname and port number ngrok returns are what you should plug into the hostname and port number in the IoT Quickstarts.

Use ngrok to forward a websocket connection

You can't use the port from above as the remote host for the HiveMQ websocket based MQTT client.

To enable this, run the following to set up a second tunnel to support a remote Websocket connection:

ngrok tcp 8000

The resulting host and port are what you should use to substitute in the web tool.

Let's add encryption!

If you were setting up the broker for the Broadband or Narrowband MQTT Quickstart, feel free to visit there now.

For production, you must build on top of encryption. The following sections show you how to set this up on the provided MQTT broker image.

Add encryption to the MQTT Broker

In production, you need to secure communication when using MQTT. This will prevent others from eavesdropping or obtaining your credentials.

mosquitto's tls man page details how to create a self-signed certificate authority both for your server's certificate and (optionally) client certificates.

The CN for the server certificate you create must match the hostname provided to the MQTT client.

You can temporarily override this using the equivalent of mosquitto_pub and mosquitto_sub's --insecure mode and disabling server hostname checking. This is only recommended for local testing; do not leave this on in production.

After creating the certificates above (or providing your own), place them into the config/ directory in the following manner:

config/ca/ca.crt
config/server.crt
config/server.key

We're using a directory and capath below instead of a single cafile. This will help support Twilio Trust Onboard authentication in a later step.

To enable TLS communications with your server, edit the mosquitto.conf and add the following lines:

listener 8883 0.0.0.0
capath /mosquitto/config/ca/
certfile /mosquitto/config/server.crt
keyfile /mosquitto/config/server.key
tls_version tlsv1.2

Now, restart the docker instance. On *NIX or OSX, use ctrl-c if you are running interactively, or docker kill if in a container. This time, we will add port forwarding for MQTTS (8883):

Mac OSX/*NIX

docker run -it -p 8883:8883 -p 8000:8000 -v $(pwd)/config:/mosquitto/config eclipse-mosquitto

Windows 10 (Pro, Enterprise, or Educational)

Docker requires Windows Hyper-V which is not available on Windows 10 Home Edition. You can check out mosquitto's website for a Windows installer instead.

This section is coming soon.

Let's explore certificate authentication

About Certificate Authentication

Certificate based authentication relies on two trust factors - the client trusting the server and the server trusting the client.

Server trust is established by the MQTT client trusting the issuer of the server's certificate. This is typically accomplished by including a trusted CA certificate or chain in your client.

Client trust is established by the client presenting a certificate signed by a trusted issuer known to the server. The server is configured with a list of trusted certificate issuers that client certificates are validated against.

If your client's certificates are generated by an authority you control, validating the certificate is issued by a trusted party may be sufficient for your needs. If the client certificates are issued by a third party that issues many certificates, you may have one additional step: authorizing specific certificates by common name or fingerprint.

Self-Generated Certificate Authentication

To use the self-issued certificates mentioned in the mosquitto documentation above, you just need to require certificates and the ca.crt you provided for the server's certificate will also apply for clients.

In your mosquitto.conf file add this under the capath, certfile, keyfile parameters:

require_certificate true

Please Note: Also should remove the password_file directive from the mosquitto.conf file as well. (Unless you want to require both a valid certificate and a username/password.)

Integrate Twilio Trust Onboard

Twilio Trust Onboard

If you are integrating Twilio Trust Onboard or are utilizing other client certificates from a shared provider, you will want to also authorize individual clients instead of trusting all certificates with a given chain. This prevents someone else's Trust Onboard SIM (or other shared certificate) from being implicitly trusted.

To trust a Twilio Trust Onboard client certificate, the server will need to trust the certificates that make up the intermediate issuer and root issuer.

(You can find these certificates here.)

mkdir config/ca
pushd config/ca
curl -O https://iot-cdn.twilio.com/trust_onboard/Twilio_Trust_Onboard_G2.pem
curl -O https://iot-cdn.twilio.com/trust_onboard/Twilio_Trust_Onboard_G2_Available.pem
curl -O https://iot-cdn.twilio.com/trust_onboard/Twilio_Trust_Onboard_G2_Signing.pem
openssl rehash .
popd

You will likely either use Available or Signing – not both – so you can choose which is the most appropriate for your use-case.

There is no harm in specifying both though. You'll need to explicitly allow individual clients in the ACL as described below.

openssl rehash . generates symbolic links using the SHA-1 fingerprint for quick lookup by the mosquitto process. If your install of the openssl client utils doesn't support the rehash command, you can try c_rehash . instead. This is needed by some webservers to find the intermediates, so is an important step.

If the issuer of your client certificates also issues certificates for others (this is the case for Twilio Trust Onboard), you should also explicitly allow the certificates for your clients.

You have two choices, depending of how to map the certificate CN to a mosquitto username in your mosquitto.conf file:

use_subject_as_username true

will produce usernames of the form: CN=WCXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,OU=Wireless,O=Twilio Inc.,L=San Francisco,ST=CA,C=US

while:

use_identity_as_username true

will produce usernames of the form: WCXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

use_identity_as_username produces more easily managed usernames while use_subject_as_username produces more fine-grained certificate information which may be useful for certificate issuers other than Twilio Trust Onboard.

Generate usernames for the tutorial

For this tutorial, we will use use_identity_as_username true to make the ACL more readable.

In either situation, you are relying on the trusted CA certificates you have configured to prevent spoofed certificates with a matching subject or identity from being allowed access.

The WC* SID is available by downloading your certificates in bulk on the download certificates page in the Twilio console.

The certificate SID is the WC... portion of the file name, without the trailing -signing.pem or -available.pem.

Setting up the ACL for many clients is best deferred to an auth_plugin within mosquitto. However, for our quick test we'll use mosquitto's standard ACL file for identities (mapped to usernames) instead.

In the sample below, we will give all authorized clients equal access to all topics in the system.

(You can easily restrict this in any way your use-case requires, for example a temperature sensor that can only public to its own endpoint: topic write sensors/temperature/building_1 )

Create a file named mosquitto.acl and populate it with your Trust Onboard SIM's certificate SIDs:

user WCXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
topic readwrite #
Let's authenticate using Trust Onboard

Authenticating with mosquitto Using Twilio Trust Onboard

The easiest path to connecting to mosquitto using Trust Onboard is using the Trust Onboard SDK to obtain the Available certificate and private key at runtime in your application.

You can then provide this certificate and key to mosquitto during client setup. If you are working with the command line mosquitto pub and sub tools, you can also use the trust_onboard_tool CLI to instead export the certificate and private key to the file system for authentication.

Please keep in mind that writing the private key out to disk can be risky. If anyone else can access that disk, consider using the private key only in RAM (or a RAM disk) to reduce the chances of it leaking

Next

Now you have an MQTT broker with authentication, and you are able to authenticate to it using a username and password or trusted identities. If you haven't yet connected an IoT Device via MQTT, you can try one of the MQTT quickstarts now (substitute the credentials for your publically accessible credentials):

Or, start to build up your solution using MQTT for messaging. The possibilities are endless, and we can't wait to see what you build.

Suggested Content

Rate this page:

Need some help?

We all do sometimes; code is hard. Get help now from our support team, or lean on the wisdom of the crowd browsing the Twilio tag on Stack Overflow.