In this tutorial, we show you how to create a local MQTT test server using Eclipse Mosquitto for testing or use with our tutorials and TwilioQuest missions. You can run mosquitto in many environments, but launching it via Docker is one of the quickest paths.
After you complete this tutorial, you can run our broadband or Narrowbnad MQTT Quickstart substituting your server credentials.
MQTT is a publish/subscribe messaging protocol built on top of TCP/IP. It stands for Message Queuing Telemetry Transport.
Lightweight protocols like MQTT are perfect for both low bandwidth Narrowband and Broadband IoT connectivity. They help you mix higher powered and low powered devices in the same communications channels.
We'll get a local mosquitto instance up and running quickly using Docker on your machine or virtual host.
If you prefer, mosquitto is also available via
homebrew, or an installer package - see the mosquitto download page for more details on alternate installation methods.
The service we stand up here is unencrypted and NOT not suitable for production.
A production ready implementation of mosquitto includes:
- Strong username/password or PKI certificate credentials
- Perhaps an ACL customized to your needs
To make future tweaks to our Docker image config easier, we are going to create a directory to hold our mosquitto configuration:
To start up a very public, very lightly protected, mosquitto broker that you (soon anyone on the internet) can access using the default configuration on your local machine we will first set up a basic configuration file with a name of
connection_messages true allow_anonymous false password_file /mosquitto/config/mosquitto.passwd listener 1883 protocol mqtt listener 8000 protocol websockets
Next, create a simple plaintext password file (our connectivity is in plaintext – again, note this is not suitable for production).
Name this file
Please change both
password to something unique to you.
In a real-world deployment, we recommend running your password file through mosquitto's hashing function:
mosquitto_passwd -U mosquitto.passwd
to update the passwords to hashes. Note this command will not differentiate between cleartext and previously hashed passwords... so don't run it twice or the resulting string will be unusable.
To start the
eclipse-mosquitto docker image using this modified configuration file, follow the instructions for your operating system below.
On *NIX or OSX, you should run this command from the command line:
docker run -it -p 1883:1883 -p 8000:8000 -v $(pwd)/config:/mosquitto/config eclipse-mosquitto
Let's dig into the parameters we're passing to docker here:
-it runs docker interactively and allocates a pseudo-tty so you can see output. You can use
-d instead here to run the process completely in the background.
--name mosquitto specifies a name for our docker container to make referencing it easier later.
-p 1883:1883 and
-p 8000:8000 forward both 1883 (MQTT) and 8000 (MQTT over websockets) from the container's environment to the host computer. If you wish to use TLS to protect the communications you would replace the 1883 forward with
-p 8883:8883 and supply certificate information - see the mosquitto-tls man page or further in this tutorial.
-v $(pwd)/config:/mosquitto/config forwards our local
config directory to the container for the mosquitto.conf and mosquitto.password files.
This initial configuration does not enable mosquitto's ability to either write a log file or a persistent data storage location for cross-server-run topic subscriptions. As-is, if the server restarts any un-received messages sent to subscribed topics will be lost, as will any retained or will messages.
The final parameter,
eclipse-mosquitto tells docker we wish to use the eclipse-mosquitto image for the container.
If you are running this on a host with a publicly accessible IP address, your server will be up and available upon running this command.
If you are running it on your local network, you are probably behind a NAT gateway and will need port forwarding or a tool like ngrok before your address is publicly accessible.
If you have a local firewall running, you may have to explicitly allow connections to the port of your choosing (today we're using 1883 for cleartext MQTT and 8000 for cleartext Websocket MQTT first. Later in the tutorial we will use 8883.).
Before talking to the broker from your IoT device, let's check everything is working properly first.
You can subscribe or publish to your new MQTT broker using:
- HiveMQ's web based client or
- A local, command line MQTT client
Let's look at both now.
In this step, you can test your server using the HiveMQ websocket client even if you are only running the service locally. This is because the MQTT client connection comes from your browser, not HiveMQ.
- Change the
- Leave the port at
8000for websocket cleartext connection
- We recommend leaving the
ClientIDat the default random value. (ClientIDs must be unique in MQTT... collisions result in prior clients sharing the ClientID being disconnected.)
Passwordto the values you assigned in the
Now that we are connected, subscribe to a topic:
- Click on
Add New Topic Subscriptionunder
- Subscribe to
YOURRANDOMSTRING/#where YOURRANDOMSTRING is any alphanumeric string (with hyphens), for example
#indicates a wildcard for any child channels, for example
/sensors/basement/temperature, and so on.
- Click on the
Subscribebutton. Do not press
Enter/Returnon your keyboard, it will reload the page. The default QOS of
2is ok to leave.
Finally, we will publish a message to our topic:
Publish, ensure the
Topichas a topic name that matches the pattern you subscribed to such as
- Type a nice message to yourself in
You should see your message appear in the
Mosquitto distributes its own client applications as packages for many operating systems (details can be found on the mosquitto download page.)
The commands below assume you are running this in a docker image local to your computer. If you are running on a different host, substitute the correct IP address for
MQTT works by clients subscribing to "topics" and messages being published to those topics. We will first subscribe to a topic by running this command in a dedicated shell window:
mosquitto_sub -h localhost -p 1883 -u username -P password -t 'testtopic/#'
Then, in another window:
mosquitto_pub -h localhost -p 1883 -u username -P password -t 'testtopic/hello' -m 'Hello, world'
... and you should see your message!
Next, let's connect an IoT device to the broker.
If you are setting up this test server to experiment with one of Twilio's IoT Development Kits, you probably need to take one additional step. You need to make your broker publically accessible to allow the device to reach the MQTT broker over the internet.
If you are running this code on an internet connected host such as a VPS, you may already have access.
But, if you are running this locally, you probably need to use a proxy to tunnel TCP traffic to your local port 1883. One popular option is to use ngrok.
ngrok is typically used to forward web traffic, but we wish to forward a raw TCP port. Lucky for us,
ngrok supports this. Simply run:
ngrok tcp 1883
ngrok will provide you a new hostname and port number.
It's important to note that this port number will not be
1883. The hostname and port number ngrok returns are what you should plug into the hostname and port number in the IoT Quickstarts.
You can't use the port from above as the remote host for the HiveMQ websocket based MQTT client.
To enable this, run the following to set up a second tunnel to support a remote Websocket connection:
ngrok tcp 8000
The resulting host and port are what you should use to substitute in the web tool.
For production, you must build on top of encryption. The following sections show you how to set this up on the provided MQTT broker image.
In production, you need to secure communication when using MQTT. This will prevent others from eavesdropping or obtaining your credentials.
mosquitto's tls man page details how to create a self-signed certificate authority both for your server's certificate and (optionally) client certificates.
The CN for the server certificate you create must match the hostname provided to the MQTT client.
You can temporarily override this using the equivalent of
--insecure mode and disabling server hostname checking. This is only recommended for local testing; do not leave this on in production.
After creating the certificates above (or providing your own), place them into the
config/ directory in the following manner:
config/ca/ca.crt config/server.crt config/server.key
We're using a directory and
capath below instead of a single
cafile. This will help support Twilio Trust Onboard authentication in a later step.
To enable TLS communications with your server, edit the
mosquitto.conf and add the following lines:
listener 8883 0.0.0.0 capath /mosquitto/config/ca/ certfile /mosquitto/config/server.crt keyfile /mosquitto/config/server.key tls_version tlsv1.2
Now, restart the docker instance. On *NIX or OSX, use ctrl-c if you are running interactively, or
docker kill if in a container. This time, we will add port forwarding for MQTTS (8883):
docker run -it -p 8883:8883 -p 8000:8000 -v $(pwd)/config:/mosquitto/config eclipse-mosquitto
Docker requires Windows Hyper-V which is not available on Windows 10 Home Edition. You can check out mosquitto's website for a Windows installer instead.
This section is coming soon.
Certificate based authentication relies on two trust factors - the client trusting the server and the server trusting the client.
Server trust is established by the MQTT client trusting the issuer of the server's certificate. This is typically accomplished by including a trusted CA certificate or chain in your client.
Client trust is established by the client presenting a certificate signed by a trusted issuer known to the server. The server is configured with a list of trusted certificate issuers that client certificates are validated against.
If your client's certificates are generated by an authority you control, validating the certificate is issued by a trusted party may be sufficient for your needs. If the client certificates are issued by a third party that issues many certificates, you may have one additional step: authorizing specific certificates by common name or fingerprint.
To use the self-issued certificates mentioned in the mosquitto documentation above, you just need to require certificates and the
ca.crt you provided for the server's certificate will also apply for clients.
mosquitto.conf file add this under the
Please Note: Also should remove the
password_file directive from the
mosquitto.conf file as well. (Unless you want to require both a valid certificate and a username/password.)
If you are integrating Twilio Trust Onboard or are utilizing other client certificates from a shared provider, you will want to also authorize individual clients instead of trusting all certificates with a given chain. This prevents someone else's Trust Onboard SIM (or other shared certificate) from being implicitly trusted.
To trust a Twilio Trust Onboard client certificate, the server will need to trust the certificates that make up the intermediate issuer and root issuer.
(You can find these certificates here.)
mkdir config/ca pushd config/ca curl -O https://iot-cdn.twilio.com/trust_onboard/Twilio_Trust_Onboard_G2.pem curl -O https://iot-cdn.twilio.com/trust_onboard/Twilio_Trust_Onboard_G2_Available.pem curl -O https://iot-cdn.twilio.com/trust_onboard/Twilio_Trust_Onboard_G2_Signing.pem openssl rehash . popd
You will likely either use Available or Signing – not both – so you can choose which is the most appropriate for your use-case.
There is no harm in specifying both though. You'll need to explicitly allow individual clients in the ACL as described below.
openssl rehash . generates symbolic links using the SHA-1 fingerprint for quick lookup by the mosquitto process. If your install of the openssl client utils doesn't support the rehash command, you can try
c_rehash . instead. This is needed by some webservers to find the intermediates, so is an important step.
If the issuer of your client certificates also issues certificates for others (this is the case for Twilio Trust Onboard), you should also explicitly allow the certificates for your clients.
You have two choices, depending of how to map the certificate CN to a mosquitto username in your
will produce usernames of the form:
CN=WCXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,OU=Wireless,O=Twilio Inc.,L=San Francisco,ST=CA,C=US
will produce usernames of the form:
use_identity_as_username produces more easily managed usernames while
use_subject_as_username produces more fine-grained certificate information which may be useful for certificate issuers other than Twilio Trust Onboard.
For this tutorial, we will use
use_identity_as_username true to make the ACL more readable.
In either situation, you are relying on the trusted CA certificates you have configured to prevent spoofed certificates with a matching subject or identity from being allowed access.
The WC* SID is available by downloading your certificates in bulk on the download certificates page in the Twilio console.
The certificate SID is the
WC... portion of the file name, without the trailing
Setting up the ACL for many clients is best deferred to an auth_plugin within mosquitto. However, for our quick test we'll use mosquitto's standard ACL file for identities (mapped to usernames) instead.
In the sample below, we will give all authorized clients equal access to all topics in the system.
(You can easily restrict this in any way your use-case requires, for example a temperature sensor that can only public to its own endpoint:
topic write sensors/temperature/building_1 )
Create a file named
mosquitto.acl and populate it with your Trust Onboard SIM's certificate SIDs:
user WCXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX topic readwrite #
The easiest path to connecting to mosquitto using Trust Onboard is using the Trust Onboard SDK to obtain the
Available certificate and private key at runtime in your application.
You can then provide this certificate and key to mosquitto during client setup. If you are working with the command line mosquitto pub and sub tools, you can also use the
trust_onboard_tool CLI to instead export the certificate and private key to the file system for authentication.
Please keep in mind that writing the private key out to disk can be risky. If anyone else can access that disk, consider using the private key only in RAM (or a RAM disk) to reduce the chances of it leaking
Now you have an MQTT broker with authentication, and you are able to authenticate to it using a username and password or trusted identities. If you haven't yet connected an IoT Device via MQTT, you can try one of the MQTT quickstarts now (substitute the credentials for your publically accessible credentials):
Or, start to build up your solution using MQTT for messaging. The possibilities are endless, and we can't wait to see what you build.