MQTT on the Raspberry Pi and with Micropython on the ESP family - Part 1

Image1: MQTT with Raspi and ESP family

1. MQTT - Raspberry Pi - Mosquitto and Node-Red

After looking at TCP web servers and the UDP protocol in various blog posts (frost watchdog in the greenhouse 1 + 2; Violin meter, Mosquito test ) Today we take a close look at data exchange via MQTT. In a way, it combines the advantages of transfer via UDP with the data integrity of a TCP connection. The latter protocol is the basis for the "Message Queuing Telemetry Transport" protocol. This means that MQTT works with a secure connection. On the other hand, the frame for the transfer of measurement data to a central server, which is called a broker in this context, is as simple as under UDP. Each participant can send and receive at will. Stations that send data to the broker are called publishers. Clients that recall data from the broker is called subscriber or subscribers. As with UDP, you can easily combine both processes on one station. So that nothing gets messed up, there are subject areas under which you can publish or call up data, the so-called topics. But unlike UDP, not two network devices can talk directly to each other, but all transfers always run over the broker, i.e. the server. He collects data and distributes it again. However, the topics are not set on the server, but by the clients. A topic exists the moment a client publishes a message on this topic. This makes the system flexible and easy to care for.

Image 2: MQTT - Working method

Measurement value buyers or display units with an ESP32 or ESP8266 are used as clients. Image 1 shows a typical scenario. For example, the ESP32 provides temperature and moisture values ​​of a DHT22 and receives information from the broker on whether the heating is to be switched on. The ESP8266 has subscribed to the news about the temperature and the air humidity and receives the new values ​​from the broker as soon as they are available. Furthermore, the controller sends the switch to the broker under the topic heat.

A LAN or WLAN connection is a basic requirement. Naked ATMEGA328 Compatible microcontrollers are excluded in this topic alone because they cannot offer radio connection and cannot be programmable in this topic under Micropython. The variety of sensors that can dock on an ESP directly via ADC, I2C, UART, or SPI makes the ESP family ideally suited together with the modules based on Micropython as clients for MQTT.

However, ESPs are too narrow-breasted as brokers, so at least one Raspberry Pi has to serve. As a broker, we use the free software Mosquitto. It can be installed without problems on Linux machines, including on a Raspberry Pi. The Mosquitto server collects the incoming messages published by clients (publishers) under a topic and distributes them to the clients on request that have subscribed to these news issues (subscribers). The differences in the type of connection are shown in the following graphics.

Image 3: TCP - individual connections No ad hoc messages from the server

TCP builds on secure individual connections. HTTP clients can only be contacted by the server on request. The data integrity is guaranteed. Monitoring of data traffic is not possible.

Image 4: UDP - multiple compounds possible

Multiple contacts between the participants are possible under UDP. Each station can spontaneously replace data with any other. Shipping can be done with Broadcast to all participants, as well as specifically addressed individual stations. Data integrity is not ensured. Monitoring of data traffic is only possible if the corresponding broadcasting sets are included in the program of the transmission station.

Image 5: MQTT - targeted news shipping on request

MQTT puts on TCP and therefore offers secure connections. The data integrity is guaranteed. Each client can set up topics on the broker that other clients can subscribe to as desired. If a topic is subscribed to, the client receives the latest data on this topic when asked by the broker. Unsubscribed topics are not served by the broker to the requesting client. Monitoring of data traffic is very easy to subscribe to the corresponding topics.

So that we can listen to the traffic, we still need a tool with which we can register the broker, also as a subscriber. It is also easily possible for several clients to subscribe to the same topic. This is an advantage of MQTT compared to the other news protocols. We will first use this option in the Terminal on the Raspi. Later we will use the also free software Node-Red for this purpose. The program, which is also installed on the Raspi, has a graphical surface, the help of which can be easily built in a simple manner that can be used to graphically present the measurement results from the ESP measurement servants. By default, Node-Red is already included in the installation scope of the Raspberry Pi OS operating system and is usually installed via the software pool. But it also goes manually over the command line.

Let us summarize the hardware and software for our project.

Hardware:

1	Raspberry Pi Bundle with PI 1 B+, PI Pico and Accessories or
1	Raspberry Pi (1b+, 2, 2b, 3, 3b) ff.
1	SD card 8-16GB to match the selected Raspi
1	Breakboard
various	Jumper cable
1	Raspi power supply, 5V, 2.5A
1	Monitor (only to set up the Raspi)
1	Keyboard (only to set up the Raspi)
1	Mouse (only to set up the Raspi)
1	HDMI cable (only to set up the Raspi)
1	Network cable

software

Raspberry Pi Os-Image

Imager.exe Burning program for the SD-Card

Mosquitto broker

Node-Red

Software for the Windows machine

Xming

putty

Prepare the Raspi

Get Raspberry Pi OS

Raspberry Pi Os (formerly Raspian) is derived from Debian-Linux and is available in various releases. The version used here is suitable for all Raspi boards of versions 1b+, 2, 2b, 2b+, 3, and 3b.

Firmware releases with a different scale are offered here for download https://www.raspberrypi.com/software/operating-systems/#raspberry-pi-os-32-bit. I work in this post with a Raspberry 2b and an image of the program Imager is automatically downloaded and is very easy to install with just a few clicks. An SD card in category 10 with 8GB upwards is required. According to the setup, 45% are now available on my 8GB card.

Preparing the card is very easy. We invite Imager Down and start the file by double-clicking.

Image 6: download Raspberry Imager

Image 7: Install Imager

After a few seconds, the installation in folder C: \ Program Files (X86) \ Raspberry Pi Imager has been completed. We can start the application right away.

Image 8: Imager Start

First, an operating system must be selected -> Choose OS.

I selected the recommended system.

Image 9: Selection of the operating system

The SD card must be in the card slot and have been recognized by the system. Now comes the most dangerous part of the whole, the selection of the storage medium. This is dangerous because the content of the memory card is deleted. So you have to be sure that no important data is stored on the map. They would be irretrievably gone after the start of the burning process!

Image 10: Card selection - very easy here

We select the card and acknowledge the safety query with yes. The download of the firmware begins, and then the writing process starts, which, depending on the selected firmware, can take a good half an hour upwards.

Image 11: The system was successfully written

Next, we use an editor to create an empty file with the name SSH and copy it to the card. If Windows wants to format the card just created, leave that not to. Now we remove the card from the reader shaft and use it on the Raspi.

For further actions, it is favorable to temporarily connect a keyboard, mouse, and monitor on the Raspi. We also need a free IP address in the local network that we can assign to the Raspi. Because we want to operate servers, it is better to assign a fixed IP address to the Raspi. The easiest way to get there is over the Raspi itself. We can then do all further steps via an SSH connection from another computer. So we quickly connect the little one to the router via network cable.

Now we start the Raspi. After a few settings, we are on the Raspberry Pi Os -Desktop -Next, Set Country -> Germany, Next, Change Password -> Enter and confirm, Next, Update Software, Skip, Done.

With my mini-tft display it was now necessary to adapt the screen size.

Image 12: Raspi - Call up settings

Headless Resolution can also be set directly.

Image 13: Set Raspi screen

The activation/control of the SSH access is also in a dish so that we can access the Raspi from the network. The screen, keyboard, and mouse on the device then become obsolete. But - we are not yet that far.

Image 14: RASPI - Activate SSH

We end the configuration with OK.

We can do most of what is still to be done from the command line. To do this, we open a terminal with the key combination Ctrl + Alt + T. The standard image contains the editor nano that we start with the following command.

sudo nano /etc/dhcpcd.conf

Find the following lines in the lower part of the file dhcpcd.conf. Remove the "#" at the start of the line and set the values ​​according to your home network.

# Example Static IP Configuration:
#Interface ETH0
#Static IP_Address = 192.168.0.10/24
#Static IP6_Address = FD51: 42F8: Caae: D92E :: FF/64
#Static Routers = 192.168.0.1
#static domain_name_servers = 192.168.0.1 8.8.8.8 FD51: 42F8: Caae: D92E :: 1

We change this position in:

# Example Static IP Configuration:
Interface ETH0
static ip_address=10.0.1.99/24
static routers=10.0.1.25
static domain_name_servers=10.0.1.25

With Ctrl + O Save the change and with Ctrl + X End the editor. Restart the Raspi now.

After registration, we open a terminal again and check with the following command whether our changes have fertilized.

ifconfig

The output should look like this:

ETH0: flag=4163<Up, broadcast, running, multicast> mtu 1500
 inet 10.0.1.99 Netmask 255.255.255.0 Broadcast 10.0.1.255
 Inet6 Fe80 :: 9F0D: 6726: B163: 74C0 Prefixlen 64  Scopeid 0x20 
 inet6 2003: f3: 7722: 6200: D933: 6671: 26dB: 57ec prefixlen 64  Scopeid 0x0 

We also check immediately whether the network connection on the Internet works. Ping to an external URL or IP clarifies it.

Pi@Raspberrypi: ~ $ ping regensburg.de
Ping regensburg.de (62.116.156.60) 56(84) Bytes of Data.
64 Bytes from 60-156-116-62.rev.cstomer-net.de (62.116.156.60): ICMP_SEQ=1 TTL=59 time=15.8 MS
64 Bytes from 60-156-116-62.rev.cstomer-net.de (62.116.156.60): ICMP_SEQ=2 TTL=59 time=15.5 MS
64 Bytes from 60-156-116-62.rev.cstomer-net.de (62.116.156.60): ICMP_SEQ=3 TTL=59 time=15.3 MS

Abort with Ctrl + C, And if we are already there, we will also test the SSH access right away. We enter the following command in the terminal.

SSH local host

Image 15: Raspi - SSH test

We have to confirm the "attack" with "Yes" and then register with the password that we awarded above. We have now penetrated practically from our system through the homeier entrance without having left it beforehand. Ok - something like that only works on one PC. But if that worked out, it also works from another computer.

Install Mosquitto

The last action that we carry out from the command line is the installation of the Mosquitto broker.

sudo apt-get update

A lot of packages are loaded and installed, it takes a few minutes.

sudo apt install -y Mosquitto Mosquitto client

Again a number of packages are brought, unpacked, and furnished. Then we ensure that Mosquitto is started every time when the system is booted.

sudo Systemctl enable mosquitto.service

The following command tells us the version of the program and that the Mosquitto server lists on port 1883.

mosquitto -v

1638040611: Mosquitto version 2.0.11 Starting
1638040611: Using default config.
1638040611: Starting in Local only fashion. Connections wants to only be possible from clients running on this machine.
1638040611: Create A Configuration File which Definent A Listener To Allow Remote Access.
1638040611: For more details See https://mosquitto.org/documentation/authentication-methods/
1638040611: Opening IPV4 Lists Socket on Port 1883.
1638040611: Error: Address Already in use
1638040611: Opening IPV6 lists socket on port 1883.

We can confidently ignore the error message that the address is already being used (Error: Address Already in Use) because she only tells us that Mosquitto is already running.

In preparation for the next blog post, however, we should change a little something in the configuration of the Mosquitto-Deamon. If we were now trying to access the broker from another network device, such as our ESP32, we would get a rejection. This is because anonymous access, i.e. without user authentication and password, is prohibited by default. We want to change that, at least in the test phase.

So let's switch to the directory on the Raspi /etc/mosquitto/conf.d and create a file with the name there anonymous.conf.

CD /etc/mosquitto/conf.d
sudo nano anonymous.conf

Now we enter the following two lines, save the file and end nano.

listener 1883
Allow_anonymous true

Then we restart Mosquitto

systemctl remaining start mosquitto

background:

Each file in the directory /etc/mosquitto/conf.d With the ending .conf is understood as a configuration file and on the start of Mosquitto via the file /etc/mosquitto/mosquitto.conf called. So here we can accommodate our extensions for configuration.

Installation of Node-Red

The Node-Red package is now missing on the Raspi. Let's start the installation. I chose the way again over the command line, so you know what to do.

sudo apt install nodured

The installation takes a little longer again. Finally, the command and we can make sure that the node-Red teamon also starts up with the start of the system.

sudo Systemctl enable nodured.service

output:

Created symlink /etc/systemd/multi-user.target.wants/nodened.service → /lib/systemd/system/noded.service.

This ends the installation session on the Raspi. We restart the little one. Now that everything that is supposed to run on the Raspi does its service and works correctly, we can calm down the keyboard, monitor, and mouse from Raspi.

Software for the Windowskiste

During this, we get a terminal program on the Windows computer with which we can contact the Raspi via the network. Is widespread putty. We invite them to File for our system down. The EXE file can be run without installation. We save you in any directory and lie down on the desktop. With a few clicks, the terminal program is set up after the first start. Under Connection - X11 we switch X11 forwarding a.

Image 16: Putty - X11 Forwarding Switch on

Under session Let's enter the IP of our Raspis, the port stops at 22. Then we give the connection a speaking name and save the configuration.

Image 17: Putty - secure furnishings

With Open we open a terminal to the Raspi. It looks like the one we have already worked on the Raspi. Only the menu line is missing.

Image 18: Putty - terminal window

Let's test the Mosquitto and check whether it also stings. For this purpose, we need two terminal windows, each of which emulates a mosquitto client. So let's open a second terminal.

In the first terminal, we enter the following command to act as a subscriber. So this window should receive messages from the Brocker.

Image 19: Mosquitto - Subscriber lists

In the second window, we let a publisher send a message to the broker.

Image 20: Mosquitto - Publisher - Message is sent

With the sending of the message, it appears in the window of the subscriber.

Image 21: Mosquitto - Subscriber - Message has arrived

While Mosquitto is approached via the command line, Node-Red reports as a web interface in a browser. We also test this installation. Node-Red should have started automatically at the last boat process. So it should look roughly when we Node-Red with the IP of our Raspis and the Port 1880 Call up.

Image 22: Node-Red on the PC in Chrome

Now only a little something is missing for today. The Raspi no longer has a screen. To work on the command line, we can log in via Putty. But what do we do if we want or have to do something with the graphic surface of the Raspi? We activated the X11 forwarding when setting up Putty. We can use them if we install an X11 server on the Windowskiste. The responsible tool is called Xming And can from Sourceeforge-Net be downloaded. We start the setup with a double click, ignore Windows's message, and then follow the assistant with "Next".

Image 23: Xming - Setup

Image 24: Xming - destination

Image 25: Xming - component selection

Image 26: Xming - Select the start folder

Image 27: Xming - Set links

Image 28: Xming - Completion

From Windows Desktop we start the program by right-clicking on the Xming icon As an administrator. We have to allow the changes to the system, then Xming withdraws into the background. If we now open a Putty Terminal and write the following command on the command line, then the Thonny-IDE from our Raspi opens on the Windows screen after a few seconds.

Thonny & Disown

Thanks to the "& Disown" appendage, we can continue to use the terminal for other purposes. The terminal would be blocked without the addition.

We can also find out whether Python is running on Raspi.

python3

Output:

python 3.9.2 (Default, Mar 12 2021, 04:06:34)
[GCC 10.2.1 20210110] on Linux
Type "Help", "copyright", "Credits" or "License" for More information.
>>>

Lo and behold, it is already the latest version 3.9 that greets us friendly with the PythonPrompt. However, it must be said that it is not micropython but the adult version of Cpython. There are therefore a number of differences compared to the ESP32/ESP8266, regarding the scope of language, but also the hardware.

The basics for our MQTT project are hereby created. In the next post, we will set up an ESP32 / ESP8266 as a MQTT client. Due to the fantastic properties of Node-Red, it will later be an easy, appealing application for monitoring measuring points.

This blog post is also as PDF document accessible.