Node-RED

Node-RED is a flow-based development tool for visual programming developed by IBM. It is used to wire together hardware devices, APIs, and online services in new and interesting ways. It provides a browser-based editor that makes it easy to wire together flows using the wide range of nodes in the palette that can be deployed to its runtime in a single-click.

Designing Flows

The purpose of Node-RED is to create flows that can be used to process data streams. A flow is a sequence of nodes that are connected together to perform a specific task. Each node in the flow represents a function or service that can be used to process data. The nodes are connected together using wires that carry the data between them.

For example, LEAF adapters stream the measurements that is acquired from the adapters to various topics on the MQTT broker.

The HelloWorld example that is part of this setup submits a measurement every second that it is running.

The topic structure is as follows:

<organisation>/<department>/<adapter_id>/<instance_id>/experiment/<experiment_id>/measurement/<measurement>

# Where <xxx> is a placeholder for the actual value.

This is all coordinated from the YAML file in combination with the adapter used.

Getting Started

This is the flow currently used in production. It listens on multiple MQTT brokers and keeps a set of lookup caches warm so incoming messages never block on a database round-trip:

• unlock/#, wur/#, ssb/#: separate MQTT-in nodes, one per institution/deployment, each subscribed to its own topic prefix.
• switch → org/dept label fix: normalizes the organisation/department tags before the message reaches the transform step, falling back to the topic prefix when a message doesn’t already carry them.
• EAV Transform → postgresql: transforms the message into the EAV shape and inserts it into sensor_data, feeding an EAV Debug node so you can inspect the generated INSERT statement.
• Cache-refresh group (“Triggered to collect mapper and organisation data”): three scheduled branches keep in-memory lookups fresh —
  • Every 5 min → Org / dept lookup: refreshes the organisation/department → UUID cache.
  • Every hour → Translation lookup: refreshes metric/label translations.
  • Every hour → Semantics lookup: refreshes semantic mapping rules.
• heartbeat: a periodic debug node used to confirm the flow is still alive.

Download the flow to import it directly into Node-RED (hamburger menu → Import): leaf-example-flow.json.

A fresh install ships with a simpler starting point: a single mqtt node subscribed to all topics (#), feeding straight into the EAV transform and insert. Grow into the multi-broker flow above once you’re onboarding more than one institution/department and want cached lookups instead of a query per message.

Semantic metric mapping

The Semantics lookup cache backs a resolveMetric() helper inside the EAV Transform function, which renames a raw metric key before it’s written to sensor_data. Each row of the mapper_semantics table (see TimescaleDB schema) is a key → value rename, optionally scoped by department_id and/or entity. On every message, the first matching rule wins, in this order:

1. A rule scoped to this entity and department (entity is a comma-separated list; the message’s entity must appear in it).
2. A rule scoped to this department only (entity is null).
3. A global rule (department_id and entity both null).
4. No match — the raw metric key is kept as-is.

When a rename does apply, the original raw metric is preserved in tags.original_metric on the stored row, so nothing is lost — you can always trace a resolved metric name back to what the device actually reported.

This is what lets two devices report the same underlying measurement under different raw metric names (or the same raw name meaning different things in different departments) and still land in sensor_data under one consistent metric name for querying and dashboards.

Network Configuration

Node-RED is configured to operate on both the backend and frontend networks:

• Backend Network: Communication with TimescaleDB, VerneMQ, and other internal services
• Frontend Network: Web interface access for users
• Port: 1880 for the Node-RED web interface

Database Integration

Node-RED is configured with dedicated database access to TimescaleDB:

• Database User: nodered_user (configured in .env)
• Database Password: Secure password generated during setup
• Database Host: timescaledb (internal Docker service name)
• Database Port: 5432 (internal only)

This allows Node-RED flows to:

• Store processed data directly in TimescaleDB
• Read historical data for analysis
• Perform complex data transformations before storage

MQTT Integration

Node-RED is configured to connect to the VerneMQ broker:

• MQTT Host: vernemq (internal Docker service name)
• MQTT User: Configured through environment variables
• MQTT Password: Secure password managed through Docker secrets
• MQTT Port: 1883 (internal communication)

Resource Management

Node-RED is configured with resource limits to ensure stable operation:

• Memory Limit: 512MB maximum
• CPU Limit: 0.5 CPU cores
• Health Checks: Automated monitoring of Node-RED service health

GIT integration

Node-RED has a built-in GIT integration that allows you to version control your flows. This is done by connecting Node-RED to a GIT repository.

Git repository

First create a GIT repository on your favorite GIT hosting service. This can be GitHub, GitLab, Bitbucket, or any other service that supports GIT.

Configure Node-RED

• Click on the hamburger menu in the top right corner.
• Click on Git config.
• Set the git workflow to either:
  • Manual : All changes must be manually committed under the ‘history’ sidebar
  • Automatic: Changes are committed automatically with every deploy

SSH Key

If you want to use SSH authentication, you need to generate a private key in the SSH Keys section.

• Fill in a name
• Click on Generate Key
• Copy the public key and add it to your GIT repository as a deploy key.

Connect to repository

• Click on the hamburger menu in the top right corner.
• Click on Projects.
• Click on New.
• Click on Clone repository.
• Fill in your username and email address.
• Next
• Fill in the Project name
• Fill in the repository URL (SSH or HTTPS)
  • If you are using HTTPS, fill in your username and password.
  • If you are using SSH, select the SSH key you generated earlier.
• Click on Clone project
• Click on Done

Committing changes

If you have set the git workflow to Manual, you need to manually commit your changes. All git operations can be done from the Project History button in the top right corner.

When set to automatic workflow, all changes are committed automatically with every deploy. WARNING: Commit does not mean push. You still need to push the changes to the remote repository manually. Check the commit history for unpushed commits.

Advanced: Node-RED Palette

There are an amazing amount of nodes available in the palette that can be used to create complex flows. The nodes can be installed from the palette manager. Other nodes can be installed from the Node-RED library.

• Click on the hamburger menu in the top right corner.
• Click on Manage palette.
• Click on the Install tab.
• Search for the node you want to install.
• Click on Install

This way you can expand the functionality of Node-RED to suit your needs.

Troubleshooting

Common issues and solutions:

1. Node-RED not accessible: Check if the service is running with docker compose ps nodered
2. MQTT connection failed: Verify VerneMQ is running and credentials are correct
3. Database connection issues: Check TimescaleDB status and database user credentials
4. High memory usage: Monitor flow complexity and optimize node usage

For detailed logs, use:

docker compose logs -f nodered