Node-RED

Node-RED is a flow-based development tool for visual programming developed by IBM. It is used to wire together hardware devices, APIs, and online services in new and interesting ways. It provides a browser-based editor that makes it easy to wire together flows using the wide range of nodes in the palette that can be deployed to its runtime in a single-click.

Designing Flows

The purpose of Node-RED is to create flows that can be used to process data streams. A flow is a sequence of nodes that are connected together to perform a specific task. Each node in the flow represents a function or service that can be used to process data. The nodes are connected together using wires that carry the data between them.

For example, LEAF adapters stream the measurements that is acquired from the adapters to various topics on the MQTT broker.

The HelloWorld example that is part of this setup submits a measurement every second that it is running.

Topic Structure

The topic structure is as follows:

leaf/<organisation>/<department>/<adapter_id>/<instance_id>/experiment/<experiment_id>/measurement/<measurement>

# Where <xxx> is a placeholder for the actual value.

This is all coordinated from the YAML file in combination with the adapter used. This is also the namespace per-department MQTT credentials are scoped to (leaf/<organisation>/<department>/#) - a device connecting with one of those credentials can only publish/subscribe within its own department’s slice of this structure.

Getting Started

This is the flow currently used in production. A single subscription covers the whole leaf/ namespace, and a set of lookup caches stay warm so incoming messages never block on a database round-trip:

• leaf/#: one MQTT-in node subscribed to the entire leaf/<organisation>/<department>/... namespace - see Topic Structure above.
• switch -> org/dept label fix: splits off heartbeat messages (to their own heartbeat debug node) from real data, then normalizes the organisation/department tags before the message reaches the transform step - reading them directly from the topic for leaf/... messages, falling back to institution-specific rules for anything else.
• EAV Transform: transforms the message into the EAV shape, feeding three places in parallel - the maple postgresql node, which runs the actual INSERT into sensor_data; a second postgresql node running the same dynamic query against a separate connection (see Database Access for why these are two distinct roles); and an EAV Debug node so you can inspect the generated INSERT statement.
• switch -> debug 23: filters that second query’s result down to non-empty responses for inspection.
• Cache-refresh group (“Triggered to collect mapper and organisation data”): three scheduled branches keep in-memory lookups fresh -
  • Every 5 min -> Org / dept lookup: refreshes the organisation/department -> UUID cache.
  • Every hour -> Translation lookup: refreshes metric/label translations.
  • Every hour -> Semantics lookup: refreshes semantic mapping rules.

Download the flow to import it directly into Node-RED (hamburger menu -> Import): leaf-example-flow.json.

A fresh install ships with exactly this: a single mqtt node subscribed to the whole leaf/# namespace, feeding straight into the EAV transform and insert, plus the cache-refresh group so lookups don’t hit the database on every message.

Semantic metric mapping

The Semantics lookup cache backs a resolveMetric() helper inside the EAV Transform function, which renames a raw metric key before it’s written to sensor_data. Each row of the mapper_semantics table (see TimescaleDB schema) is a key -> value rename, optionally scoped by department_id and/or entity. On every message, the first matching rule wins, in this order:

1. A rule scoped to this entity and department (entity is a comma-separated list; the message’s entity must appear in it).
2. A rule scoped to this department only (entity is null).
3. A global rule (department_id and entity both null).
4. No match - the raw metric key is kept as-is.

When a rename does apply, the original raw metric is preserved in tags.original_metric on the stored row, so nothing is lost - you can always trace a resolved metric name back to what the device actually reported.

This is what lets two devices report the same underlying measurement under different raw metric names (or the same raw name meaning different things in different departments) and still land in sensor_data under one consistent metric name for querying and dashboards.

Network Configuration

Node-RED is configured to operate on both the backend and frontend networks:

• Backend Network: Communication with TimescaleDB, VerneMQ, and other internal services
• Frontend Network: Web interface access for users
• Port: 1880 for the Node-RED web interface

Editor authentication

The Node-RED editor requires a login - write access to flows is equivalent to arbitrary code execution (any Function node runs whatever JavaScript you give it), so it’s never left open, even on internal-only networks.

Rather than a separate, standalone Node-RED credential, settings.js checks logins live against user_account - any existing LEAF superadmin can log in with their normal portal email and password, and nobody else can:

adminAuth: {
    type: "credentials",
    users: async function (username) { /* looks up user_account, checks is_superadmin */ },
    authenticate: async function (username, password) { /* bcryptjs.compare() against password_hash */ },
}

This connects using its own role (NODERED_AUTH_DB_USER/leaf_nodered_auth_user), column-limited to SELECT (email, password_hash, is_superadmin) ON user_account - it cannot read anything else, including other columns on that same table (totp_secret, etc.). See Database Access for the full picture of how this fits together with the portal and Grafana.

Current production status: leaf_nodered_auth_user exists with these exact grants already applied, but production is temporarily connecting through a maple stand-in role instead, pending a pg_hba.conf entry for the real role (a DB-admin action, not a deploy.sql change). The access described above is what the connection is scoped to become, not yet what’s live.

One practical consequence: because this checks the database live on every login attempt rather than caching or mirroring an account, the editor is unreachable if the database connection is down - there’s no local fallback credential.

Database Integration

Node-RED actually holds two separate database connections, not one: one moves sensor data, the other only ever checks a login - each runs under a different, unrelated access level, so it matters which one a problem actually traces back to.

	Purpose	Env vars	Access
Data processing	Moves sensor data: reads organisation/department/mapper_language/mapper_semantics for the lookup caches, writes to sensor_data	POSTGRES_USER/POSTGRES_PASSWORD (NODERED_DB_USER/PASSWORD in .env)	Full read/write
Editor auth	Checks editor logins (see above)	NODERED_AUTH_DB_USER/NODERED_AUTH_DB_PASSWORD	SELECT on 3 columns of user_account only

Both connect to:

• Database Host: timescaledb (internal Docker service name), or PGHOST for the auth connection specifically
• Database Port: 5432 (internal only)

This allows Node-RED flows to:

• Store processed data directly in TimescaleDB
• Read historical data for analysis
• Perform complex data transformations before storage

MQTT Integration

Node-RED is configured to connect to the VerneMQ broker:

• MQTT Host: vernemq (internal Docker service name)
• MQTT User: Configured through environment variables
• MQTT Password: Secure password managed through Docker secrets
• MQTT Port: 1883 (internal communication)

Resource Management

Node-RED is configured with resource limits to ensure stable operation:

• Memory Limit: 512MB maximum
• CPU Limit: 0.5 CPU cores
• Health Checks: Automated monitoring of Node-RED service health

GIT integration

Node-RED has a built-in GIT integration that allows you to version control your flows. This is done by connecting Node-RED to a GIT repository.

Git repository

First create a GIT repository on your favorite GIT hosting service. This can be GitHub, GitLab, Bitbucket, or any other service that supports GIT.

Configure Node-RED

• Click on the hamburger menu in the top right corner.
• Click on Git config.
• Set the git workflow to either:
  • Manual : All changes must be manually committed under the ‘history’ sidebar
  • Automatic: Changes are committed automatically with every deploy

SSH Key

If you want to use SSH authentication, you need to generate a private key in the SSH Keys section.

• Fill in a name
• Click on Generate Key
• Copy the public key and add it to your GIT repository as a deploy key.

Connect to repository

• Click on the hamburger menu in the top right corner.
• Click on Projects.
• Click on New.
• Click on Clone repository.
• Fill in your username and email address.
• Next
• Fill in the Project name
• Fill in the repository URL (SSH or HTTPS)
  • If you are using HTTPS, fill in your username and password.
  • If you are using SSH, select the SSH key you generated earlier.
• Click on Clone project
• Click on Done

Committing changes

If you have set the git workflow to Manual, you need to manually commit your changes. All git operations can be done from the Project History button in the top right corner.

When set to automatic workflow, all changes are committed automatically with every deploy. WARNING: Commit does not mean push. You still need to push the changes to the remote repository manually. Check the commit history for unpushed commits.

Advanced: Node-RED Palette

There are an amazing amount of nodes available in the palette that can be used to create complex flows. The nodes can be installed from the palette manager. Other nodes can be installed from the Node-RED library.

• Click on the hamburger menu in the top right corner.
• Click on Manage palette.
• Click on the Install tab.
• Search for the node you want to install.
• Click on Install

This way you can expand the functionality of Node-RED to suit your needs.

Troubleshooting

Common issues and solutions:

1. Node-RED not accessible: Check if the service is running with docker compose ps nodered
2. Can’t log into the editor: The login is your LEAF portal email/password, and only works for accounts with is_superadmin = true - a non-admin portal account cannot log in here. If the database is unreachable, the login itself will fail (see Editor authentication above).
3. MQTT connection failed: Verify VerneMQ is running and credentials are correct
4. Database connection issues: Check TimescaleDB status and database user credentials - remember there are two separate DB connections (data processing and editor auth), so confirm which one is actually failing from the logs.
5. High memory usage: Monitor flow complexity and optimize node usage

For detailed logs, use:

docker compose logs -f nodered