Trendz integration #84

RomenKost · 2025-12-04T12:36:48Z

This code chages allow to run Trendz in the same docker compose cluster as ThingsBoard.

To enable Trendz, it's necessary:

Set TRENDZ_ENABLED=true

create and check log folders for Trendz

Run ./docker-install-trendz.sh
Trendz Database will be created.

Start all services

Login to ThingsBoard as sysadmin to check Trendz sync status

Trendz logs have ThingsBoard style:

Stop all services:

Trendz can be updated with ./docker-upgrade-trendz.sh

This reverts commit d3c627f.

# Conflicts: # .env

smatvienko-tb

Hi! You can find my comments below:

smatvienko-tb · 2026-01-19T15:31:08Z

trendz-python-executor/conf/python-executor.conf

+export JAVA_OPTS="$JAVA_OPTS -XX:+CMSEdenChunksRecordAlways -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly"
+
+export JAVA_OPTS="$JAVA_OPTS -Xms512m -Xmx512m"
+export JAVA_OPTS="$JAVA_OPTS -Xss1m"


I never saw such a small stack . are you really sure that is up and running ?

Removed stack settigns from .conf file.

smatvienko-tb · 2026-01-19T15:36:05Z

trendz-python-executor/conf/python-executor.conf

+export LOADER_PATH=/usr/share/python-executor/conf
+
+export JAVA_OPTS="$JAVA_OPTS -Dplatform=deb"
+export JAVA_OPTS="$JAVA_OPTS -Xlog:gc=debug:file=/var/log/python-executor/gc.log -XX:+IgnoreUnrecognizedVMOptions -XX:+PrintGCDateStamps"


I believe you can drop all those tunings with such a small heap size.

Kept simular tuning as ThingsBoard services

smatvienko-tb · 2026-01-19T15:37:39Z

trendz/conf/logback.xml

+    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
+        <encoder>
+            <Pattern>
+                %green(%d{yyyy-MM-dd} | %d{HH:mm:ss.SSS}) | %highlight(%-5level) [%cyan(%-40thread)] %-60yellow(%C{1}): %msg %n%throwable


please, stick to the ThingsBoard log format. It should be standard for log sinking

Sticked to the ThingsBoard log format.

smatvienko-tb · 2026-01-19T15:38:16Z

trendz/conf/logback.xml

+    </appender>
+
+    <logger name="org.thingsboard.trendz" level="INFO" />
+    <logger name="org.thingsboard.trendz.service.task.TaskRuntimeService" level="DEBUG" />


no debug in production!!! comment out all debug by default

Commented DEBUG logs

smatvienko-tb · 2026-01-19T15:41:39Z

advanced/docker-compose.trendz.yml

+      - ../trendz-python-executor/data:/data
+  trendz-postgres:
+    restart: always
+    image: "postgres:16"


the latest is 17

We set postgres 16, because this version used in docker-compose.postgres.yml. But we can use the latest version if that's necessary

smatvienko-tb · 2026-01-19T15:56:03Z

basic/docker-compose.trendz.yml

+      - ../trendz-python-executor/conf:/python-executor-config-files
+      - ../trendz-python-executor/log:/var/log/python-executor
+      - ../trendz-python-executor/data:/data
+  trendz-postgres:


thingsboard postgresql is already deployed. Why we need a dedicated postgresql instance for Trendz. It might be enough to add another database within the same postgres. backups and restores might be easier. Isolation is fine for me, but as you even use a postgres/postgres credentials I consider your docker compose requires tuning before production

Replaced trendz-postgres with postgres service. Added trendz-db-init service to be able create Trendz database in the same postgres service as ThingsBoard database.

smatvienko-tb · 2026-01-19T16:01:13Z

trendz/conf/trendz.conf

+export JAVA_OPTS="$JAVA_OPTS -XX:+CMSEdenChunksRecordAlways -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly"
+
+export JAVA_OPTS="$JAVA_OPTS -Xms2000m -Xmx2000m"
+export JAVA_OPTS="$JAVA_OPTS -Xss1m"


1m for stack for reactive Java Trendz ? Are you really run the app with this settings or it appears only in this PR at first time?

Removed all memory settings from .conf file

smatvienko-tb · 2026-01-19T16:01:27Z

trendz/conf/trendz.conf

+export JAVA_OPTS="$JAVA_OPTS -Xss1m"
+export JAVA_OPTS="$JAVA_OPTS -XX:MaxMetaspaceSize=256m"
+export JAVA_OPTS="$JAVA_OPTS -XX:MaxDirectMemorySize=256m"
+export JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError"


no location specified

Specified folders for heap dump

smatvienko-tb · 2026-01-19T16:02:20Z

trendz/conf/trendz.conf

+export JAVA_OPTS="$JAVA_OPTS -Dplatform=deb -Dinstall.data_dir=/usr/share/trendz/data"
+export JAVA_OPTS="$JAVA_OPTS -Xlog:gc=debug:file=/var/log/trendz/gc.log -XX:+IgnoreUnrecognizedVMOptions  -XX:+PrintGCDateStamps"
+export JAVA_OPTS="$JAVA_OPTS -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10"
+export JAVA_OPTS="$JAVA_OPTS -XX:GCLogFileSize=10M -XX:-UseBiasedLocking -XX:+UseTLAB -XX:+ResizeTLAB -XX:+PerfDisableSharedMem -XX:+UseCondCardMark"


UseBiasedLocking are likely deprecated /removed

Removed UseBiasedLocking from .conf files

smatvienko-tb · 2026-01-19T16:03:39Z

.env

 EDQS_ENABLED=false

 TB_VERSION=4.2.1.1PE
+TRENDZ_VERSION=1.14.0


I believe the version is not the latest, please, update if required.

Updated to 1.15.0.2

# Conflicts: # .env

Copilot

Pull request overview

This PR integrates Trendz Analytics as an optional add-on to the ThingsBoard deployment system. Trendz is a data analytics platform that works alongside ThingsBoard, consisting of two services: the main Trendz application and a Python executor for script execution.

Changes:

Added Trendz and Trendz Python Executor service configurations and Docker Compose files for all deployment types (basic, monolith, advanced)
Integrated Trendz routing into HAProxy configurations across all deployments
Updated all deployment management scripts to support conditional Trendz deployment via TRENDZ_ENABLED flag
Added database initialization script and Docker Compose configuration for Trendz

Reviewed changes

Copilot reviewed 28 out of 29 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
trendz/conf/trendz.conf	Java runtime configuration for Trendz service
trendz/conf/logback.xml	Logging configuration for Trendz service
trendz.env	Environment variables for Trendz service
trendz-python-executor/conf/python-executor.conf	Java runtime configuration for Python executor
trendz-python-executor/conf/logback.xml	Logging configuration for Python executor
trendz-python-executor.env	Environment variables for Python executor
trendz-db-init.sh	Database initialization script for Trendz
tb-node.trendz.env	ThingsBoard configuration when Trendz is enabled
monolith/haproxy/config/haproxy.cfg	HAProxy routing rules for Trendz endpoints
monolith/docker-compose.trendz.yml	Docker Compose for Trendz in monolith deployment
monolith/docker-compose.trendz.db-init.yml	Database init for Trendz in monolith deployment
basic/haproxy/config/haproxy.cfg	HAProxy routing rules for Trendz in basic deployment
basic/docker-compose.trendz.yml	Docker Compose for Trendz in basic deployment
basic/docker-compose.trendz.db-init.yml	Database init for Trendz in basic deployment
advanced/haproxy/config/haproxy.cfg	HAProxy routing rules for Trendz in advanced deployment
advanced/docker-compose.trendz.yml	Docker Compose for Trendz in advanced deployment
advanced/docker-compose.trendz.volumes.yml	External volume configuration for Trendz
advanced/docker-compose.trendz.db-init.yml	Database init for Trendz in advanced deployment
docker-upgrade-trendz.sh	Script to upgrade Trendz service
docker-install-trendz.sh	Script to install and initialize Trendz
docker-upgrade-tb.sh	Updated to include Trendz compose args
docker-update-service.sh	Updated to include Trendz compose args
docker-stop-services.sh	Updated to include Trendz compose args
docker-start-services.sh	Updated to include Trendz compose args
docker-remove-services.sh	Updated to include Trendz compose args
docker-install-tb.sh	Updated to include Trendz compose args
compose-utils.sh	Added Trendz helper functions and permissions
.gitignore	Added Trendz and Python executor directories
.env	Added Trendz configuration variables

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-28T10:50:47Z

basic/docker-compose.trendz.yml

+    env_file:
+      - ../trendz-python-executor.env
+    volumes:
+      - ../trendz-python-executor/conf:/python-executor-config-files


Missing log volume mount: the trendz-python-executor service is missing the log volume mount. This is inconsistent with the monolith deployment (monolith/docker-compose.trendz.yml:63) which includes '../trendz-python-executor/log:/var/log/python-executor'. This will cause logs to be written to the container's filesystem instead of the host.

Suggested change

- ../trendz-python-executor/conf:/python-executor-config-files

- ../trendz-python-executor/conf:/python-executor-config-files

- ../trendz-python-executor/log:/var/log/python-executor

Removed /trendz-python-executor/log

Copilot · 2026-01-28T10:50:48Z

compose-utils.sh


+    if [ "$TRENDZ_ENABLED" = true ]; then
+      PERMISSION_LIST="$PERMISSION_LIST
+      799  799  trendz/log


Missing permission entry for trendz-python-executor/log: The permission list includes 'trendz/log' but does not include 'trendz-python-executor/log'. Based on the docker-compose files that mount '../trendz-python-executor/log:/var/log/python-executor', this directory should also have permissions set. Consider adding '799 799 trendz-python-executor/log' to maintain consistency with other service log directories.

Suggested change

799 799 trendz/log

799 799 trendz/log

799 799 trendz-python-executor/log

Removed /trendz-python-executor/log

Copilot · 2026-01-28T10:50:48Z

trendz.env

+SCRIPT_ENGINE_DOCKER_PROVIDER_URL=trendz-python-executor:8181
+SCRIPT_ENGINE_TIMEOUT=30000
+
+TB_API_URL: http://haproxy:80


Environment variable syntax error: this line uses a colon instead of an equals sign. Environment variable assignments should use '=' not ':'. This should be 'TB_API_URL=http://haproxy:80'.

Suggested change

TB_API_URL: http://haproxy:80

TB_API_URL=http://haproxy:80

Copilot · 2026-01-28T10:50:48Z

trendz.env

@@ -0,0 +1,10 @@
+# Python executor service configuration


Misleading comment: The comment states "Python executor service configuration", but this file (trendz.env) contains configuration for the Trendz service itself, not just the Python executor. The comment should be updated to reflect that this is the main Trendz service configuration.

Suggested change

# Python executor service configuration

# Trendz service configuration

Copilot

Pull request overview

Copilot reviewed 28 out of 29 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-28T15:44:56Z

docker-upgrade-trendz.sh

+#
+# ThingsBoard, Inc. ("COMPANY") CONFIDENTIAL
+#
+# Copyright © 2016-2022 ThingsBoard, Inc. All Rights Reserved.


The copyright year in docker-upgrade-trendz.sh is inconsistent with the newly added files. Other new Trendz files use "Copyright © 2016-2026" while this file uses "Copyright © 2016-2022". This should be updated to match the other new Trendz-related files for consistency.

Fixed license header

Copilot · 2026-01-28T15:44:56Z

basic/docker-compose.trendz.yml

+      - ../trendz/data:/data
+    depends_on:
+      - postgres
+      - tb-monolith


The trendz service is configured with SCRIPT_ENGINE_DOCKER_PROVIDER_URL pointing to trendz-python-executor:8181, but trendz-python-executor is not listed in the depends_on section. For consistency with the rest of the codebase (e.g., tb-core depends on tb-js-executor), and to ensure proper startup ordering, consider adding trendz-python-executor to the depends_on list of the trendz service. This applies to all three deployment configurations (basic, monolith, and advanced).

Suggested change

- tb-monolith

- tb-monolith

- trendz-python-executor

Added to depends_on

Copilot · 2026-01-28T15:44:57Z

monolith/docker-compose.trendz.yml

+      - ../trendz/data:/data
+    depends_on:
+      - postgres
+      - tb-monolith


The trendz service is configured with SCRIPT_ENGINE_DOCKER_PROVIDER_URL pointing to trendz-python-executor:8181, but trendz-python-executor is not listed in the depends_on section. For consistency with the rest of the codebase (e.g., tb-core depends on tb-js-executor), and to ensure proper startup ordering, consider adding trendz-python-executor to the depends_on list of the trendz service. This applies to all three deployment configurations (basic, monolith, and advanced).

Suggested change

- tb-monolith

- tb-monolith

- trendz-python-executor

Added to depends_on

Copilot · 2026-01-28T15:44:57Z

advanced/docker-compose.trendz.yml

+    depends_on:
+      - postgres
+      - tb-core1
+      - tb-core2


The trendz service is configured with SCRIPT_ENGINE_DOCKER_PROVIDER_URL pointing to trendz-python-executor:8181, but trendz-python-executor is not listed in the depends_on section. For consistency with the rest of the codebase (e.g., tb-core depends on tb-js-executor), and to ensure proper startup ordering, consider adding trendz-python-executor to the depends_on list of the trendz service. This applies to all three deployment configurations (basic, monolith, and advanced).

Suggested change

- tb-core2

- tb-core2

- trendz-python-executor

Added to depends_on

smatvienko-tb · 2026-01-28T16:01:10Z

🔍 Vulnerabilities of `thingsboard/trendz:1.15.0.2`

📦 Image Reference thingsboard/trendz:1.15.0.2

digest	`sha256:39a3a32d087a998632ccef6d834da587e10a24f9072a258551cc34c3e9ceae1b`
vulnerabilities
platform	linux/arm64
size	674 MB
packages	419

openjdk-25 25.0.1+8-1~deb13u1 (deb)

pkg:deb/debian/openjdk-25@25.0.1%2B8-1~deb13u1?os_distro=trixie&os_name=debian&os_version=13

Affected range	`<=25.0.1+8-1~deb13u1`
Fixed version	Not Fixed
EPSS Score	`0.052%`
EPSS Percentile	`16th percentile`

Description

Vulnerability in the Oracle Java SE, Oracle GraalVM for JDK, Oracle GraalVM Enterprise Edition product of Oracle Java SE (component: Security). Supported versions that are affected are Oracle Java SE: 8u471, 8u471-b50, 8u471-perf, 11.0.29, 17.0.17, 21.0.9, 25.0.1; Oracle GraalVM for JDK: 17.0.17 and 21.0.9; Oracle GraalVM Enterprise Edition: 21.3.16. Easily exploitable vulnerability allows unauthenticated attacker with network access via multiple protocols to compromise Oracle Java SE, Oracle GraalVM for JDK, Oracle GraalVM Enterprise Edition. Successful attacks of this vulnerability can result in unauthorized ability to cause a hang or frequently repeatable crash (complete DOS) of Oracle Java SE, Oracle GraalVM for JDK, Oracle GraalVM Enterprise Edition. Note: This vulnerability applies to Java deployments, typically in clients running sandboxed Java Web Start applications or sandboxed Java applets, that load and run untrusted code (e.g., code that comes from the internet) and rely on the Java sandbox for security. This vulnerability does not apply to Java deployments, typically in servers, that load and run only trusted code (e.g., code installed by an administrator). CVSS 3.1 Base Score 7.5 (Availability impacts). CVSS Vector: (CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H).

openjdk-8 (bug https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1126119)

openjdk-11 11.0.30+7-1

openjdk-17 17.0.18+8-1

openjdk-21 21.0.10+7-1

openjdk-25 25.0.2+10-1
https://openjdk.org/groups/vulnerability/advisories/2026-01-20

Affected range	`<=25.0.1+8-1~deb13u1`
Fixed version	Not Fixed
EPSS Score	`0.031%`
EPSS Percentile	`9th percentile`

Description

Vulnerability in the Oracle Java SE, Oracle GraalVM for JDK, Oracle GraalVM Enterprise Edition product of Oracle Java SE (component: AWT, JavaFX). Supported versions that are affected are Oracle Java SE: 8u471, 8u471-b50, 8u471-perf, 11.0.29, 17.0.17, 21.0.9, 25.0.1; Oracle GraalVM for JDK: 17.0.17 and 21.0.9; Oracle GraalVM Enterprise Edition: 21.3.16. Easily exploitable vulnerability allows unauthenticated attacker with network access via multiple protocols to compromise Oracle Java SE, Oracle GraalVM for JDK, Oracle GraalVM Enterprise Edition. Successful attacks require human interaction from a person other than the attacker and while the vulnerability is in Oracle Java SE, Oracle GraalVM for JDK, Oracle GraalVM Enterprise Edition, attacks may significantly impact additional products (scope change). Successful attacks of this vulnerability can result in unauthorized creation, deletion or modification access to critical data or all Oracle Java SE, Oracle GraalVM for JDK, Oracle GraalVM Enterprise Edition accessible data. Note: This vulnerability applies to Java deployments, typically in clients running sandboxed Java Web Start applications or sandboxed Java applets, that load and run untrusted code (e.g., code that comes from the internet) and rely on the Java sandbox for security. This vulnerability does not apply to Java deployments, typically in servers, that load and run only trusted code (e.g., code installed by an administrator). CVSS 3.1 Base Score 7.4 (Integrity impacts). CVSS Vector: (CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:C/C:N/I:H/A:N).

openjdk-8 (bug https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1126119)

openjdk-11 11.0.30+7-1

openjdk-17 17.0.18+8-1

openjdk-21 21.0.10+7-1

openjdk-25 25.0.2+10-1
https://openjdk.org/groups/vulnerability/advisories/2026-01-20

RomenKost added 12 commits December 3, 2025 13:47

Add Trendz configuration

5de0e3e

Add Trendz upgrade script

998c57f

Simplify Trendz upgrade script

08969a0

Fix .gitignore, fix python executor log volumes

95f2207

Fix haproxy.cfg

559b73c

Add Trendz license key placeholder

d3c627f

Revert "Add Trendz license key placeholder"

555d095

This reverts commit d3c627f.

Add TB_API_URL, remove depends_on

b5f6f87

Change Trendz Sync envs

a26cbc4

Fix haproxy

2bbdcd6

Merge branch 'refs/heads/master' into trendz-configuration

469fac5

# Conflicts: # .env

Fix haproxy (2)

c039866

smatvienko-tb requested changes Jan 19, 2026

View reviewed changes

RomenKost added 5 commits January 20, 2026 15:14

Fix Trendz/python-executor logback.xml and .conf

d492bcd

Fix license header

893bdfd

Fix logback for Trendz

a907480

Merge branch 'refs/heads/master' into trendz-configuration

1dc3ad1

# Conflicts: # .env

Add Trendz db init logic

394bf0a

RomenKost requested a review from smatvienko-tb January 28, 2026 09:12

ViacheslavKlimov requested a review from Copilot January 28, 2026 10:44

Copilot started reviewing on behalf of ViacheslavKlimov January 28, 2026 10:44 View session

Copilot AI reviewed Jan 28, 2026

View reviewed changes

RomenKost added 2 commits January 28, 2026 16:21

Minor fixes

0728aba

Fix permissions

f77a78d

ViacheslavKlimov requested a review from Copilot January 28, 2026 15:34

Copilot started reviewing on behalf of ViacheslavKlimov January 28, 2026 15:34 View session

Copilot AI reviewed Jan 28, 2026

View reviewed changes

Minor fixes

94e72af

	- ../trendz-python-executor/conf:/python-executor-config-files
	- ../trendz-python-executor/conf:/python-executor-config-files
	- ../trendz-python-executor/log:/var/log/python-executor

	799 799 trendz/log
	799 799 trendz/log
	799 799 trendz-python-executor/log

	# Python executor service configuration
	# Trendz service configuration

Trendz integration #84

Are you sure you want to change the base?

Trendz integration #84

Uh oh!

Conversation

RomenKost commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smatvienko-tb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

RomenKost Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

RomenKost commented Dec 4, 2025 •

edited

Loading

RomenKost Jan 28, 2026 •

edited

Loading

🔍 Vulnerabilities of `thingsboard/trendz:1.15.0.2`