Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to deploy tb-integration-executor with redis sentinel #88

Open
TheSamabo opened this issue Sep 3, 2024 · 2 comments
Open

Unable to deploy tb-integration-executor with redis sentinel #88

TheSamabo opened this issue Sep 3, 2024 · 2 comments

Comments

@TheSamabo
Copy link

Hello,
i'm facing an issue with deployment of tb-integration-executor and redis sentinel. I was able to deploy tb-node with sentinel configured
(even tho some important information is missing in the TB Core component docs) with the following StatefulSet env vars:

        - name: CACHE_TYPE
          value: "redis"
        - name: REDIS_CONNECTION_TYPE
          value: "sentinel"
        - name: REDIS_MASTER
          value: "sts-redis-master"
        - name: REDIS_SENTINELS
          value: "sts-redis-node-0.sts-redis-headless.redis.svc.cluster.local:26379,sts-redis-node-1.sts-redis-headless.redis.svc.cluster.local:26379,sts-redis-node-2.sts-redis-headless.redis.svc.cluster.local:26379"
        - name: REDIS_SENTINEL_PASSWORD
          value: <Password>
        - name: REDIS_PASSWORD
          value: <Password>

So after seeing that the tb-integration-executor also specifies redis as a parameters in the minikube deployment yaml files. I thought i could just copy this configuration and it will work. But to my surprise it didn't.
When is set my thingsboard.yaml for the integration-executor env vars like this:

            - name: CACHE_TYPE
              value: "redis"
            - name: REDIS_CONNECTION_TYPE
              value: "sentinel"
            - name: REDIS_MASTER
              value: "sts-redis-master"
            - name: REDIS_SENTINELS
              value: "sts-redis-node-0.sts-redis-headless.redis.svc.cluster.local:26379,sts-redis-node-1.sts-redis-headless.redis.svc.cluster.local:26379,sts-redis-node-2.sts-redis-headless.redis.svc.cluster.local:26379"
            - name: REDIS_SENTINEL_PASSWORD
              value: <Password>
            - name: REDIS_PASSWORD
              value: <Password>

the pod logs look like this :

thingsboard-tb-integration-executor-0-1725375316634848100.log

Caused by: java.lang.IllegalArgumentException: Sentinel Master Id must not be null or empty
    at org.springframework.util.Assert.hasText(Assert.java:289)
    at org.springframework.data.redis.connection.SentinelMasterId.<init>(SentinelMasterId.java:34)
    at org.springframework.data.redis.connection.RedisConfiguration$SentinelConfiguration.setMaster(RedisConfiguration.java:357)
    at org.thingsboard.server.cache.TBRedisSentinelConfiguration.loadFactory(TBRedisSentinelConfiguration.java:65)
    at org.thingsboard.server.cache.TBRedisCacheConfiguration.redisConnectionFactory(TBRedisCacheConfiguration.java:105)
    at org.thingsboard.server.cache.TBRedisSentinelConfiguration$$EnhancerBySpringCGLIB$$59e3a0.CGLIB$redisConnectionFactory$31(<generated>)
    at org.thingsboard.server.cache.TBRedisSentinelConfiguration$$EnhancerBySpringCGLIB$$59e3a0$$FastClassBySpringCGLIB$$4055d0d5.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
    at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)
    at org.thingsboard.server.cache.TBRedisSentinelConfiguration$$EnhancerBySpringCGLIB$$59e3a0.redisConnectionFactory(<generated>)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
    ... 56 common frames omitted

So after looking in some thingsboard code i found that the key to provide redis sentinel master name is: redis.sentinel.master So i thought of env variable would look like this: REDIS_SENTINEL_MASTER, when i tried that in the env vars for the integration-executor i got the following logs:

thingsboard-tb-integration-executor-0-1725375710414817500.log

Caused by: redis.clients.jedis.exceptions.JedisConnectionException: All sentinels down, cannot determine where is sts-redis-master master is running...
    at redis.clients.jedis.JedisSentinelPool.initSentinels(JedisSentinelPool.java:293)
    at redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:217)
    at redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:205)
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createRedisSentinelPool(JedisConnectionFactory.java:415)
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createPool(JedisConnectionFactory.java:398)
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.afterPropertiesSet(JedisConnectionFactory.java:348)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1863)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1800)
    ... 52 common frames omitted

Please bear in mind that this redis sentinel cluster is working fine for the tb-node.
Any help is appreciated, Thank you
Sam

@TheSamabo
Copy link
Author

EDIT:
I was able to finally deploy the integration executor with redis sentinel but i had to change the env variable keys:
REDIS_MASTER -> REDIS_SENTINEL_MASTER
REDIS_SENTINELS -> REDIS_SENTINEL_SENTINELS

I forgot to mention currently we are running TB 3.6.2 as a k8 deployment.

@TheSamabo TheSamabo reopened this Sep 3, 2024
@TheSamabo
Copy link
Author

The documentation of the integration executor should be updated to reflect this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant