hazelcast-kubernetes: Hazelcast-Kubernetes plugin does not work in OrientDB cluster mode

I have been trying to setup OrientDB in a distributed mode by following the instructions in Readme about setting up Hazelcast plugin for Kubernetes. I am using a stateful set with quite a lot of volumes. Also, I took a DNS approach. I checked from all the pods in a cluster that they find each other. Also, the same selectors defined in a headless service can be applied to the pods. It seems like there are no problems in a Kubernetes cluster setup. Also, I checked that I defined a proper DNS name for the service. But I see in the logs for each pod that they form a single-member cluster instead of joining a shared one:

Members [1] {
	Member [ip-address]:5701 - ... this
}
 [DiscoveryJoiner]

Also, OrientDB enters a shared mode and all the database related configs are set up. What could be the cause of the problem? That`s headless service configuration:

kind: Service
apiVersion: v1
metadata:
  name: orientdbservice
  labels:
    service: orientdb
    type: headless-service
spec:
  type: ClusterIP
  clusterIP: None
  selector:
    service: orientdb
    type: container-deployment
  ports:
  - port: 2424
    name: port-binary
  - port: 2480
    name: port-http
  - port: 5701
    name: hazelcast

This is my hazelcast file:

<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.3.xsd"
           xmlns="http://www.hazelcast.com/schema/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <group>
        <name>orientdb-test-cluster-1</name>
        <password>Ml835yTe92</password>
    </group>
    <properties>
        <property name="hazelcast.discovery.enabled">true</property>
    </properties>
    <network>
        <join>
            <multicast enabled="false"/>
            <tcp-ip enabled="false"/>
            <discovery-strategies>
                <discovery-strategy enabled="true"
                                    class="com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy">
                    <properties>
                        <property name="service-dns">orientdbservice.default.svc.cluster.local</property>
                        <property name="service-dns-timeout">10</property>
                    </properties>
                </discovery-strategy>
            </discovery-strategies>
        </join>
    </network>
</hazelcast>

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 24

Most upvoted comments

@leszko The error was indeed in it. I have mounted this jar file into Orientdb/plugins directory in the command section of my Stateful Set and the plugin started to work properly. I think you can close this issue. These are the logs now:

2018-11-06 12:30:07:809 INFO  [172.17.0.3]:5701 [orientdb-test-cluster-1] [3.10.4] Kubernetes Discovery activated resolver: DnsEndpointResolver [DiscoveryService]
2018-11-06 12:30:08:264 INFO  [172.17.0.3]:5701 [orientdb-test-cluster-1] [3.10.4] Activating Discovery SPI Joiner [Node]
2018-11-06 12:30:11:271 INFO  [172.17.0.3]:5701 [orientdb-test-cluster-1] [3.10.4] Starting 2 partition threads and 3 generic threads (1 dedicated for priority tasks) [OperationExecutorImpl]

Members {size:3, ver:3} [
	Member [172.17.0.3]:5701 - hash
	Member [172.17.0.4]:5701 - hash
	Member [172.17.0.8]:5701 - hash
]

roksolana-d on Nov 6, 2018

2020-02-03 21:02:17:139 SEVER Plugin definition file ('plugin.json') is not found for dynamic plugin 'hazelcast-kubernetes-2.0.1.jar'Error on installing dynamic plugin 'hazelcast-kubernetes-2.0.1.jar'
java.lang.IllegalArgumentException: Plugin definition file ('plugin.json') is not found for dynamic plugin 'hazelcast-kubernetes-2.0.1.jar'

This error does not look like coming from Hazelcast. I’m not sure what 'plugin.json` is, but it does not look like Hazelcast specific.

Can you please advise how you overcame this. Also, if you can please show your current hazelcast.xml config I think that would be helpful.

The hazelcast.xml is actually listed in the beginning of this GH Issue. To quote it here as well:

<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.3.xsd"
           xmlns="http://www.hazelcast.com/schema/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <group>
        <name>orientdb-test-cluster-1</name>
        <password>Ml835yTe92</password>
    </group>
    <properties>
        <property name="hazelcast.discovery.enabled">true</property>
    </properties>
    <network>
        <join>
            <multicast enabled="false"/>
            <tcp-ip enabled="false"/>
            <discovery-strategies>
                <discovery-strategy enabled="true"
                                    class="com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy">
                    <properties>
                        <property name="service-dns">orientdbservice.default.svc.cluster.local</property>
                        <property name="service-dns-timeout">10</property>
                    </properties>
                </discovery-strategy>
            </discovery-strategies>
        </join>
    </network>
</hazelcast>

You can also use shorter form, check https://github.com/hazelcast/hazelcast-kubernetes for details.

leszko on Feb 4, 2020

Great to hear that it worked 😃

leszko on Nov 6, 2018

Yes, the configuration is fine.

Actually, the exception you get: Caused by: com.hazelcast.config.properties.ValidationException: There is no discovery strategy factory to create 'DiscoveryStrategyConfig{properties={service-dns=orientdbservice2.default.svc.cluster.local, service-dns-timeout=10}, className='com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy', discoveryStrategyFactory=null}' Is it a typo in a strategy classname? Perhaps you forgot to include implementation on a classpath? can occur only in two situation AFAIK:

hazelcast-kubernetes.jar is not on your classpath
you have multiple Hazelcast Plugins on your classpath (and they override the service definition)

Neither seems to be your case, because you clearly have only one plugin on your classpath and it’s hazelcast-kubernetes.jar.

I just checked the classpath of OrientDB. Can you check that hazelcast-kubernetes.jar is in one of these folders: $ORIENTDB_HOME/lib/orientdb-server-@VERSION@.jar:$ORIENTDB_HOME/lib/*:$ORIENTDB_HOME/plugins/*?

leszko on Nov 6, 2018