spring-cloud-deployer-kubernetes: Unable to utilize spring-cloud-dataflow instanceIndex when reading from namedDestination

My team and I are attempting to make a set of Streams in spring-cloud-dataflow defined as:

MySource1 > :topic
MySource2 > :topic
:topic > MySink

and we’d like to run 1 of each MySource and 4 instances of MySink. The key detail is that we need to avoid any potential partition assignment instability/inconsistency, so we’re attempting to enable static partition assignments on MySink. To do so, we set autoReblanceEnabled=false. We’ve managed to build two streams with MySource writing to topic with 4 partitions (confirmed data on all partitions) and build a stream with MySink that reads from topic. The problem is that all instances of MySink pull from partition 0 of topic instead of pulling the partition equal to its index.

We’ve confirmed that the instances of MySink have /config/application.properties files where INSTANCE_INDEX and spring.application.index are set to 0-3 in accordance with the id of the kubernetes pod name. However, all of the Sinks still pull from partition 0 instead of their index. We’ve done our best to trace through the various spring repos and make sense of why it’s not being set correctly. spring.application.index seems like it may not matter here, but INSTANCE_INDEX looks like it should be pulled in by BindingServiceProperties in spring-cloud-stream. We were able to attach a debugger to the Sink and confirm that it is always 0. We also confirmed that the partition successfully changes when we set INSTANCE_INDEX as an environment variable. Is there something we’re missing in terms of getting the index to set correctly?

We’re using 3.2.4 of spring-cloud-stream/kafka-binder and spring-cloud-dataflow 2.9.6 (effective POM shows the others should be 3.1.6, but I can’t imagine that’d be the issue).

https://github.com/spring-cloud/spring-cloud-deployer-kubernetes/blob/0a96e5dbdb259f0276b9c65a276a2e5609893a35/src/main/java/org/springframework/cloud/deployer/spi/kubernetes/KubernetesAppDeployer.java#L494-L495