spring-integration: max-messages-per-poll and max-fetch-size not working as expected
In what version(s) of Spring Integration are you seeing this issue?
6.2.4
Describe the bug
According to the documentation the following combination of values max-messages-per-poll=-1 and max-fetch-size=1 should have the following effect: the sftp synchronizer fetch all the files from sftp server but one at time. As soon as a file is downloaded, a message with the file as payload, is fired in the channel.
However with this setting, in a single pool, the synchronizer only fetch 1 file.
To Reproduce
Sftp server containing more than 1 files and max-messages-per-poll=-1 and max-fetch-size=1
Expected behavior
Fetch all the files and sent a message as soon as a file is downloaded.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 17 (9 by maintainers)
Commits related to this issue
- GH-9124: Mention `FileListFilter` in the `max-fetch.adoc` Fixes to: #9124 — committed to spring-projects/spring-integration by artembilan 3 years ago
- GH-9124: Mention `FileListFilter` in the `max-fetch.adoc` Fixes to: #9124 — committed to EddieChoCho/spring-integration by artembilan 3 years ago
Yes. That’s exactly what I said on one of mine previous comment. In fact, with Java DSL we do use that
SftpPersistentAcceptOnceFileListFilterfor simple conditions:But for use-case like yours we indeed have to mention something in that doc.
Thanks for your patience!
OK. Let’s see what
max-fetch-sizeis exactly about from the code perspective and the logic behind it:So, the
listreturns all the file names in the remote directory, in some order. Usually it is alphabetic. Then we apply the filter for those names. And since we ask only for a single file, the first one accepted is returned. Since you don’t track processed files, it is not a surprise that we always get the same first file, but since it has been processed by theFileSystemPersistentAcceptOnceFileListFilter, nothing is going to be emitted as a message.For the use-case you describe apparently a custom
FileListFiltercould be implemented, but that’s already outside of the framework.You can have something similar to the
SftpPersistentAcceptOnceFileListFilter, but reset its cache on each poll. SeeReceiveMessageAdvice.However same has to be applied to the
localFilteras well. This one also caches already filtered files to avoid duplication.