spring-integration: max-messages-per-poll and max-fetch-size not working as expected

In what version(s) of Spring Integration are you seeing this issue?

6.2.4

Describe the bug

According to the documentation the following combination of values max-messages-per-poll=-1 and max-fetch-size=1 should have the following effect: the sftp synchronizer fetch all the files from sftp server but one at time. As soon as a file is downloaded, a message with the file as payload, is fired in the channel.

However with this setting, in a single pool, the synchronizer only fetch 1 file.

To Reproduce

Sftp server containing more than 1 files and max-messages-per-poll=-1 and max-fetch-size=1

Expected behavior

Fetch all the files and sent a message as soon as a file is downloaded.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 17 (9 by maintainers)

Commits related to this issue

Most upvoted comments

Yes. That’s exactly what I said on one of mine previous comment. In fact, with Java DSL we do use that SftpPersistentAcceptOnceFileListFilter for simple conditions:

public SftpInboundChannelAdapterSpec patternFilter(String pattern) {
	return filter(composeFilters(new SftpSimplePatternFileListFilter(pattern)));
}
...
private CompositeFileListFilter<SftpClient.DirEntry> composeFilters(FileListFilter<SftpClient.DirEntry>
			fileListFilter) {

	CompositeFileListFilter<SftpClient.DirEntry> compositeFileListFilter = new CompositeFileListFilter<>();
	compositeFileListFilter.addFilters(fileListFilter,
				new SftpPersistentAcceptOnceFileListFilter(new SimpleMetadataStore(), "sftpMessageSource"));
	return compositeFileListFilter;
}

But for use-case like yours we indeed have to mention something in that doc.

Thanks for your patience!

OK. Let’s see what max-fetch-size is exactly about from the code perspective and the logic behind it:

F[] files = session.list(remoteDirectory);
List<F> filteredFiles = applyFilter(files, haveFilter, filteringOneByOne, maxFetchSize);

So, the list returns all the file names in the remote directory, in some order. Usually it is alphabetic. Then we apply the filter for those names. And since we ask only for a single file, the first one accepted is returned. Since you don’t track processed files, it is not a surprise that we always get the same first file, but since it has been processed by the FileSystemPersistentAcceptOnceFileListFilter, nothing is going to be emitted as a message.

For the use-case you describe apparently a custom FileListFilter could be implemented, but that’s already outside of the framework.

You can have something similar to the SftpPersistentAcceptOnceFileListFilter, but reset its cache on each poll. See ReceiveMessageAdvice.

However same has to be applied to the localFilter as well. This one also caches already filtered files to avoid duplication.