openfl: E0425 [tcp_posix.cc:598] recvmsg encountered uncommon error: Message too long
My workspace: FederatedCrowdGuard.zip
I am trying to get a CrowdGuard Running in a federated manner with fx aggregator and fx collaborator but I am running into issues.
In my example I have two collaborators. One of which is reporting “Response code: StatusCode.RESOURCE_EXHAUSTED” right after init. It is not entering training.
[04/25/25 15:11:29] INFO FL-Plan hash is 26e3f237d41a87071605031488c27d48fe15ae40c9e11aaf080d7ff5423ff878fc9d397f89edcefb3a26eb8c05069394 plan.py:266
INFO Parsing Federated Learning Plan : SUCCESS : ~/Git/fork/openfl/FederatedCrowdGuard/plan/plan.yaml. plan.py:161
INFO aggregator: plan.py:167
settings:
rounds_to_train: 1
template: openfl.experimental.workflow.component.Aggregator
collaborator:
settings: {}
template: openfl.experimental.workflow.component.Collaborator
federated_flow:
settings:
aggregation_algorithm: CrowdGuard
model: null
optimizer: null
pmr: 0.25
top_model_accuracy: 0
total_rounds: 3
template: src.flow.FederatedFlow
network:
settings:
agg_addr: localhost
agg_port: 55332
client_reconnect_interval: 5
disable_client_auth: false
template: openfl.federation.Network
INFO 🧿 Starting the Collaborator Service. collaborator.py:92
Files already downloaded and verified
Files already downloaded and verified
[04/25/25 15:11:36] INFO Importing 🡆 Object collaborator_private_attrs from src.collaborator_private_attrs Module. plan.py:237
Files already downloaded and verified
Files already downloaded and verified
[04/25/25 15:11:41] INFO Building `openfl.experimental.workflow.component.Collaborator` Module. plan.py:207
[04/25/25 15:12:09] INFO Initializing collaborator. collaborator.py:56
[04/25/25 15:12:10] INFO Waiting for tasks... collaborator.py:180
[04/25/25 15:12:15] INFO Response code: StatusCode.RESOURCE_EXHAUSTED
However the other collaborator runs the first round but the aggregator reports that the result message is too long.
Sending state from aggregator to collaborators
INFO Starting Aggregator gRPC Server aggregator_server.py:229
[04/25/25 15:14:58] INFO Waiting for 2/2 collaborators to connect... aggregator.py:227
[04/25/25 15:15:05] INFO Collaborator col2 is connected. aggregator.py:305
INFO Sending tasks to collaborator col2 for round 1... aggregator.py:335
[04/25/25 15:15:08] INFO Waiting for 1/2 collaborators to connect... aggregator.py:227
E0425 15:15:19.551040000 15468621824 tcp_posix.cc:598] recvmsg encountered uncommon error: Message too long
E0425 15:15:21.543904000 15468621824 tcp_posix.cc:598] recvmsg encountered uncommon error: Message too long
[04/25/25 15:15:21] INFO Collaborator col2 sent task results for round 1. aggregator.py:464
[04/25/25 15:15:28] INFO Waiting for 1/2 collaborators to connect...
I would appreciate any hints. Thank you in advance.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 19 (12 by maintainers)
Commits related to this issue
- [WIP][Workflow Interface] Resolving "Response code: StatusCode.RESOURCE_EXHAUSTED" (Issue #1565) in FederatedRuntime (#1572) * grpc resource issue partial fix Signed-off-by: Ishant Thakare <ishantro... — committed to securefederatedai/openfl by ishant162 2 years ago
- [WIP][Workflow Interface] Resolving "Response code: StatusCode.RESOURCE_EXHAUSTED" (Issue #1565) in FederatedRuntime (#1572) * grpc resource issue partial fix Signed-off-by: Ishant Thakare <ishantro... — committed to tayfunceylan/openfl by ishant162 2 years ago
Hello @tayfunceylan
We have done some improvements and optimized the size of messages exchanged between participants in #1589. With this change we are able to run the
FederatedCrowdGuardTutorial shared in #1569 on openfldevelopbranchAs the size of messages that can be exchanged is currently limited to 1GB, issues can still be observed if larger models are being trained. While this limit is being analyzed and addressed, in-progress version of changes in #1572 can be used to extend the limit to 2GB
Additional Notes: Following adjustments were made to run the Tutorial
@teoparvanov, I have created a pull request with the the code and needed changes to run CrowdGuard on the new FederatedRuntime in #1569 .
I hope there is everything you need to solve this issue smoothly.
Hi @tayfunceylan,
Thank you for your feedback and contributions. It’s great to hear that you successfully ran CrowdGuard in a distributed environment👍
Regarding your question: current implementation of
FederatedRuntimeis designed based on the assumption that the entire Federated Learning experiment is defined within a Jupyter Notebook. This requires users or data scientists to annotate their notebook with#| exportdirectives and ensure that all user-defined code is also included directly within the notebook.At present, importing user-defined modules from separate Python files is not supported and will result in errors when executed on distributed nodes. We will update the FederatedRuntime tutorial to clarify this guideline and in addition consider the possibility of supporting this functionality in a future release
Hello @tayfunceylan
Thank you for the feedback and update. We were able to reproduce the
Response code: StatusCode.RESOURCE_EXHAUSTEDand are in process of addressing it.However, we have not observed the
E0430 tcp_posix.cc:598 recvmsg encountered uncommon error: Message too long. Could you please try reproducing this error in a Linux environment? This will help us determine if the issue is environment-specificAdditionally, the pull request was modified recently, possibly while you were testing. While not mandatory, you might consider rebasing to ensure compatibility
Thanks @tayfunceylan, we’ll refer to your PR to further analyze the issue. In fact, @scngupta-dsp has been able to reproduce it in a simpler scenario, where he artificially allocated a large model in one of the workflow steps.
A possible solution may involve some sort of chunking when transferring the state between agg/cols, which will likely need to be implemented at a lower, framework-level.
We’ll keep you updated as soon as we have a root-cause or a fix/workaround.
@teoparvanov
I can confirm, it seems to be an issue with the message/payload size. For example when I remove one of the models being sent, I am getting further in the training flow. But eventually the flow stops with the same error codes at a later point. Notice how
E0428 tcp_posix.cc:598 recvmsg encountered uncommon error: Message too longis printed all over the place by both collaborators and aggregator. Eventually leading to one or both of the collaborators to reportResponse code: StatusCode.RESOURCE_EXHAUSTED.col1
col2
aggregator
Thanks for the great discussions and collaboration around this topic! I think we can now close this issue as the original problem has been resolved.
Hi @teoparvanov, it has been a few days. I re-tested federated CrowdGuard and it works flawlessly now. Even without that change in code_analyzer.py that was necessary earlier. Great work!
Is there any interest in a federated CrowdGuard example? I would like to update the pull request so that the example is a complete setup with 4 collaborators.
One question: Currently all the code must be contained in a Jupyter Notebook and marked with ‘#| export’. What would be the recommended way if you wanted to include a class from a second python file? For CrowdGuard I had to copy a huge class that was inside
CrowdGuardValidation.py.Hi @tayfunceylan, all the intended fixes have been merged. Please re-test federated CrowdGuard with the latest code from the
developbranch and let us know how it goes.@tayfunceylan: Thanks for the feedback that
E0430 tcp_posix.cc:598 recvmsg encountered uncommon error: Message too longis environment specificWe have further analyzed the latest issue
google.protobuf.message.DecodeError: Error parsing message with type 'openfl.experiment.aggregator.GetTasksResponse'being observed and are in the process of doing some more improvements and optimizations to overcome this issue. We will update you on this thread as soon as this change is readyHi @teoparvanov , thank you for the quick reply!
running the CrowdGuard example in the
LocalRuntimeworks perfectly.However I am also running into similar issues when using the new
FederatedRuntimewhere the collaborators are reportingResponse code: StatusCode.RESOURCE_EXHAUSTEDright afterWaiting for tasks....Envoys
Director
In the end I would like to run CrowdGuard in a federated environment and in SGX. Unfortunatley I am kinda stuck.