openfl: E0425 [tcp_posix.cc:598] recvmsg encountered uncommon error: Message too long

My workspace: FederatedCrowdGuard.zip

I am trying to get a CrowdGuard Running in a federated manner with fx aggregator and fx collaborator but I am running into issues.

In my example I have two collaborators. One of which is reporting “Response code: StatusCode.RESOURCE_EXHAUSTED” right after init. It is not entering training.

[04/25/25 15:11:29] INFO     FL-Plan hash is 26e3f237d41a87071605031488c27d48fe15ae40c9e11aaf080d7ff5423ff878fc9d397f89edcefb3a26eb8c05069394        plan.py:266
                    INFO     Parsing Federated Learning Plan : SUCCESS : ~/Git/fork/openfl/FederatedCrowdGuard/plan/plan.yaml.                       plan.py:161
                    INFO     aggregator:                                                                                                             plan.py:167
                               settings:                                                                                                                        
                                 rounds_to_train: 1                                                                                                             
                               template: openfl.experimental.workflow.component.Aggregator                                                                      
                             collaborator:                                                                                                                      
                               settings: {}                                                                                                                     
                               template: openfl.experimental.workflow.component.Collaborator                                                                    
                             federated_flow:                                                                                                                    
                               settings:                                                                                                                        
                                 aggregation_algorithm: CrowdGuard                                                                                              
                                 model: null                                                                                                                    
                                 optimizer: null                                                                                                                
                                 pmr: 0.25                                                                                                                      
                                 top_model_accuracy: 0                                                                                                          
                                 total_rounds: 3                                                                                                                
                               template: src.flow.FederatedFlow                                                                                                 
                             network:                                                                                                                           
                               settings:                                                                                                                        
                                 agg_addr: localhost                                                                                                            
                                 agg_port: 55332                                                                                                                
                                 client_reconnect_interval: 5                                                                                                   
                                 disable_client_auth: false                                                                                                     
                               template: openfl.federation.Network                                                                                              
                                                                                                                                                                
                    INFO     🧿 Starting the Collaborator Service.                                                                            collaborator.py:92
Files already downloaded and verified
Files already downloaded and verified
[04/25/25 15:11:36] INFO     Importing 🡆 Object collaborator_private_attrs from src.collaborator_private_attrs Module.                               plan.py:237
Files already downloaded and verified
Files already downloaded and verified
[04/25/25 15:11:41] INFO     Building `openfl.experimental.workflow.component.Collaborator` Module.                                                  plan.py:207
[04/25/25 15:12:09] INFO     Initializing collaborator.                                                                                       collaborator.py:56
[04/25/25 15:12:10] INFO     Waiting for tasks...                                                                                            collaborator.py:180
[04/25/25 15:12:15] INFO     Response code: StatusCode.RESOURCE_EXHAUSTED

However the other collaborator runs the first round but the aggregator reports that the result message is too long.

Sending state from aggregator to collaborators
                    INFO     Starting Aggregator gRPC Server                                                                            aggregator_server.py:229
[04/25/25 15:14:58] INFO     Waiting for 2/2 collaborators to connect...                                                                       aggregator.py:227
[04/25/25 15:15:05] INFO     Collaborator col2 is connected.                                                                                   aggregator.py:305
                    INFO     Sending tasks to collaborator col2 for round 1...                                                                 aggregator.py:335
[04/25/25 15:15:08] INFO     Waiting for 1/2 collaborators to connect...                                                                       aggregator.py:227
E0425 15:15:19.551040000 15468621824 tcp_posix.cc:598]                 recvmsg encountered uncommon error: Message too long
E0425 15:15:21.543904000 15468621824 tcp_posix.cc:598]                 recvmsg encountered uncommon error: Message too long
[04/25/25 15:15:21] INFO     Collaborator col2 sent task results for round 1.                                                                  aggregator.py:464
[04/25/25 15:15:28] INFO     Waiting for 1/2 collaborators to connect...  

I would appreciate any hints. Thank you in advance.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 19 (12 by maintainers)

Commits related to this issue

Most upvoted comments

Hello @tayfunceylan

We have done some improvements and optimized the size of messages exchanged between participants in #1589. With this change we are able to run the FederatedCrowdGuard Tutorial shared in #1569 on openfl develop branch

As the size of messages that can be exchanged is currently limited to 1GB, issues can still be observed if larger models are being trained. While this limit is being analyzed and addressed, in-progress version of changes in #1572 can be used to extend the limit to 2GB

Additional Notes: Following adjustments were made to run the Tutorial

#| export

seed_random_generators(RANDOM_SEED)
 
model = Net()
optimizer = default_optimizer(model,"SGD")
pmr_val = NUMBER_OF_MALICIOUS_CLIENTS / TOTAL_CLIENT_NUMBER
n_rounds = 5
top_model_accuracy = 0
aggregation_algo="CrowdGuard"
optimizer_type="SGD"
 
flflow = FederatedFlow_CrowdGuard(
    model,
    optimizer,
    n_rounds,
    top_model_accuracy,
    pmr_val,
    aggregation_algo,
    optimizer_type="SGD"
)
       
flflow.runtime = federated_runtime
flflow.run()
class FederatedFlow_CrowdGuard(FLSpec):
    def __init__(
        self,
        model=None,
        optimizer=None,
        total_rounds=10,
        top_model_accuracy=0,
        pmr=0.25,
        aggregation_algorithm='CrowdGuard',
        optimizer_type = "SGD",
        **kwargs, 
    ):
        super().__init__(**kwargs)
<snip>
        self.device = "cpu"
        self.optimizer_type = optimizer_type
        self.round_num = 0  # starting round
<snip>

    @collaborator
    def train(self):
        self.collaborator_name = self.input
<snip>
        print("Why is self.optimizer a string suddenly?", type(self.optimizer))
        self.optimizer = default_optimizer(self.model, optimizer_type=self.optimizer_type, optimizer_like=self.optimizer)

        self.model.train()
<snip>

@teoparvanov, I have created a pull request with the the code and needed changes to run CrowdGuard on the new FederatedRuntime in #1569 .

I hope there is everything you need to solve this issue smoothly.

Hi @tayfunceylan,

Thank you for your feedback and contributions. It’s great to hear that you successfully ran CrowdGuard in a distributed environment👍

One question:  Currently all the code must be contained in a Jupyter Notebook and marked with ‘#| export’. What would be the recommended way if you wanted to include a class from a second python file? For CrowdGuard I had to copy a huge class that was inside CrowdGuardValidation.py.

Regarding your question: current implementation of FederatedRuntime is designed based on the assumption that the entire Federated Learning experiment is defined within a Jupyter Notebook. This requires users or data scientists to annotate their notebook with #| export directives and ensure that all user-defined code is also included directly within the notebook.

At present, importing user-defined modules from separate Python files is not supported and will result in errors when executed on distributed nodes. We will update the FederatedRuntime tutorial to clarify this guideline and in addition consider the possibility of supporting this functionality in a future release

Hello @tayfunceylan

Thank you for the feedback and update. We were able to reproduce the Response code: StatusCode.RESOURCE_EXHAUSTED and are in process of addressing it.

However, we have not observed the E0430 tcp_posix.cc:598 recvmsg encountered uncommon error: Message too long. Could you please try reproducing this error in a Linux environment? This will help us determine if the issue is environment-specific

Additionally, the pull request was modified recently, possibly while you were testing. While not mandatory, you might consider rebasing to ensure compatibility

Thanks @tayfunceylan, we’ll refer to your PR to further analyze the issue. In fact, @scngupta-dsp has been able to reproduce it in a simpler scenario, where he artificially allocated a large model in one of the workflow steps.

A possible solution may involve some sort of chunking when transferring the state between agg/cols, which will likely need to be implemented at a lower, framework-level.

We’ll keep you updated as soon as we have a root-cause or a fix/workaround.

@teoparvanov

I can confirm, it seems to be an issue with the message/payload size. For example when I remove one of the models being sent, I am getting further in the training flow. But eventually the flow stops with the same error codes at a later point. Notice how E0428 tcp_posix.cc:598 recvmsg encountered uncommon error: Message too long is printed all over the place by both collaborators and aggregator. Eventually leading to one or both of the collaborators to report Response code: StatusCode.RESOURCE_EXHAUSTED.

col1

                    INFO     🧿 Starting the Collaborator Service.                                                                            collaborator.py:92
[04/28/25 13:16:27] INFO     Importing 🡆 Object collaborator_private_attrs from src.collaborator_private_attrs Module.                               plan.py:237
[04/28/25 13:16:32] INFO     Building `openfl.experimental.workflow.component.Collaborator` Module.                                                  plan.py:207
[04/28/25 13:17:00] INFO     Initializing collaborator.                                                                                       collaborator.py:56
[04/28/25 13:17:01] INFO     Waiting for tasks...                                                                                            collaborator.py:180
E0428 13:17:06.870410000 8447265920 tcp_posix.cc:598]                  recvmsg encountered uncommon error: Message too long
E0428 13:17:07.955135000 8447265920 tcp_posix.cc:598]                  recvmsg encountered uncommon error: Message too long
[04/28/25 13:17:11] INFO     Received the following tasks: train.                                                                            collaborator.py:145

Calling train
####################
Performing model training for collaborator col1 in round 0
Benign Train set: Avg. loss: 2.302695268078854, Accuracy: 117/1200 (9.750%)
Benign Test set: Avg. loss: 2.302213430404663, Accuracy: 134/1200 (11.167%)
Backdoor Test set: Avg. loss: 2.296622395515442, Accuracy: 0/1200 (0.000%)
Benign Train set: Avg. loss: 2.3024386606718363, Accuracy: 136/1200 (11.333%)
Benign Test set: Avg. loss: 2.3022998571395874, Accuracy: 158/1200 (13.167%)
Backdoor Test set: Avg. loss: 2.298975944519043, Accuracy: 0/1200 (0.000%)
Sending state from collaborator to aggregator
Should transfer from train to collect_models
[04/28/25 13:17:22] INFO     Round 1, collaborator col1 is sending results...                                                                collaborator.py:163
                    INFO     Waiting for tasks...                                                                                            collaborator.py:180
[04/28/25 13:17:37] INFO     Response code: StatusCode.RESOURCE_EXHAUSTED

col2

                    INFO     🧿 Starting the Collaborator Service.                                                                            collaborator.py:92
[04/28/25 13:16:29] INFO     Importing 🡆 Object collaborator_private_attrs from src.collaborator_private_attrs Module.                               plan.py:237
[04/28/25 13:16:33] INFO     Building `openfl.experimental.workflow.component.Collaborator` Module.                                                  plan.py:207
[04/28/25 13:17:02] INFO     Initializing collaborator.                                                                                       collaborator.py:56
[04/28/25 13:17:03] INFO     Waiting for tasks...                                                                                            collaborator.py:180
E0428 13:17:03.848257000 8447265920 tcp_posix.cc:598]                  recvmsg encountered uncommon error: Message too long
[04/28/25 13:17:07] INFO     Received the following tasks: train.                                                                            collaborator.py:145

Calling train
####################
Performing model training for collaborator col2 in round 0
Benign Train set: Avg. loss: 2.302695268078854, Accuracy: 117/1200 (9.750%)
Benign Test set: Avg. loss: 2.302213430404663, Accuracy: 134/1200 (11.167%)
Backdoor Test set: Avg. loss: 2.296622395515442, Accuracy: 0/1200 (0.000%)
Benign Train set: Avg. loss: 2.3024386606718363, Accuracy: 136/1200 (11.333%)
Benign Test set: Avg. loss: 2.3022998571395874, Accuracy: 158/1200 (13.167%)
Backdoor Test set: Avg. loss: 2.298975944519043, Accuracy: 0/1200 (0.000%)
Sending state from collaborator to aggregator
Should transfer from train to collect_models
[04/28/25 13:17:19] INFO     Round 1, collaborator col2 is sending results...                                                                collaborator.py:163
[04/28/25 13:17:20] INFO     Waiting for tasks...                                                                                            collaborator.py:180
E0428 13:17:32.285404000 8447265920 tcp_posix.cc:598]                  recvmsg encountered uncommon error: Message too long
E0428 13:17:33.320887000 8447265920 tcp_posix.cc:598]                  recvmsg encountered uncommon error: Message too long
[04/28/25 13:17:35] INFO     Received the following tasks: local_validation.                                                                 collaborator.py:145

Calling local_validation
Performing model validation for collaborator col2 in round 0
Suspicious Models detected by 0: []
Sending state from collaborator to aggregator
Should transfer from local_validation to defend
[04/28/25 13:17:42] INFO     Round 1, collaborator col2 is sending results...                                                                collaborator.py:163
[04/28/25 13:17:43] INFO     Waiting for tasks...

aggregator

####################
Round 0...
####################
[04/28/25 13:16:23] INFO     Building `openfl.experimental.workflow.component.Aggregator` Module.                                                    plan.py:207
                    INFO     MetaflowInterface creation.                                                                                       aggregator.py:129
                    INFO     Starting round 1...                                                                                               aggregator.py:207

Calling start
Performing initialization for model
Sending state from aggregator to collaborators
                    INFO     Waiting for 2/2 collaborators to connect...                                                                       aggregator.py:227
                    INFO     Starting Aggregator gRPC Server                                                                            aggregator_server.py:229
[04/28/25 13:16:33] INFO     Waiting for 2/2 collaborators to connect...                                                                       aggregator.py:227
[04/28/25 13:17:03] INFO     Collaborator col2 is connected.                                                                                   aggregator.py:305
                    INFO     Sending tasks to collaborator col2 for round 1...                                                                 aggregator.py:335
                    INFO     Waiting for 1/2 collaborators to connect...                                                                       aggregator.py:227
[04/28/25 13:17:04] INFO     Collaborator col1 is connected.                                                                                   aggregator.py:305
                    INFO     Sending tasks to collaborator col1 for round 1...                                                                 aggregator.py:335
[04/28/25 13:17:13] INFO     Waiting for 2/2 collaborators to send results...                                                                  aggregator.py:240
E0428 13:17:18.560094000 14098624512 tcp_posix.cc:598]                 recvmsg encountered uncommon error: Message too long
E0428 13:17:19.890422000 14098624512 tcp_posix.cc:598]                 recvmsg encountered uncommon error: Message too long
[04/28/25 13:17:20] INFO     Collaborator col2 sent task results for round 1.                                                                  aggregator.py:464
E0428 13:17:21.233090000 14098624512 tcp_posix.cc:598]                 recvmsg encountered uncommon error: Message too long
E0428 13:17:22.655461000 14098624512 tcp_posix.cc:598]                 recvmsg encountered uncommon error: Message too long
[04/28/25 13:17:22] INFO     Collaborator col1 sent task results for round 1.                                                                  aggregator.py:464

Calling collect_models
Sending state from aggregator to collaborators
[04/28/25 13:17:24] INFO     Waiting for 2/2 to make requests for tasks...                                                                     aggregator.py:233
[04/28/25 13:17:30] INFO     Sending tasks to collaborator col2 for round 1...                                                                 aggregator.py:335
[04/28/25 13:17:33] INFO     Sending tasks to collaborator col1 for round 1...                                                                 aggregator.py:335
[04/28/25 13:17:34] INFO     Waiting for 2/2 collaborators to send results...                                                                  aggregator.py:240
E0428 13:17:40.846470000 14098624512 tcp_posix.cc:598]                 recvmsg encountered uncommon error: Message too long
E0428 13:17:42.857041000 14098624512 tcp_posix.cc:598]                 recvmsg encountered uncommon error: Message too long
[04/28/25 13:17:43] INFO     Collaborator col2 sent task results for round 1.                                                                  aggregator.py:464
[04/28/25 13:17:44] INFO     Waiting for 1/2 collaborators to send results...                                                                  aggregator.py:240
[04/28/25 13:17:54] INFO     Waiting for 1/2 collaborators to send results...                                                                  aggregator.py:240

Thanks for the great discussions and collaboration around this topic! I think we can now close this issue as the original problem has been resolved.

Hi @teoparvanov, it has been a few days. I re-tested federated CrowdGuard and it works flawlessly now. Even without that change in code_analyzer.py that was necessary earlier. Great work!

Is there any interest in a federated CrowdGuard example? I would like to update the pull request so that the example is a complete setup with 4 collaborators.

One question:  Currently all the code must be contained in a Jupyter Notebook and marked with ‘#| export’. What would be the recommended way if you wanted to include a class from a second python file? For CrowdGuard I had to copy a huge class that was inside CrowdGuardValidation.py.

Hi @tayfunceylan, all the intended fixes have been merged. Please re-test federated CrowdGuard with the latest code from the develop branch and let us know how it goes.

@tayfunceylan: Thanks for the feedback that E0430 tcp_posix.cc:598 recvmsg encountered uncommon error: Message too long is environment specific

We have further analyzed the latest issue google.protobuf.message.DecodeError: Error parsing message with type 'openfl.experiment.aggregator.GetTasksResponse' being observed and are in the process of doing some more improvements and optimizations to overcome this issue. We will update you on this thread as soon as this change is ready

Hi @teoparvanov , thank you for the quick reply!

running the CrowdGuard example in the LocalRuntime works perfectly.

However I am also running into similar issues when using the new FederatedRuntime where the collaborators are reporting Response code: StatusCode.RESOURCE_EXHAUSTED right after Waiting for tasks....

Envoys

INFO     Building `openfl.experimental.workflow.component.Collaborator` Module.                                                  plan.py:207
INFO     Initializing collaborator.                                                                                       collaborator.py:56
INFO     Waiting for tasks...                                                                                            collaborator.py:180
INFO     Response code: StatusCode.RESOURCE_EXHAUSTED                                                                aggregator_client.py:53

Director

Sending state from aggregator to collaborators
INFO     Waiting for 2/2 collaborators to connect...                                                                       aggregator.py:227
INFO     Collaborator Bangalore is connected.                                                                              aggregator.py:305
INFO     Sending tasks to collaborator Bangalore for round 1...                                                            aggregator.py:335
INFO     Waiting for 1/2 collaborators to connect...                                                                       aggregator.py:227
INFO     Collaborator Chandler is connected.                                                                               aggregator.py:305
INFO     Sending tasks to collaborator Chandler for round 1...                                                             aggregator.py:335
INFO     Waiting for 2/2 collaborators to send results...                                                                  aggregator.py:240
INFO     Waiting for 2/2 collaborators to send results...       

In the end I would like to run CrowdGuard in a federated environment and in SGX. Unfortunatley I am kinda stuck.