libzmq: zmq_msg_send and zmq_poll return whey they shouldn't (MacOS)

I am having an issue using zmq_msg_send and zmq_poll on MacOS.

When using zmq_poll with ZMQ_POLLOUT, it triggers and zmq_msg_send allows me to send messages even when nobody is connected (socket is binding). The sending does not actually send/queue anything. I have tested it on a PUSH socket with ZeroMQ versions 4.1.3 and 4.2.0.

The following is a minimal code to reproduce the problem:

#include <iostream>
#include <cassert>
#include "zmq.h"

int main()
{
    void* ctx = zmq_ctx_new();
    void* socket = zmq_socket(ctx, ZMQ_PUSH);
    int rc = zmq_bind(socket, "tcp://127.0.0.1:5555"); assert(rc == 0);

    zmq_pollitem_t items[1];
    items[0].socket = socket;
    items[0].fd = 0;
    items[0].events = ZMQ_POLLOUT;

    int counter = 0;

    while (true)
    {
        rc = zmq_poll(items, 1, 1000);
        std::cout << "zmq_poll => " << rc << std::endl;

        if (items[0].revents & ZMQ_POLLOUT)
        {
            zmq_msg_t msg;
            rc = zmq_msg_init_size(&msg, 1000); assert(rc >= 0);
            int nbytes = zmq_msg_send(&msg, socket, 0);
            std::cout << "sent msg nr. " << ++counter << " (" << nbytes << " bytes)." << std::endl;
        }
    }

    zmq_close(socket);
    zmq_ctx_term(ctx);

    return 0;
}

clang++ zmq-poll-send.cxx -o zmq-poll-send -lzmq

When running the executable by itself (without a receiver) on a Linux system (Ubuntu 16.10 + GCC 6.2.0), it does the right thing. No queue is created (binding socket without connected peers) and sending cannot be done:

zmq_poll => 0
zmq_poll => 0
zmq_poll => 0
zmq_poll => 0
...

But when running it on MacOS (10.12.1 + Apple LLVM version 8.0.0 (clang-800.0.42.1)), poll (mostly) reports that the sending can be done and zmq_msg_send “sends” it:

zmq_poll => 0
zmq_poll => 1
sent msg nr. 1 (1000 bytes).
zmq_poll => 1
sent msg nr. 2 (1000 bytes).
zmq_poll => 1
sent msg nr. 3 (1000 bytes).
zmq_poll => 1
sent msg nr. 4 (1000 bytes).
zmq_poll => 1
sent msg nr. 5 (1000 bytes).
zmq_poll => 1
sent msg nr. 6 (1000 bytes).
zmq_poll => 1
sent msg nr. 7 (1000 bytes).
zmq_poll => 1
sent msg nr. 8 (1000 bytes).
zmq_poll => 1
sent msg nr. 9 (1000 bytes).
zmq_poll => 1
sent msg nr. 10 (1000 bytes).
zmq_poll => 1
sent msg nr. 11 (1000 bytes).
zmq_poll => 1
sent msg nr. 12 (1000 bytes).
zmq_poll => 1
sent msg nr. 13 (1000 bytes).
zmq_poll => 1
sent msg nr. 14 (1000 bytes).
zmq_poll => 1
sent msg nr. 15 (1000 bytes).
zmq_poll => 1
sent msg nr. 16 (1000 bytes).
zmq_poll => 1
sent msg nr. 17 (1000 bytes).
zmq_poll => 0
zmq_poll => 1
sent msg nr. 18 (1000 bytes).
zmq_poll => 1
sent msg nr. 19 (1000 bytes).
...

My hope was that at least the messages would go into queue (although it should not exist yet since no peers are connected), but this goes on over the default queue size of 1000. And if i connect a receiver the already sent messages do not arrive (new ones do). So the messages are lost!

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Comments: 18 (10 by maintainers)

Most upvoted comments

Without looking at the code in details yet, the first question that pops up in my head is: could it be down to the poller implementation? On OSX the default in kqueue, while on Linux is epoll.

Could you please try and see if you can reproduce the problem on Linux when using kqueue? ./configure --with-poller=kqueue

Also when using 4.2.0 there’s a new poller implementation which is DRAFT, so if you built with draft try with --enable-drafts=no if the draft api were enabled, to try and rule out the initial first suspects.