zephyr: socket: send fails instead of blocking when there are no more net buffers

Describe the bug When trying to send a large amount of data to a TCP socket an application can quickly exhausts the net buffers. The standard behavior of send it to block until buffers are released (see man page extract below), but in Zephyr send returns an error.

Send man page:

When the message does not fit into the send buffer of the socket, send() normally blocks, unless the socket has been placed in nonblock‐ing I/O mode.

To Reproduce Steps to reproduce the behavior:

1. Apply patch in test-tcp2.zip
2. west build -b mimxrt1050_evk zephyr/samples/net/sockets/dumb_http_server
3. west flash
4. load the page with curl while true; do time curl http://192.168.1.241:8080 --output test || break; done

Expected behavior send should block until more data can be send and not fail.

Impact Since we need to serve “large” files via HTTP this problem prevents us to deliver stable firmware to our customers.

Logs and console output Here is what is printed when the error occurs:

Connection #150 from 192.168.1.146
Err[00:00:55.861,000] <err> net_pkt: Data buffer (1500) allocation failed.

Environment (please complete the following information):

• OS: Kubuntu 20.04
• Toolchain: Zephyr SDK 0.11.4, west 0.7.3
• Zephyr: v2.4.0-rc1