sinatra: Cannot keep stream open with puma server

I cannot get a stream to stay open (which is necessary for Server-Sent Events to work) with Sinatra running on Puma. I don’t really have any idea why, but the connections close right away.

require 'sinatra'

set :server, :puma
set :connections, []

get '/stream', provides: 'text/event-stream' do
  stream :keep_open do |out|
    settings.connections << out
    out.callback do
      puts 'stream closed'
      settings.connections.delete(out)
    end
  end
end

Then I run curl http://localhost:4567/stream and the output is

$ ruby example.rb 
Puma 2.13.4 starting...
* Min threads: 0, max threads: 16
* Environment: development
* Listening on tcp://localhost:4567
== Sinatra (v1.4.6) has taken the stage on 4567 for development with backup from Puma
stream closed
127.0.0.1 - - [14/Sep/2015 09:45:33] "GET /stream HTTP/1.1" 200 - 0.0047

If I have the server as :thin, it works fine. I don’t want to use Thin though, for other reasons. And if I understand puma/puma#3 correctly, Puma can do this kind of streaming just fine.

This could be related to the problems on #962, but I’m using MRI (tested on 2.1.5p273) so it might be unrelated to any problems JRuby has.

About this issue

  • Original URL
  • State: closed
  • Created 11 years ago
  • Comments: 26 (14 by maintainers)

Most upvoted comments

@TiagoCardoso1983 - It’s been a bit of a journey, but I’ve got a stable SSE working for my application. I’ll post what I’ve learned below – it may be more than you need to know but hopefully other people can find this a useful primer in the future.

TL;DR

Long story short, building SSEs in Puma requires a fairly decent understanding of concurrency. Generally, this means you must protect concurrent access to shared resources (I/O, memory, etc) via Mutexes.

Problem Definition

Firstly, I am not letting connections die. SSE’s are long running streams of data – reconnections would make the technology no better than AJAX polling. My application is already pretty chatty with an SSE, I certainly don’t want to increase this by using a broken SSE implementation…

Thin (EventMachine)

The main difference between Thin and Puma is the use of threads to enable concurrency. Thin uses a single thread, with EventMachine providing time-sharing capabilities via callbacks to mock concurrency. IE: EventMachine keeps track of the tasks the server has to do and switches between them fairly so that they all get done. This approach shields you from the horrors that come with concurrent programming, but doesn’t take advantage of modern multi-core CPUs. You cannot scale a single process vertically, but you can run multiple processes side by side and use taskset to set the affinity of each process to ensure each runs on a specific core (to take advantage of L1 and L2 CPU caches).

Puma (Threads + Workers)

Puma has a pool of threads for each worker. The worker is essentially a fork of the process, so it will transparently enable the above model of running multiple processes pinned to a CPU. Each forked process (worker) allows you to spawn multiple threads. When hooked up to Rack, Puma will spawn a new thread for every request made to the Web Server. You can also spawn your own threads for things like DB access, file I/O and in-memory collection access. This allows you to scale very naturally with little effort, but you start to discover the evils of threads. Any time a limited resource is shared, issues around contention for that resource will occur. What happens when two threads want to do a good long read of the same file? Similarly, imagine you have a collection stored in memory and two threads attempt to append an item to it at the same instant – which one wins?

Why Threads?

The single-process callback model increases operational complexity in terms of ability to scale these apps. Containers can help out with resource utilization and scheduling across huge clusters of machines, but if you want to scale vertically on a big server you’ll have to manually manage all of this stuff. Have fun when your configuration script accidentally sets the affinity for two of your processes to the same core and response time on your shopping cart submit page goes up >200%.

From a strictly development point of view, scaling across processes is fine until you want to share state across processes. Any information that is to be shared across processes requires a persistence layer (database, Memcache, Redis, whatever). This means that some state information MUST be duplicated (like the connection to the persistence layer) instead of being pooled.

Why Not Threads?

Threads do not have the same limitations on state access – they all have access to the same memory locations. The blessing of concurrency also comes with the curse of contention for resources, so it’s wise to minimize the amount of mutable data that is shared across threads. In general, the naive approach to most applications is to share all the mutable data all the time, everywhere (or at least it is for me!). For me, this means that developing with threads takes a bit longer because of the discipline and rigor involved in both developing and testing.

Good primer on concurrency: https://github.com/jruby/jruby/wiki/Concurrency-in-jruby

Enter the Semaphore

So, threads are great if you want to do something embarrassingly parallel, but what about when you want to do those pesky side-effect ridden things that your crusty, boring old users love to do. Users love mutable data and accessing limited resources. Things like keeping track of a user’s session in memory so they can stay logged in to your dating website for beautiful stock photo models, saving their order for t-shirts to your database so that they’ll eventually receive some hilarious swag to make their co-workers jealous or upload that really cute picture of their cat in a party hat to Facebook over their 3G connection.

To ensure that access to shared resources and mutable state is done safely, the Semaphore was created. The Semaphore (also called a mutex) acts a gate keeper that decides whether a scarce resource is available. Think of the bathrooms at a convenience store (scarce resource) – if you want to access them you must first ask the clerk (semaphore) if they are available. If one is available, they will give you a key, if not you will have to wait. If you’re waiting, the clerk will ask you to queue up until someone returns the key. Once the key is returned, the queue will be processed first-in-first-out until all requests for the bathroom are processed. Sadly, it’s the only bathroom in town and there is a line around the block…

A New Class of Bug

The best part about threads is that they open up an entirely new class of bug: deadlocks( and livelocks). These occur when two semaphores depend on each other. If both semaphores will not change state until the state changes in it’s companion, then you have a deadlock. If both semaphores change state at the same moment, causing a new conflict which results in a recursive series of state changes and resulting conflicts. Returning to the real life example above, imagine there are two bathrooms but the store had to cut costs and there is only one roll of toilet paper and one bar of soap between the two bathrooms. If two people want to use the bathrooms at the same time, this is fine as long as only one needs toilet paper and one needs soap. One day, a customer enters one bathroom with the toilet paper – saying they’re pretty sure they don’t need soap. Another customer enters the other bathroom, stating that they only need soap and it’s fine that the toilet paper is not available. Halfway through their business, both customers change their mind and realize they can’t leave the bathroom until they get the other item (soap or toilet paper). Since neither occupant will surrender their item until they leave the bathroom and neither can leave the bathroom until they can access the others item, both bathroom occupants will eventually die of starvation. This is a deadlock!

For a great example of a livelock, see this wiki article.

Okay, that’s great, but seriously how do I use Puma with Sinatra?

Carefully! If you follow the slightly crazy rant about concurrency above and do some of your own research, you’ll find that reduction of mutable state access across threads and conservative usage of semaphores to protect important access to shared resources will be all the tools you need to replace EventMachine.

There are more things you can investigate if you want, like green threads, network socket event syscalls, etc etc, but ruby is relatively limited in support for all those.

I’ve been stalled for the last 4 weeks, making progress relatively slowly, but still promising, as my main proof of concept has been done. So, for my long-running case:

  1. Along my Puma Server, I start a supervised Celluloid::IO Actor.;
  2. puma worker handles request, hijacks the IO and passes it to the supervised actor;
  3. Actor receives the IO (a TCPSocket) and will transform it into a Celluloid::IO::TCPSocket;
  4. This actor is a request handler, and depending of the request type (chunked, SSE, websocket), will act accordingly;
  1. and 2), 3) are done, 4) is the more time-consuming, as I’d prefer not to reinvent the wheel. When I have time, I’d like to adapt what the reel server already does (regarding request handling), and after that, it should be a piece of cake.

celluloid-io uses nio4r underneath, which is powering the current rails 4 actioncable event loop ditty. The concept they are using is basically the same, they’re just reinventing the wheel a bit because they don’t want the celluloid-suite as a dependency, but to sum it up, they also hijack the socket and pass it to a separate thread running the event loop, and then give the socket to the websocket-driver, which does the whole protocol magic. As I like the abstraction that celluloid-io provides, I prefer to go with it. Will keep you posted.

@dentarg yep, I think this can be closed as resolved.

@jkowens You think we can close this one now after https://github.com/sinatra/sinatra/pull/1858 was merged?

Anyone else on the thread is also free to chime in of course.

Check out https://github.com/sinatra/sinatra/blob/master/lib/sinatra/base.rb#L395-L457

Main issues: it’s dependent of EventMachine-only API (env['async.callback'] is one, the defer call being the other), doesn’t directly use the Rack Hijack API (I think this was due to compatibility reasons, this API exists only since v1.5 of rack, we can forget that for v.2) and relies on a closed callback on the stream/socket (?) object (also eventmachine API). I will see if I can create a WIP PR with proposed changes as soon as I find some time, and we can continue discussing about it.