Blog

Performance Comparison of Ruby Frameworks, App Servers, Template Engines, and ORMs (2016)

Eugene Melnikov

performance-ruby-rails-sinatra-rack

The Ruby ecosystem is constantly evolving. There have been many changes in the engineering world since our comparison of Ruby frameworks in 2014. During the two years, we received a few requests from fellow engineers asking for an updated benchmark. So, here is the 2016 edition with more things tested.

 

What’s under evaluation?

Of course, there are a lot of performance benchmarks on the web. For instance, this one is very good and compares different frameworks while they operate in a viable production configuration.

However, I wanted to understand how Ruby frameworks behave when used as basic solutions with default settings. The idea was to measure only framework speed, not the performance of the whole setup. All my tests were extremely simple, which allowed me to compare almost anything and to avoid side effects caused by very specific optimizations.

So, I implemented bare minimum testing samples for this benchmark and evaluated the performance of:

  • Ruby frameworks (compared with Ruby)
  • Ruby template engines
  • Rack application servers
  • Ruby ORM tools
  • other languages and frameworks

All of the frameworks and tools were tested in the production mode and with disabled logging. The performance of the technologies was measured while they were executing the following tasks:

  • Languages: Print a “Hello, World!” message.
  • Frameworks: Generate a “Hello, World!” web page.
  • Template engines: Render a template with one variable.
  • Application servers: Run successively five simple apps that carry out one action each, such as using database records, showing environment variables, or just sleeping for a second.
  • ORMs: Do different database operations—for example, loading all records with associations, selecting a fragment of a saved JSON object, and updating JSON.

 

Performance of Ruby frameworks

For comparing Ruby frameworks, I chose the ones that are popular enough and quite actively developed: Grape, Hanami, Padrino, Rack, Ruby on Rails, and Sinatra. If you have been following my series of benchmarks, you might have noticed the absence of Goliath on the list. Sadly, it has been more than two years since its latest release, so I did not include the framework into the tests. For all Rack-based frameworks, Puma was used as an application server.

Benchmarking tool. I began my study from selecting a testing tool, and the search was not in vain—I found GoBench. Really, I liked this tool better than ApacheBench, which hasn’t been updated for 10 years. Wrk is also good for local testing. There are more benchmarking tools available, which even allow you to create a cluster.

To get started with GoBench, I ran the gobench -t 10 -u http://127.0.0.1 command and set the concurrency level to 100 (the -c parameter).

Testing samples. In addition, I prepared a number of “Hello, World!” web applications and used them to measure the number of HTTP requests that the technologies under comparison could serve per second.

Performance results. As you might expect, Rack and Sinatra demonstrated the best results.

performance-banchmark-ruby-rack-sinatra-rails-grape-padrino-hanami-v148

A simple, single-thread Ruby server was a bit faster than Rails 5 in API mode, while a modern Rails killer—Hanami—got just a bit better results. In my previous benchmark, Sinatra was three times faster than Rails, and now it is seven times faster. Good progress, Sinatra!

 

Performance of Ruby template engines

The list of the tested template engines includes ERB, Erubis, Haml, Liquid, Mustache, and Slim. Here is the Ruby script I used to measure their performance—the number of templates that can be compiled per second. The image below sums up the results of testing the template engines speed.

performance-testing-ruby-template-engines

Erubis had the best performance among the tested solutions. Slim, as always, was the most compact, and it was also faster than ERB.

 

Performance of Rack application servers

Among the Rack application servers included in the benchmark are Phusion Passenger, Puma, Rhebok, Thin, and Unicorn. Similar to the frameworks, I measured the number of HTTP requests served per second when each of the servers was used. Here, you can find the source code of the tests.

Five sample applications perform the following tasks:

  • active_record shows a list of users from database records.
  • pi calculates the pi number.
  • server lists environment variables.
  • sleep waits one second before responding.
  • random selects which application to start: server, sleep, or pi.

As you can see in the image below, the fastest Rack application server was Passenger, and Rhebok came really close to it.

performance-rack-app-servers-results

Puma, which was recently made the default server in Rails, won just one test. Compared to my previous benchmark, Unicorn has done a good job and now demonstrates results very similar to Thin.

 

Performance of ORMs

In this part of the benchmark, I focused on ORMs and tested how much time Active Record, Sequel, and Mongoid needed to process a given request.

Seven years ago, Jeremy Evans created an amazing benchmarking tool, simple_orm_benchmark. I have updated it and tested Active Record and Sequel with MySQL, PostgreSQL, and SQLite, as well as Mongoid with MongoDB. The six new tests I added to the existing 34 are related to the feature common for all modern databases—the support of JSON types.

The versions that I used in the benchmark include MySQL 5.7.15, PostgreSQL 9.5.4, SQLite 3.14.2, MongoDB 3.2.9, Active Record 5, Sequel 4.38, Mongoid 6, and Ruby 2.3.1.

Here, I include the most interesting results of the tests (fewer seconds means better performance). Red-colored rows in the images below represent measurements with transactions while blue-colored—without transactions. You can find all the performance results for the Ruby ORMs in this CSV file.

Pseudocode for the test: Party.where(id: id).update(:stuff=>{:pumpkin=>2, :candy=>1})

Time needed to perform the test with updating a JSON field:

performance-testing-active-record-sequel-mongoid

Pseudocode for the test: Party.where(id: id).update_all(["stuff = JSON_SET(stuff, '$.pumpkin', ?)", '2'])

Time needed to perform the test with updating a record by a JSON fragment:

performance-ruby-frameworks-orm

Pseudocode for the test: Party.find_by(:id=>id)

Time needed to perform the test with selecting records by a specified attribute:

performance-of-ruby-frameworks-orm-tests

Pseudocode for the test: Party.find_by("stuff->>'$.pumpkin' = ?", '1')

Time needed to perform the test with selecting records by a JSON fragment:

orms-performance-active-record-sequel-mongoid

Despite the fact that Active Record announced the support of JSON types, it is not quite true. In reality, the support only means creating a new JSON object painlessly and getting it back from the database. If you want to update the object in the database, it will override the whole object. To update a part of the object or select the object using a part of it, you have to use raw SQL, which is different for every database.

Moreover, for having the support of JSON fields in SQLite, you should compile SQLite with the json1 module enabled via a SQL query. Alternatively, if you do not need Active Record, you can simply use the Amalgalite gem, because the activerecord-amalgalite-adapter gem is obsolete.


Operations
Mongoid
Active Record
Sequel
Inserting JSON Yes Yes Yes
Reading JSON Yes Yes Yes
Supporting JSON types in models Yes Yes (except SQLite) Yes (only for PostgreSQL)
Updating a JSON object partially Yes Raw SQL Raw SQL
Searching by JSON fragments Yes Raw SQL Raw SQL

In general, Sequel was faster than Active Record and Mongoid, but I cannot say that I enjoyed using it. I really liked working with Mongoid: everything was intuitively clear, and I spent less time implementing the tests compared to Active Record and Sequel.

 

Performance of other languages and frameworks

In the final part of the benchmark, I compared Ruby frameworks with a number of other languages and frameworks, such as Crystal, Python, Elixir, Go, Java, Express, Meteor, Django, Phoenix, and Spring.

Test environment. Similar to the evaluation of Ruby frameworks described earlier, GoBench was used as a testing tool, and the behavior of the frameworks was maximally close to the default. See these “Hello, World” web apps for the testing samples.

Performance results. Here is the surprising part of this benchmark.

performance-benchmark-ruby-frameworks-results-v666

You can see that the Go language and a Go framework Iris are absolute winners that manage to handle about 60,000 requests per second. They are followed by Ring (Clojure), Scalatra (Scala), and Warp (Haskell). Elixir, Scala, and Crystal (a Ruby-like language inspired by the speed of C) share the third place. They are followed by Spring (Java), Hyper (Rust), C, Phoenix (Elixir), Ur/Web, Dart, and Node.js.

Unfortunately, the Ruby frameworks—Rack, Sinatra, Padrino, Grape, Rails, and Hanami—are at the bottom of the benchmark together with Express (Node.js), Stream (Dart), Haskell, Rust, PHP, Java, Python, Meteor (Node.js), and Django (Python).

 

Conclusion

The frameworks and tools in this benchmark were configured as close as possible to their default behavior, which essentially allowed me to create equal conditions for all of them.

Even though Rack, Sinatra, Padrino, Grape, Rails, and Hanami demonstrated relatively low performance compared to other frameworks in our “Hello, World” benchmark, it does not mean that Ruby is the wrong choice. In real life—when you scale your app horizontally, use multiple workers and threads, and know all the advantages of a framework—the results may be very different. It is also true that your preferences and needs may change at some point.

But, first of all, you should enjoy your language. If you don’t, just try something else.

 


Find the full version of this study (16 pages, 17 graphs) in this document.


 

About the author

Eugene Melnikov is a senior software engineer at Altoros. He mainly specializes in Ruby and Ruby-based frameworks, as well as in JavaScript development, including Node.js, jQuery, and AngularJS. Working at Altoros, Eugene has also designed and implemented a variety of SQL and NoSQL database solutions. He recently became engaged in creating data-driven software for IoT applications. Check out Eugene’s GitHub profile.


For the next parts of this series, subscribe to our blog or follow @altoros.

Get new posts right in your inbox!

19 Comments
  • Isn’t Mongoid using BSON? ActiveRecord cannot work with BSON.

    The last benchmark does not make any sense. You are comparing languages and frameworks with different properties. For example, for PHP you are using web server which should be used only for development (why not php-fpm?), nobody uses it for production. Rails you are running in production environment (why not development then?). Go program by default would use all available cores to run the server (since go1.5), and gobench, the tool you are using to run benchmark, uses 100 concurrent request (you haven’t even mentioned that).

    I think your benchmark is very shallow and results do not worth trust.

    • Eugene Melnikov

      Hi Sergey

      > Isn’t Mongoid using BSON? ActiveRecord cannot work with BSON.
      ActiveRecord uses Ruby to decode JSON and to encode its own implementation.

      > I think your benchmark is very shallow and results do not worth trust.
      I think any comparison has its own assumptions. If you run more realistic cases, you will get completely different results, because every language and framework have different implementations, behave differently under a heavy load, and scale differently with a process manager or in a cluster.
      My scenario is as simple as possible, which makes the comparison very synthetic. But I selected this scenario, because I was curious what results I could get without using optimizations that are specific to a particular language or framework (I just disabled logging and enabled production mode for the frameworks). Plus, it was easy to prepare so many examples by implementing such simple applications.

      > why not php-fpm?
      I didn’t use php-fpm because I assume it as a kind of optimization.

      > gobench, the tool you are using to run benchmark, uses 100 concurrent request (you haven’t even mentioned that)
      mentioned)

      • >> why not php-fpm?
        > I didn’t use php-fpm because I assume it as a kind of optimization.
        In text you are saying “All of the frameworks and tools were tested in the production mode”, what makes you think that php -S is production mode?

        >> gobench, the tool you are using to run benchmark, uses 100 concurrent request (you haven’t even mentioned that)
        >mentioned)
        but why not writing the code to be ready to handle 100 concurrent requests? Your java version runs unbounded number of threads and not using NIO or thread pooling. The C version does not even use pthreads. Is it production mode? I thought that on linux using epoll is more wide-spread in production. Ruby has Kernel#select to deal with asynchronous IO. Why did you choose to use pure synchronous IO in some cases, and threaded or even threaded asynchronous in the others? NIO/epoll/Kernel.select all of them accessible out of the box without external dependencies, but you chose not to consider them production enough?

        • Eugene Melnikov

          Now I see what you mean by “production” mode. Meaning that I put into this word is more narrow. Some frameworks have predefined configurations for production, development and test environment, so I used production.

          As for using threads. It was hard to choose between simplicity and performance. I did prefer simplicity for most cases, but for some languages the most simple example on the first page of documentation does support threads. From my point of view if documentation recommend to use threads from first page it’s not an optimization, it’s just basic way to use language.

          I understand that you think it’s unfair and will think about it. May be in the next version of this benchmark all examples will support threads.

          • If you cannot choose between simplicity and performance, why do you put everything on the same chart? You have to render two charts then: “Simple solution, you will never see in the wild” and “Proper solution, which is complex, but similar to production case”. And you will see that almost all your tools/languages/libraries will go into first bucket.

            > The frameworks and tools in this benchmark were configured
            > as close as possible to their default behavior, which essentially
            > allowed me to create equal conditions for all of them.

            equal what? the number of lines? or number of commands in the terminal? Why does it matter at all?

            You call results of you “benchmark” surprising, they are not surprising at all, you are comparing apples to oranges. The chart and “study” does not mean or worth anything.

  • mega_tux

    I found Hanami results somewhat unexpected. I’ll try to test it here. Regards

    • Eugene Melnikov

      I also expected to see Hanami more close to Rails

      • mega_tux

        can you update the results with the above suggestions from the Hanami maintainer?

        • Eugene Melnikov

          I got 1091 hits/sec on the same environment with Luca’s suggestions. I will update images a bit later. Thanks a lot for comments.

    • This was surprising for me as well. There are a couple of issues with the benchmark. Please have a look at the comment above.

  • nori3tsu

    What version of application servers are you using for this benchmark? And is Passenger the enterprise edition?

    • Eugene Melnikov

      Sorry, forgot to mention:
      Phusion Passenger 5.0.30
      Puma 3.6.0
      Rhebok 0.9.3
      Thin 1.7.0
      Unicorn 5.1.0

      You can run using your own versions. Just clone https://github.com/melnikaite/app-server-arena and execute setup.rb then run.rb

      • nori3tsu

        Thank you, I’ll do that. But I want to hear that Passenger used by this benchmark is the enterprise edition? The reason is whether Passenger’s evaluation in this benchmark will change depending on whether it is the enterprise version or not. There is a difference in the thread model between the enterprise edition and the community edition.

        • Eugene Melnikov

          I used community edition, but it’d be interesting to see your comparison results with both editions.

  • Hi there, I’m the author of Hanami.
    There are two issues with the Hanami benchmark:

    1. It’s fundamental to generate the assets manifest. Without it, I get all the requests as not successful. To fix it `HANAMI_ENV=production bundle exec hanami assets precompile`
    2. The server is started with code reloading. Please note that `hanami server` is for development only. For production, I suggest to use directly servers commands (eg. `puma`). To fix this `HANAMI_ENV=production bundle exec hanami server –no-code-reloading`

    The difference in results is huge. It goes from 59 req/s to 1039 req/s.

  • Eugene Melnikov

    Updated images for Hanami and Rust

  • Rushil Agrawal

    Eugene you should try a benchmark with Cuba & Roda. They are popular ruby frameworks and quite a bit faster then the others.

  • Faustino Aguilar

    Hi, good post!

    A question, Did you compile crystal with –release flag?

    Because it allow to optimize a Crystal program a lot.

    By example: https://github.com/tbrand/which_is_the_fastest/pull/10

Benchmarks and Research

Subscribe to new posts

Get new posts right in your inbox!