Testing representational state transfer (REST) APIs can be a tricky business. Many who resort to manually testing REST APIs overlook details and miss out on ways they could save time.

Any REST API service being tested properly will have these areas covered:

  • Testing of whether the API service complies with REST standards
  • Testing of API service response codes
  • Assertion of API service response content
  • Performance benchmarking of API service

Tools for REST test

For this, you need the right tools. Apache JMeter and REST Assured are two popular REST test tools for testing REST API services. You can learn more about how you can use them in practice from here and here.

In this article, you will learn the pros and cons of both of these tools and where they fit best in your existing REST API projects.

Ease of Setting Up

There is one prerequisite: Both JMeter and REST Assured require Java, so you need to have Java installed on your computer in order to run JMeter or REST Assured.

Installing Apache JMeter is very simple: Go to the download page, pick the ZIP or TAR, download the archive, unpack it, and you’re ready to go.

Go to folder where you have unpacked the archive, go to bin subfolder, and run jmeter.bat (or similar for your operating system, if other than Windows). For more detailed information on how to set up Apache JMeter, see this page.

Installing REST Assured is a bit of a different experience. REST Assured is not a standalone tool. It is a Java library. In order to use it, you have to set up a new Java project and include it as a library for your project. Of course, there is support for many build tools, so you can just point your dependency resolver to some public (Maven) repo. You can find more detailed information on how to set up REST Assured here.

Documentation

Apache JMeter documentation can be found on the official JMeter website. It provides some pretty thorough documentation from getting started to complex features such as listeners. Also, in the official JMeter website, you can find links to some very rich resources such as video tutorials. If you are less into reading and more into watching, this is definitely a plus for you.

REST Assured documentation, as the tool/library itself, can be found inside the Github repository. It has a big list of features listed in online documentation with very detailed explanations of how to achieve something. However, the entire documentation is one huge web page—not the most user-friendly way of exploring a tool.

Automated REST Test Creation and Maintenance

Apache JMeter, as a standalone application with a graphical user interface (GUI), allows you to create your tests from the GUI. This can be handy especially because of the entities hierarchy introduced from JMeter—everything you create inside your test will be shown in a tree-like representation in JMeter GUI.

In order to create a test, one does not have to have any coding skills. However, the GUI does not limit any of the possibilities regarding test planning. If you would still like to add some custom programming/flow, BeanShell component(s) will allow you to do that.

JMeter GUI Screenshot

When you save your JMeter project, all that you have created will be saved to one huge XML file (with the .jmx extension). Generally, even for a small change in one of tests, you will (very likely) have to reopen the project in the GUI in order to make the change.

If you need to collaborate with others on the same JMeter project file, this huge XML file will not be easy to maintain using any of source control management (SCM) systems (such as Git, SVN, Perforce, etc).

REST Assured, on the other hand, requires Java coding skills to create the tests. There is no user-friendly GUI for test creation. However, REST Assured allows you to write the tests using Gherkin language (Given-When-Then syntax) so even if you need some Java coding skills, tests are fairly human readable.

given(http://via.placeholder.com/2000x200?text=image+placeholder).
    config(RestAssured.config(http://via.placeholder.com/2000x200?text=image+placeholder).jsonConfig(jsonConfig(http://via.placeholder.com/2000x200?text=image+placeholder).numberReturnType(BIG_DECIMAL))).
when(http://via.placeholder.com/2000x200?text=image+placeholder).
    get(“/price”).
then(http://via.placeholder.com/2000x200?text=image+placeholder).
    body(“price”, is(new BigDecimal(12.12)));

Since REST Assured is a library and tests are actually written in Java, you can organize your tests in separate Java files so when you need to change one test, you just need to open one file. You don’t need a GUI, so you can edit the test easily even through an SSH terminal. Collaboration with others is fairly simple because it is easy to maintain Java files using any SCM.

Types of REST Tests

JMeter is well known as a performance test tool for testing web apps and services. Out of the box, it has nice reporting options with different chart types that you can use to show the performance test metrics.

JMeter Response Times Screenshot

On the other hand, also it is not the primary usage scenario for JMeter. You can create functional tests for REST APIs with JMeter, including response assertion.

REST Assured’s primary usage scenario is for functional testing of REST API services. It even comes with its own response assertion mechanism (the Gherkin syntax). Since REST Assured is a Java library, it can easily be integrated with unit tests.

When it comes to performance testing, REST Assured does not support performance testing out of the box. If you, however, would like to use it for performance testing, you would have to develop your own custom framework for performance testing around REST Assured.

Test Execution

JMeter tests can be invoked in two ways.

From the user interface by clicking on Run button:

JMeter Run Button Screenshot

From the command line:

jmeter -n -t your_script.jmx

The official recommendation from JMeter is to not run the performance tests using the GUI, as the tests will run slower (in terms of number of requests per second) compared to running them from command line.

REST Assured, on the other hand, requires a separate test runner, for example jUnit or TestNG (or any other way for executing Java code).

From Eclipse using TestNG:

REST Assured Eclipse Screenshot

From the command line using, for example, the Ant build tool:

ant run-tests

Types of Reports

Apache JMeter, due to its primary purpose, has a large number of report types that you can create from the results of your tests. Since you can run performance tests from JMeter, performance metrics are usually shown using charts like this:

JMeter Performance Report

You can learn more about JMeter reports and the types of reports you can create from here.

REST Assured itself does not create any reports, so it is up to you and a test framework you chose. However, when integrated with some unit testing framework, you can get pretty usable reports in HTML, text or XML formats.

REST Assured HTML Report

Integration with Jenkins

When it comes to continuous integration of products under test, it is very important to integrate your test automation framework with CI tools for both running tests and reporting on results.

JMeter automated tests can be easily integrated with Jenkins using the Performance plugin which allows you to set up Post-build actions and to get a nice-looking dashboard page out of a Jenkins job page.

JMeter Jenkins Screenshot

REST Assured can also be easily integrated with Jenkins because Jenkins has plugins for almost all testing frameworks; for example, the TestNG plugin which again can create a solid trend chart for test results.

REST Assured Jenkins Screenshot

Community Activity

This a category that cannot be measured easily. Community activity depends on a number of factors such as maturity of the tool or number of contributors.

Apache JMeter has a larger community nowadays. This is mostly because of the fact that JMeter showed up in the market long ago and is considered as industry standard when it comes to performance testing of web apps and services. JMeter publishes official releases in every 5-6 months. Its GitHub repository is very active, with new commits on a daily basis. JMeter uses Bugzilla as issue tracker and has a list of open issues that are frequently changed. There are 23000+ results for JMeter in StackOverflow search. From the social networking sites, JMeter has an official Twitter account. On Facebook and LinkedIn, there is no official account for JMeter, but there are couple of groups, the larger of which is this Facebook group and this LinkedIn group.

REST Assured is cutting its way to the main stage and has become the first choice, a recommendation from multiple sources, when it comes to automating REST API service functional tests. REST Assured has been publishing official releases once every 6 months for the last two years. The GitHub repository has a lots of contributors and is a bit less active than JMeter’s, but it is also very active. REST Assured is tracking issues in the GitHub repository, so all is in one place. In StackOverflow, there are 2000+ search results for REST Assured, so we can say it’s significantly lower compared to JMeter search results. As for the social networking groups, only the LinkedIn group was found with significantly lower number of members compared to JMeter. The REST Assured community may not be as big as the JMeter community, but it is very active.

Summary

Both JMeter and REST Assured have their advantages over each other. But based on the project that you are working on, one will be a better fit than the other for your REST API test automation. Both are very robust, have useful testing options, boast nice reporting capabilities, and can easily be integrated with custom testing frameworks.

Here is a summary of all of the comparison criteria explored in this article for these two popular REST test tools:

  Apache JMeter REST Assured
Requires coding skills No Yes (Java)
Thorough docs Yes Yes
Functional testing Yes (but overkill) Yes
Performance testing Yes No (out of the box)
Suitable for SCM (Git, P4...) No Yes
Test reports Yes No
Integration with CI tools Yes Yes
Integration with unit tests No Yes
Test parallelization Yes Yes
Community activity Yes Yes

Understanding the Basics

How do you use REST Assured?

REST Assured is a Java library that can be imported in a project and used as-is or in combination with other unit testing frameworks.

About the author

Nemanja Veskovic, Serbia
member since November 23, 2016
Nemanja is a passionate QA with more than ten years of experience working with various QA teams and on complex functional and performance testing (manual and automation), creating full continuous integration setups from scratch and making sure QA is implemented in all software development stages from design to deployment. He also has 4 years of experience with QA team lead roles, including interviewing and hiring new team members. [click to continue...]
Hiring? Meet the Top 10 Freelance Java Developers for Hire in April 2018

Comments

Dejan Zivanovic
great article!
Nemanja Veskovic
Thanks Dejan :)
Michael Hicks
Can you elaborate on the issues you've seen with jmx files in git?
Nemanja Veskovic
Hi Michael, Not sure how to edit the article here so I'll reply directly in comment. The issue, or more a difficulty, is the format of the file (xml) which, bmo, is not easily human-readable so reviewing it in a code-review fashioned way is not an easy task. In order to "review" it, you'll probably need to have a Jmeter GUI to work through it. So in case merge conflict ocurrs during collaboration, it could be difficult to resolve conflicts in xml file. Beside that, Jmeter GUI, which is also used as an IDE, does not have an option to resolve conflicts directly from GUI/IDE like all development IDEs have.
Andrei Smirnov
Great article, but why is it in "java" blog? Can't we test any HTTP RESTful services (php, go, node) with these?
Nemanja Veskovic
Hi Andrei, thanks for the opinion. Yes, we can test any RESTful service with both tools, of course. As for the location of the article, that is the question for editors. It will make more sense if the article could be located in some "test" space. Let me reach out to contacts in editors team and see if they can figure out of some alternatives.
comments powered by Disqus
Subscribe
The #1 Blog for Engineers
Get the latest content first.
No spam. Just great engineering posts.
The #1 Blog for Engineers
Get the latest content first.
Thank you for subscribing!
Check your inbox to confirm subscription. You'll start receiving posts after you confirm.
Trending articles
Relevant Technologies
About the author
Nemanja Veskovic
Java Developer
Nemanja is a passionate QA with more than ten years of experience working with various QA teams and on complex functional and performance testing (manual and automation), creating full continuous integration setups from scratch and making sure QA is implemented in all software development stages from design to deployment. He also has 4 years of experience with QA team lead roles, including interviewing and hiring new team members.