04.07.2014

Keep your clients simple with real REST and HATEOAS

The last months we were moving some backend services from an old legacy architecture to a new scala based service architecture, already having an iPhone app that uses the old services.

The legacy services exposed their API via HTTP and JSON in a format that has been copied from Facebook (and Google). For example look at the URL for getting questions with user information who created the questions:

/question?fields=question.title,user,user.nickname,....

The facebook idea is highly flexible, no doubt. Allowing you to tailor down the response the exact way you want it. You get exactly what you want and you're able to build clients that use the API to get new relations without the need to change the server side.

As we tried to re-write these services in Scala but staying backward compatible, we ran into the downsides of this approach:

Downside 1: Infinite number of URL query parameter variations

It was quite a hard job to find out which URL variations our current sole and own (!) client (iPhone App) uses from this API, because the above approach allowed so many possible variations of the URL. And sure, as we finally switched we broke the App because we've missed one call.

Downside 2: The client has to have explicit knowledge how to build URL's

The facebook approach builds an implicit contract between clients and the API for how to use the API. If the API URL format changes you'll now have to worry about supporting old clients using the old URL format.

Downside 3: Infinite number of response formats 

The flexible URL format as proposed by Facebook leads to responses being as dynamic as the URL parameters, so it was hard to find out which sets of responses are used by the client and have to be re-implemented to keep backward compatibility.

Alternative solution

We're currently writing a brand new Android App in Scala for sure, stay tuned for upcoming infos) and we don't want to go down the same error prone way as the months before. So for the new services (as a replacement for the old legacy one) we try to learn from the above problems.

Use fixed set of responses

We've introduced shapes for resources (like /users or /questions). These shapes are T-Shirt sizes: S, M, L, XL. The different shapes return you a different amount of associated data for your resource. For example M for a question would return a question with some inlined user information. We're thinking about allowing shapes for associated resources as well. E.g.

/questions?shape=M&user.shape=S

(To be honest, I'm not really a fan of inlining associated resources (even with the above shape way) because it contradicts the REST principles, but we want to reduce the amount of requests for the mobile clients here.)

With the introduction of shapes you have a reasonable amount of flexibility to customize the amount of data in your response and the query costs on the server side but you now have a stable set of response formats you have to support in the future.

This shape idea adresses downside 1 and 3 from the above list, but what about clients having to know how to build URL's?

Use resource links

Our current task was to implement paging for a specific resource stream. Up to now the old and also the new services used an approach also inspired by Facebook (and a lot more):

/questions?limit20&offset=100

We encountered problems when paging here, because a lot of new questions are created every second, so using limit and offset is no fixed but a moving window and the paged responses are eventually including already returned questions in another page. With this approach the client has to filter the paged responses for already returned responses. This is a really heavy burden for a client as you can imagine.

Also the client again has to know the logic how to build the paging URL as seen above. If you want to change that format in the future, you'll have big problems, because there are a lot of clients out there that use your old format.

So we decided to move on to an alternate solution:

We move the logic how to get the previous or next paged response from the client back to the server. The server as he returns paged responses now simply returns two resource links with the relation name 'previous' and 'next'. All the client has to do to get a new paged response before or after the current one is: Follow one of the links! This is just the same idea as a human uses the WWW. You don't worry about how the URL looks like on a page that you read, you just click the link and your browser follows...

The benefits of using resource links here are:

The server encodes the state for the paging inside the relation links, so this is what the HATEOAS (Hypermedia As The Enginge Of Application State) principle of REST is all about: The state for paging is now encoded in the returned links.

Now that the state is inside the links you have more advantages:
  • If you decide to use timestamps for paging instead of Id's (as above), you simply change it on the server side. You don't brake your clients because your clients just follow the links
  • The client doesn't anymore need to know how to build paging URL's, you can eliminate that logic from your client
  • As you can see we also changed to now use beforeId or afterId params together with limit to eliminate duplicates in different pages (which happened as we use the offset param). But as we moved the pagination links generation to the server, the client doesn't have to have the logic for finding the last or first id of a page to get a new one