A Tutorial for Reverse Engineering Your Software's Private API: Hacking Your Couch

View all articles

Traveling is my passion, and I’m a huge fan of Couchsurfing. Couchsurfing is a global community of travelers, where you can find a place to stay or share your own home with other travelers. On top of that, Couchsurfing helps you enjoy a genuine traveling experience while interacting with locals. I’ve been involved with the Couchsurfing community for over 3 years. I attended meetups at first, and then I was finally able to host people. What an amazing journey it was! I’ve met so many incredible people from all over the world and made lots of friends. This whole experience truly changed my life.

Reverse engineering software is fun with a tutorial and a good project idea.

I’ve hosted a lot of travelers myself, much more than I’ve actually surfed yet. While living in one of the major touristic destinations on the French Riviera, I received an enormous amount of couch requests (up to 10 a day during high season). As a freelance back-end developer, I immediately noticed that the problem with the couchsurfing.com website is that it doesn’t really handle such “high-load” cases properly. There is no information about the availability of your couch - when you receive a new couch request you can’t be sure if you are already hosting someone at that time. There should be a visual representation of your accepted and pending requests, so you can manage them better. Also, if you could make your couch availability public, you could avoid unnecessary couch requests. To better understand what I have in mind take a look at Airbnb calendar.

Lots of companies are notorious for not listening to their users. Knowing the history of Couchsurfing, I couldn’t count on them to implement this feature anytime soon. Ever since the website became a for-profit company, the community deteriorated. To better understand what I’m talking about, I suggest reading these two articles:

I knew that lot of community members would be happy to have this functionality. So, I decided to make an app to solve this problem. It turns out there is no public Couchsurfing API available. Here is the response I’ve received from their support team:

“Unfortunately we have to inform you that our API is not actually public and there are no plans at the moment to make it public.”

Breaking Into My Couch

It was time to use some of my favorite software reverse engineering techniques to break into Couchsurfing.com. I assumed that their mobile apps must use some sort of API to query the backend. So, I had to intercept the HTTP requests coming from a mobile app to the backend. For that purpose I set up a proxy in the local network, and connected my iPhone to it to intercept HTTP requests. This way, I was able to find access points of their private API and figure out their JSON payload format.

Finally I created a website which serves the purpose of helping people manage their couch requests, and show surfers a couch availability calendar. I published a link to it on the community forums (which are also quite segmented in my opinion, and it’s difficult to find information there). The reception was mostly positive, although some people didn’t like the idea that the website required couchsurfing.com credentials, which was a matter of trust really.

The website worked like this: you log in to the website with your couchsurfing.com credentials, and after a few clicks you get the html code which you can embed into your couchsurfing.com profile, and voila - you have an automatically updated calendar in your profile. Below is the screenshot of the calendar and here the articles on how I made it:

I’ve created a great feature for Couchsurfing, and I naturally assumed that they would appreciate my work - perhaps even offer me a position in their development team. I’ve sent an email to jobs(at)couchsurfing.com with a link to the website, my resume, and a reference. A thank-you note left by one of my couchsurfing guests:

A few days later they followed up on my reverse engineering efforts. In the reply it was clear that the only thing they were concerned about was their own security, so they asked me to take down the blog posts I’ve written about the API, and eventually the website. I’ve taken down the posts immediately, as my intention was not to violate the terms of use and fish for user credentials, but rather to help the couchsurfing community. I had an impression that I was treated as a criminal, and the company focused solely the fact that my website requires user credentials.

I proposed to give them my app for free. They could host it on their environment and connect it through Facebook authentication. After all, it is a great feature, and the community needed it. Here is the final resolution I received:

“We are getting back into the swing of things here after the holidays and wanted to follow up.

We have had some internal discussion about your application and how we could both honor the creativity and initiative it shows while not potentially compromising the privacy and security of Couchsurfing users’ data when they enter their credentials into a third-party site.

The calendar clearly fills a feature hole on our site, a feature that is part of a larger project that we are working on now.

But the issue of collecting usernames and passwords remains. We couldn’t come up with an easy way to set it up so that we could host or support that on our side without either allowing you to access that data or have your site be seen as our work product.

The API that is currently available is soon to be replaced with a version that will require authentication/authorization from applications that access it.”

Today while I’m writing this reverse engineering software tutorial (a year after the events), the calendar feature is still not implemented on Couchsurfing.

Return To Innocence - Hacking My Couch, Again

A few weeks ago I was inspired to write an article about the techniques to reverse engineer private APIs. Naturally, I decided to summarise the previous articles I’ve written on this topic, and add few more details. As I started to write the new article I wanted to showcase the reverse engineering process with an up-to-date API and take another stub into API hacking. Based on my previous experience, and the fact that Couchsurfing recently announced a completely new wesbite and mobile app http://blog.couchsurfing.com/the-future-of-couchsurfing-is-on-the-way/, I’ve decided to hack their API again.

Why am I doing this reverse engineering process? Well, first of all it’s a lot of fun to reverse engineer software in general. What I particularly like about it, is that it doesn’t involve just your technical skill, but also your intuition. Sometimes, the best way to figure things out is to make an educated guess - it’ll save you lots of time compared to brute-force. Recently I heard a story from a company which had to work with proprietary APIs and little or no documentation. They’d been struggling to decrypt the API response payload in an unknown format for days, then someone decided to try ?decode=true at the end of the url and they had a proper JSON.

Another reason I’m doing this tutorial is that it takes ages for some companies to adopt a particular feature requested by their users. Rather than waiting for it to be implemented, you can harness the power of their private API and build it yourself.

So, with the new couchsurfing.com API, I started with a similar approach and I installed their latest iOS app.

First, you need to set up a proxy in your LAN to forge HTTP requests coming from the app to the API by performing a man-in-the-middle attack (MITM).

For unencrypted connections the attack is quite simple - a client connects to the proxy and you relay incoming requests to the destination server back and forth. You could possibly modify the payload, if necessary. In a public WLAN, it’s fairly easy to perform this under disguise by impersonating the WiFi router.

For encrypted connections, there is a minor difference: all the requests are encrypted end-to-end. it’s not possible for the attacker to decrypt the message, unless he somehow gets access to the private key (which of course is not sent during these interactions). Having said that, even though the API communication channel is secure, the endpoints - especially the client - are not that safe.

The following conditions have to be met in order for SSL to work properly:

  • The server’s certificate has to be signed with a trusted certificate authority (CA)
  • The server’s common name, in the certificate, must match the domain name of the server

To overcome the encryption in a MITM attack, our Proxy need to act as a CA (Certificate Authority) and generate certificates on the fly. For example if a client tries to connect to www.google.com the proxy dynamically creates a certificate for www.google.com and signs it. Now, the client thinks that the proxy is in fact www.google.com

This diagram outlines the steps to reverse engineer a private API.

To implement a sniffing proxy used to reverse engineer the private API, I’ll use tool called mitmproxy. You can use any other transparent HTTPS proxy. Charles is another example with a nice GUI. To make this work we need to set up the following things:

Configure your phone’s WiFi connection default gateway to be the proxy (so that the proxy is in middle and all the packets pass through) Install proxy’s certificate on the phone (so that the client has the proxy’s public key in its trust store)

Check your proxy’s documentation about installing the certificate. Here are the instructions for mitmproxy. And here is the certificate PEM file for iOS.

To monitor intercepted HTTP requests, you simply launch mitmproxy and connect to it from your mobile phone (default port is 8080).

Open a website in your mobile browser. At this point you should be able to see the traffic in mitmproxy.

Once you have confirmed everything is working, the reverse software engineering can begin.

Once you make sure that everything works as planned, it’s time to get started exploring the private API of your choice. Basically, at this point you can just open the app, play around with it and get an idea about API endpoints and request structure.

There is no strict algorithm on how to reverse engineer a software API - most of the time you rely on your intuition and make assumptions.

My approach is to replicate the API calls and play with different options. A good start is to replay a request you caught in mitmproxy, and see if it works (Press ‘r’ to replay a request). The first step is to figure out which headers are obligatory. It’s pretty convenient to play with headers with mitmproxy: press ‘e’ to enter editing mode, then ‘h’ to modify headers. With the shortcuts they use, vim addicts would feel right at home. You can also use browser extensions like Postman to test the API, but they tend to add unnecessary headers, so I suggest sticking to mitmproxy or curl.

I’ve made a script that reads mitmproxy dump file and generates a curl string - https://gist.github.com/nderkach/bdb31b04fb1e69fa5346

Let’s start with the request sent when you are logging in.

POST https://hapi.couchsurfing.com/api/v2/sessions
← 200 application/json

The first step in this reverse engineering tutorial is to replicate the API calls and play with the resulting options.

The first thing I noticed is that every request contains a mandatory header X-CS-Url-Signature which is different every time. I also tried to replay a request after a while to check if there is a timestamp check on the server, and there is none. Next thing to do is to figure out how this signature is calculated.

At this point I decided to reverse-engineer the binary and figure out the algorithm. Naturally, having experience developing for iPhone and having an iPhone at my disposal, I decided to start with the iPhone ipa (iPhone app deliverable). Turns out to decrypt one, I need a jailbroken phone. Stop! Hammer Time.

Then, I recalled that they have an Android app as well. I was a bit hesitant to try this approach, as I don’t know anything about Android or Java. I then thought it would be a good chance to learn something new. It turned out to be easier to get a human-readable quasi source code by decompiling java bytecode than heavily optimized iphone machine code.

Apk (Android app deliverable) is basically a zip file. You can use any zip extractor to unpack its contents. You will find a file called classes.dex, which is a Dalvik bytecode. Dalvik is a virtual machine used to run translated Java bytecode on Android.

To decompile the .dex file into .java source code I used the tool called dex2jar. The output of this tool is a jar file, which you can decompile with a variety of tools. You can even open a jar in Eclipse or IntelliJ IDEA and it will do all the work for you. Most of these tools produce a similar result. We don’t really care if we can compile it back to run it, we are merely using it to analyze the source code.

Here is a list of tools I’ve tried:

CFR and FernFlower worked the best for me. JD-GUI was unable to decompile some critical parts of the code and was useless, while the others were about the same in quality. Luckily, it seems that the Java code code hasn’t been obfuscated, but there are tools like ProGuard http://developer.android.com/tools/help/proguard.html to help you deobfuscate the code.

Java decompilation is not really the scope of this reverse engineering tutorial - there is a lot written on this topic, so let’s assume you successfully decompiled and deobfuscated your Java code.

I’ve combined all the relevant code used to calculate X-CS-Url-Signature in the following gist: https://gist.github.com/nderkach/d11540e9af322f1c1c74

Firstly, I’ve searched for mentions of X-CS-Url-Signature, which I’ve found in RetrofitHttpClient. One particular call seemed interesting - to EncUtils module. Digging into it, I realized that they are using HMAC SHA1. HMAC is a message authentication code which uses a cryptographic function (SHA1 in this case) to compute a hash of a message. It’s used to ensure integrity (i.e. to prevent a man in the middle from modifying the request) and authentication.

We need two things to calculate the X-CS-Url-Signature: the private key and the encoded message (probably some variation of HTTP request payload and URL).

final String a2 = EncUtils.a(EncUtils.a(a, s));
final ArrayList<Header> list = new ArrayList<Header>(request.getHeaders());
list.add(new Header("X-CS-Url-Signature", a2));

In the code a is a message and s is the key which are used to compute the header a2 (the double call to EncUtils just computes an HMAC SHA1 hex digest).

Finding the key wasn’t a problem - it was stored in plain text in ApiModule, and was used to initialize the second parameter of RetrofitHttpClient.

RetrofitHttpClient a(OkHttpClient okHttpClient) {
    return new RetrofitHttpClient(okHttpClient, "v3#!R3v44y3ZsJykkb$E@CG#XreXeGCh");

If we look at the call to EncUtils, we can see that the string literal above is used verbatim as a key to calculate the HMAC, except in the case when this.b is defined. In the latter case, this.b is being appended with a dot to it.

String s;
if (this.b == null) {
    s = this.a;
else {
    s = this.a + "." + this.b;

Now, just by looking at the code it wasn’t clear to me where and how this.b is initialized (the only thing I was able to discover is that it’s called in a method with a signature this.a(String b), but I couldn’t find a call to it anywhere in the code).

public void a(final String b) {
    this.b = b;

I encourage you to decompile it and find out yourself :)

Figuring out the message was pretty straight forward - in the code you can see that’s it’s a concatenation of the url path, i.e. /api/v2/sessions and a string with JSON payload (if any).

final byte[] b = this.b(request.getUrl());
byte[] a;
if (request.getBody() != null && request.getBody() instanceof JsonTypedOutput) {
    // this.a(x, y) concatenates byte arrays  
    a = this.a(b, ((JsonTypedOutput)request.getBody()).a);
else {
    a = b;

Just by looking at the code, it was difficult to figure out the exact algorithm for HMAC calculation. So, I decided to rebuild the app with debugging symbols to figure out exactly how the app works. I’ve used a tool called apktool https://code.google.com/p/android-apktool/ to dissasemble the Dalvik bytecode using smali https://code.google.com/p/smali/. I followed the guide at https://code.google.com/p/android-apktool/wiki/SmaliDebugging

After you build the apk you need to sign and install it on your device. As I didn’t have an Android device, I used the emulator which comes with Android SDK. With some spoonfeeding, here is how you do it:

jarsigner -verbose -keystore ~/.android/debug.keystore -storepass android -keypass android <path_to_your_built_apk> androiddebugkey

jarsigner -verify -verbose -certs <path_to_your_built_apk>

zipalign -v 4 <path_to_your_built_apk> <path_to_your_output_signed_apk>

I used a built-in Android emulator which comes with the sdk and an Atom x86 virtual image with HAXM enabled to ensure it runs smoothly.

tools/emulator -avd mydroid -no-boot-anim -cpu-delay 0

Here is a nice guide on how to set up a virtual image: http://jolicode.com/blog/speed-up-your-android-emulator

Make sure you see the line HAX is working and emulator runs in fast virt mode on emulator startup to make sure you have HAXM enabled.

Then, I installed the apk into the emulator and ran the app. Following the apktool guide, I leveraged IntelliJ IDEA remote debugger to connect to the emulator and set some line breakpoints:

Some reverse engineering techniques involve running the app and just seeing what happens.

Playing with the app for a little bit, I was able to figure out that the private key used to initialize RetrofitHttpClient is used for calculating HMAC of a login request signature. In the response to the login POST you receive a user ID and accessToken (X-Access-Token). The access token is used to authorize all the following requests. The HMAC for all the post login requests is constructed the same way as the login request, except that the key is composed by appending .<user_id> to the original private key.

This shows the authorization process necessary to reverse engineer this private API.

Once you are authorized, the app sends the following request:

POST https://hapi.couchsurfing.com/api/v2/users/1003669205/registerDevice
← 200 application/json

As I was able to empirically deduct, this request is optional for authentication. Bonus points if you figure out what it’s used for!

Once authenticated you can send a request to fetch your (or anyone else’s user profile), like this:

GET https://hapi.couchsurfing.com/api/v2/users/1003669205
← 200 application/json

In this reverse engineering process you can fetch anyone’s user profile.

I didn’t go much into details, but I noticed that a profile is updated with a PUT request. Just for fun, I tried to update another profile with the same request - it wasn’t authorized, so apparently the security basics are implemented.

I wrote a simple Python script to login using your couchsurfing.com credentials and get your user profile: https://gist.github.com/nderkach/899281d7e6dd0d497533. Here is the Python wrapper for the API: https://github.com/nderkach/couchsurfing-python with a package available in pypi repository (pip install couchsurfing).

Next Steps

I’m not sure what exactly I’m going to do with the API this time. HTML code in user profiles is no longer allowed, so I’ll have to come up with a different approach to the old problem. I’ll continue developing and enhancing the python API wrapper, if there is a demand for it, and assuming that couchsurfing.com will not cause too many problems. I didn’t explore the API too much, and I just tested it for some basic vulnerabilities. It seems secure enough, but it would be interesting to find out if you can get access to the data which is not available through the website. Either way, now you can use my reverse software engineering to build an alternative client for Windows Phone, Pebble, or your smart-couch.

Wrap-up With A Question

There is a discussion I’d like to open - why not publish your API and make it public? Even if I didn’t manage to hack the API, it would still be possible to scrape the website. It would be slower and more difficult to maintain, but surely they’d prefer consumers to use an API rather than a web scraper. Availability of the APIs would allow third-party developers to improve the company’s product, and build value-added service around it. One can make an argument that it would be more expensive to maintain the public API rather than a private one; but then again, the advantages of your community building services on top of your product would outweigh the API maintenance costs.

Is it possible to completely prevent usage of a private API by third-party clients? I don’t think so. Using SSL pinning would prevent sniffing into API requests using a simple transparent proxy technique as described earlier. In the end, even if you obfuscate the binary, a motivated hacker with some resources and time will always be able to reverse-engineer the app binary and get the private key/certificate. I think the assumption that the client endpoint is secure is inherently wrong. An API client is a weak spot.

By keeping an API private, a company is basically conveying a message of mistrust to their users. Surely, you can try to protect your private API even further. However, wouldn’t you rather implement a basic security for the API to prevent malicious usage; and instead focus your resources on improving the software to provide a better user experience?

Couchsurfing, pretty please, with sugar on top, open the API.

About the author

Nikolay Derkach, United States
member since November 27, 2013
Nikolay is a software developer and entrepreneur. He is an accomplished full-stack developer with experience in building web application backends and designing APIs, as well as iOS apps. He has worked extensively with database infrastructures and built full-stack iOS applications, and he is an acknowledged expert in reverse-engineering private APIs. [click to continue...]
Hiring? Meet the Top 10 Freelance Back-End Developers for Hire in October 2016


Sandy Vanderbleek
This changed my life. Thank you so much!
Dario Klaric
Nice post. I had much better luck doing the same in financial industry which should be more serious industry than traveling around. I needed couple of years (5 if I am not wrong) that someone understands me, but when I got to the right person (who understood what I'm speaking), everything become easy. I hope you will succeed at the end.
Sergey Tolmachev
Nice article! I've never heard about the smali debugging, it's interesting solution!
Jacky Liang
Awesome writeup. Is that a third-party console application? Love the font and color scheme.
Thanks! That's iTerm2 with Solarized Dark color scheme.
Thanks for satisfying my curiosity about how to use a MITM proxy to deconstruct an API. I'm keen to try!
Spencer Y
One thing I've pondered is whether there is some new legal territory to be explored when companies have hackable API's. For example, if someone puts material out on the Internet, but does not claim copyright to it, it is quasi-legal (http://en.wikipedia.org/wiki/Copyright) for you to reuse the material. What is the equivalent to "copyright" for an API?
Miguel Carrillo
Hi Nikolay, I like your post a lot, thanks for sharing! It is very interesting how you started talking about your passion about couchsurfing and then went into the technical aspects linking them together very smoothly. By the way, here is something I read in your article: "Once you are authorized, the app sends the following request: POST https://hapi.couchsurfing.com/api/v2/users/1003669205/registerDevice ← 200 application/json As I was able to empirically deduct, this request is optional for authentication. Bonus points if you figure out what it’s used for!" In Android, in order to receive push notifications on your mobile for a given app you need to register the device with GCM (Google Cloud Messaging). This way your back-end can send the push notifications to your Android app. It seems natural that after authenthication the next step was to register the device, hence the call to 'registerDevice'. Referece: https://developer.android.com/google/gcm/client.html#sample-register
You, sir, get a bonus point! :)
Hi! This is very cool - thanks for sharing. Currently I am reverse-engineering a web API which is pretty simple with a simple packet capture. I just want to let you know why a company would NOT make their API public: - Once it's public, you're bound to it. Changing becomes difficult. Came up with a better API? Listen to the sounds of thousands of apps breaking... this is what any company that has experience with building platforms has learned. That's why Apple is very reluctant to allow access to various parts of iOS. - It's difficult to maintain a public API - in part because others depend on it, in part because when it's so hard to change you have to be really sure in what you're doing. - Prevent hacking. That's the bicycle lock theory - any bicycle lock can be broken, but having one makes it more difficult to steal your bike. Anything that makes hacking your site harder is a plus even if all APIs can ultimately be hacked. I know you won't like to hear that - but do you lock your bicycle? See?! :) - You might not want 3rd party clients on your server - you might want to keep control of that. Simplest example is your client is also an ad platform, and you make money showing ads. Or you don't do that now, but want to keep the option in he future.
Dylan Pierce
> Using SSL pinning would prevent sniffing into API requests using a simple transparent proxy technique as described earlier. Similar to your story, I reverse engineered another private API with a lot of help. Like you in this story I haven't dealt with any Java or Android code before. This particular app's android client no longer transmits certain requests through my mitmproxy or burp proxies. This probably means they're cert pinning for these requests. Is it possible to force an CA on the device? Or am I better off trying to decompile the code and figure it out from there? I already have the HMAC key, in a similar auth scheme you describe.
Isuru Nanayakkara
Very cool. Downloading and decompiling apps that I like is kind of a hobby for me too. I'm an iOS developer myself but I choose to go with Android version because of the same reasons you've mentioned. The thing is I get bored easily. So after opening up the Android project and looking around in the source, I kinda throw it away. Your post inspired me to work on it more. It's fun.
Josh MacDonald
What if "X-CS-Url-Signature" was not found in the source? What would be the next steps?
comments powered by Disqus
The #1 Blog for Engineers
Get the latest content first.
No spam. Just great engineering and design posts.
The #1 Blog for Engineers
Get the latest content first.
Thank you for subscribing!
You can edit your subscription preferences here.
Trending articles
Relevant technologies
About the author
Nikolay Derkach
Python Developer
Nikolay is a software developer and entrepreneur. He is an accomplished full-stack developer with experience in building web application backends and designing APIs, as well as iOS apps. He has worked extensively with database infrastructures and built full-stack iOS applications, and he is an acknowledged expert in reverse-engineering private APIs.