Changelog: The OWASP Top 10 Project

Web applications have exploded in complexity over the past decade. They have evolved from simple containers for contact forms and polls into full-blown applications. We can compare them to the heavy desktop applications, both in size and performance. With a steep rise in complexity and an increasing number of feature-rich applications, it has become a necessity to invest a lot of time and care into making all application components as secure as possible. The massive rise of Internet users has made it even more important to tackle the issue of protecting the data and application users. There is a vast pool of threats trying to creep in and cause severe headaches to everyone involved.

In 2001, a new organization entered the stage. Its goal was to fight the security problems affecting websites and applications. It was appropriately named Open Web Application Security Project (OWASP). Nowadays, it publishes resources, organizes conferences, and proposes standards on web and application security. A de facto standard for web application security is the OWASP Top Ten Project. It lists the ten most prevalent security threats. Influential factors in deciding what gets in were an extensive amount of data and community feedback. In late 2017, there was an update of the project. Several new issues critical for a lot of modern web applications received their place, while some have escaped from the list.

This article supplements the original list and illustrates the latest changes to list. It describes the threats, tries to provide clear examples for easier understanding, and proposes ways of fighting security threats.

Issues Removed from the OWASP Top 10 List

Before the 2017 update, the list from 2013 was the most recent one. Considering the changes in the ways web applications are now built and consumed, it only made sense to do a thorough revision. Microservices are taking their piece of the pie, and new cool and shiny frameworks are replacing vanilla code battle gear. These facts mean that some of the threats listed before got removed and some new ones took their place.

We’ll make sure to refresh our memory of the long forgotten issues in this article, as well as introduce the new bad wolves. Learning about history is the only sure way of not repeating the same mistakes.

Cross-site Request Forgery

Cross-site request forgery (CSRF) is one of the “losers” in the recent iteration of the project. It went away because a lot of modern web frameworks include CSRF defense mechanisms. The probability of exposing your applications to the threat thus decreases rapidly.

Regardless of CSRF exiting the list, it’s still good to refresh our memory. Let’s make sure it doesn’t come back stronger than ever.

In essence, CSRF is an exploit which feels a smoke bomb. An attacker tricks an unsuspecting user to execute an unwanted request or action within a web application. Simply put, an attacker forces its victim to send a request to a third-party application, and the victim is unaware of the request ever being sent. The request could be an HTTP GET request to retrieve a resource, or even worse, an HTTP POST request which changes a resource under victim’s control. During the attack, the victim thinks that everything is fine, most often without even noticing that something is happening in the background. After the air clears, the damage is done or something is missing, and nobody knows what had happened.

Successful previous user authentication within the target application is what enables the trap to be effective. The user had at some point before the attack signed into an application. The application sent the victim a cookie to remember them. Once the web browser sends the malicious request, the cookie is automatically sent along with any potential payload and the application doesn’t object to serving the request to a user it knows already.

One of the most famous examples is tricking a user to transfer money from their account to the one an attacker controls. A user signs into an e-banking system to check their account balance. After that, they visit an online forum to check the reviews of a new mobile phone. An attacker, fishing with dynamite, posts a review including an image with a seemingly broken image hyperlink. Instead of a real hyperlink, the attacker uses a hyperlink which the e-banking system internally uses to transfer money from account A to account B: https://payments.dummybank.com?receiver=attacker&amount=100. The banking system makes the authenticated user the sender and the value from the “receiver” parameter the receiver of the funds. Of course, the attacker specifies their offshore account as the receiver.

Since the browser automatically loads images when rendering the page, the request happens in the background. If the bank’s payment system implements money transfers using an HTTP GET request, nothing is stopping the disaster from happening. Note that the example is simple and most probably transfers aren’t handled via HTTP GET. However, the attacker can, with only a little bit more difficulty, manage to change the attribute “action” in the forum’s HTML message posting form. The browser then sends the request to the bank’s payment system, instead of the forum’s back-end.

Stealing money is only one of the many examples out there. Changing users’ email addresses or making unintended purchases fall into this category as well. As it often happens, social engineering and some technical knowledge are effective leverage against a software engineering mistake.

When designing your systems, keep the following in mind:

• Do not use HTTP GET requests for encapsulating actions which modify a resource. You should only use these requests for retrieving information. Remember, the rule of a thumb is to make GET requests idempotent.
• Do, when transferring data internally using HTTP POST requests, tend to send the data in JSON, XML or some other format other than encoding the parameters as a query string. Using a non-trivial data format reduces the danger of someone creating a fake HTML form which will send the data to your service.
• Do make sure to create and include a unique and unpredictable token into your HTML forms. These tokens should also be unique for each request. Checking the presence and correctness of such tokens will lower the risks of threats occurring. To find out the token and use it in their fake requests, attackers would need to access your system and take a token directly from there. Since the tokens are one-time only, they can’t re-use them in the malicious code.

This vulnerability has even worse effect when coupled with cross-site scripting (XSS). If an attacker can inject malicious code into a favorite website or application, the scope of the attack becomes much more significant and dangerous. Even more critical, attackers can circumvent some of the protection mechanisms against CSRF if XSS attacks are possible.

Keep in mind that CSRF hasn’t vanished, it’s just not as common as it used to be.

Unvalidated Redirects and Forwards

A lot of applications, after finishing an action, redirect or forward a user to another part of the same, or even some other, application. For example, successfully signing into an application triggers a redirect to the home page or the page the user initially visited. Very often, the destination is part of the form’s action or links address. If the component handling the redirect or forward doesn’t make sure that the destination address is indeed the one that was generated by the application, a potential threat arises. This is a security vulnerability called “unvalidated redirects and forwards.”

The two major reasons why unvalidated redirects and forwards would ever be considered dangerous are phishing and credential hijacking. An attacker can manage to alter the redirect/forward target location and send a user to a malicious application almost indistinguishable from the original one. An unsuspecting user reveals their credentials and confidential information to a malicious third party. Before they realize what had happened, it’s already too late.

As an example, web applications very often implement sign-in with support for redirects to the last visited page. To be able to do this easily, an HTML form’s action attribute might look something like http://myapp.example.com/signin?url=http://myapp.example.com/puppies. You are a huge admirer of puppies, so it made sense to install a browser extension which replaces website favicons with the thumbnails of your favorite puppies. Unfortunately, it’s a dog-eat-dog world. The author of the browser extension decided to cash in on your unconditional love and introduce an additional “feature.” Each time you visit your favorite puppy fan site, it replaces the “url” parameter in the form’s action attribute with a link to their own site. Since the site looks exactly the same, when you give your credit card details to buy puppy playing cards, you are in fact financing a malicious attacker, not your hobby.

Solving the vulnerability involves checking the destination location by making sure it’s the intended one. If a framework or library does the complete redirect or forward logic, it’s beneficial to check the implementation and update the code if necessary. Otherwise, you need to make manual checks to protect against the attack.

There are several types of checks you can make. For the best protection, use a combination of several approaches instead of sticking with only one of them.

• Validate the outgoing URL making sure it is pointing to the domain you control.
• Instead of using explicit addresses, encode them on the front-end and then decode and validate them on the back-end.
• Prepare a whitelist of trusted URLs. Allows forwards and redirects only to the whitelisted locations. Prefer this approach to maintaining a blacklist. Blacklists are usually filled with new items only when something bad happens. Whitelists are more restrictive.
• Employ an approach used by LinkedIn and some other applications: Present your users with a page asking them to confirm a redirect/forward, making it clear they are leaving your application.

Merged Issues

Most of the issues on the list can be labeled as defects in the implementation, either due to lack of knowledge or insufficiently deep investigation of potential threats. Both of these reasons can be attributed to the lack of experience and the solution for considering them in the future is easy—just make sure to learn more and be more thorough. An especially tricky one relies on a combination of the dangerous (but very human) trait of making too many assumptions coupled with the difficulty of developing and maintaining complex computer systems. A vulnerability that fits into this category is “broken access control.”

Broken Access Control

A vulnerability is caused by inadequate, or complete lack of, authorization and access control for certain parts of the application. In the previous iterations of the OWASP Top Ten Project, there were two issues: insecure direct object references and missing function-level access control. They are now merged into one due to their similarity.

Direct Object References

Direct object references are often used in URLs to identify resources being operated on. For example, when a user signs in, they can visit their profile by clicking on a link which contains their profile identifier. If that same identifier is stored in the database and is used to retrieve profile information, and the application assumes that people can get to profile page only via a sign-in page, changing the profile identifier in the URL exposes profile information of another user.

An application which sets the URL of the delete profile form http://myapp.example.com/users/15/delete makes it clear that there are at least 14 other users within the application. Figuring out how to get to the delete form of other users isn’t rocket science in this case.

The solution to this issue is to perform authorization checks for each resource without assuming that only certain paths can be taken to get to some parts of the application. In addition, removing direct references and using indirect ones is another step forward because it makes it difficult for malicious users to figure out how the reference is created.

During development, as a precaution, write down a simple state machine diagram. Let the states represent different pages within your application and transitions the actions users can take. This makes it easier to list all transitions and pages which need special attention.

Missing Function-level Access Control

Missing function-level access control is very similar to insecure direct object references. In this case, users modify the URL or some other parameter to try to access protected parts of the application. If there is no proper access control examining levels of access, a user can gain privileged access to application resources or at least some knowledge of their existence.

Borrowing from the example for direct object references, If an application assumes that a user visiting the delete form is a superuser only because superusers can see the link to the delete form, without making any further authorization, a huge security flaw opens. Counting on the availability of some UI element is not proper access control.

The issue is solved by always making sure to perform checks in all layers of your application. The front-end interface might not be the only way malicious users can access your domain layer. Also, don’t rely on information passed from users about their access levels. Perform proper session control and always double-check the received data. Just because the request body says the user is an admin, it doesn’t really mean they are.

Broken access control now combines all issues which are related to insufficient access control, be it on the application level or the system level, like a misconfiguration of the file system.

New Issues in the OWASP Top 10 List

The advent of new front-end frameworks and adoption of new software development practices shifted the security concerns to completely new topics. New technologies also managed to solve some common issues we were dealing with manually before. Therefore, it became beneficial to revise the list and adjust it accordingly to modern trends.

XML External Entities

The XML standard offers a little-known idea called an external entity, which is part of the document’s data type definition (DTD). It allows document authors to specify links to outside entities which can then be referenced and included in the main document. A very simple example would be:

<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE foo [
  <!ELEMENT foo ANY>
  <!ENTITY bar "baz">
]>
<foo>
  &bar;
</foo>

During the parsing, a reference &bar; is replaced with the content from the defined entity, thus yielding <foo>baz</foo>.

If the application were to take external input and include it, without any checks, directly into XML document definition, a wide range of data leaks and attacks would become possible.

The magical thing is that the entity doesn’t have to be a simple string—it can be a reference to a file on the file system. The XML parser will be happy to take the contents of the specified file and include it into the generated response, potentially revealing sensitive system information. As illustrated by OWASP, it would be very easy to obtain information on the system’s users by defining the entity as

<!ENTITY xxe SYSTEM "file:///etc/passwd" >]>

An especially troublesome “feature” of this vulnerability is the possibility to easily execute a denial-of-service attack. One easy way to do it would be to list contents of an endless file like /dev/random. The other one is to create a sequence of entities, each referencing the previous one many times. This turns the final reference into a root of a potentially very wide and deep tree whose parsing could exhaust system memory. This attack is even known as Billion Laughs. As shown on Wikipedia, a series of dummy entities are defined, producing an opportunity for an attacker to include one billion lols in the final document.

<?xml version="1.0"?>
<!DOCTYPE lolz [
 <!ENTITY lol "lol">
 <!ELEMENT lolz (#PCDATA)>
 <!ENTITY lol1 "&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;">
 <!ENTITY lol2 "&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;">
 <!ENTITY lol3 "&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;">
 <!ENTITY lol4 "&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;">
 <!ENTITY lol5 "&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;">
 <!ENTITY lol6 "&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;">
 <!ENTITY lol7 "&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;">
 <!ENTITY lol8 "&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;">
 <!ENTITY lol9 "&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;">
]>
<lolz>&lol9;</lolz>

Preventing XML external entity exploits could be done by using a less complex data format. JSON is a good replacement, provided some precaution is taken as well due to possible attacks against it. Updating XML libraries is a must, coupled with disabling external entity processing and DTD. As always, validate and sanitize the data coming from untrusted sources before using it or including it in your documents.

Insecure Deserialization

When writing code, developers have the power to control the systems they are developing using the code they write. How awesome would it be to control the behavior of third-party systems writing only a little or even no code? Thanks to the fact that people are not perfect and that libraries have flaws, this is definitely possible.

Application state and configuration is often serialized and stored. Sometimes browsers serve as storage engines if the serialized data is tightly coupled to the current user. An application trying to be clever and save processing time could use a cookie to mark that a user has signed in. Since the cookie can only be created after the sign-in has been successful, it makes sense to store the username in the cookie. A user is then authenticated and authorized based on the existence and contents of the cookie. If people weren’t ill-intentioned, nothing could go wrong. To be honest, they mustn’t be curious, either.

If a curious user found a cookie on their machine, they could see something like this:

{"username": "joe.doe", "expires": "2018-06-01 10:28:16"}

A perfectly valid Python dictionary serialized to JSON, nothing special about it. The ever-curious user might change the expiration date to keep the application from forcing the sign-out. An even more curious user might try to modify the username to "jane.doe". If this username existed, it would open a whole new world for the unsuspecting user who now has access to private data.

A simple example of serializing data into JSON and keeping everything transparent is far from the worst thing that can happen to you. If an attacker achieves to modify some serialized data, they might be able to modify it in a way that forces your system to execute arbitrary code.

Let’s say you are building a REST API which allows people to write their own machine learning models in Python and upload them to your service. The service will evaluate the uploaded models and train them using your datasets. This allows people to use your computing resources and a vast amount of available datasets for quick and easy model building.

The service doesn’t store the code in plain text format. Users pickle their code, encrypt it using their private key, and send it to the API for training. When the service needs to run a model, it decrypts the code, unpickles it, and runs it. The tricky part is that the pickle protocol is unsafe. The code can be constructed in a way that allows execution of arbitrary malicious code during deserialization.

Python’s pickle protocol allows classes to define a method __reduce__, which returns information on how to deserialize a custom object. One of the supported return values is a tuple of two arguments: a callable and a tuple of arguments to be passed to the callable. Taking into account that your ML model training system aims to provide maximum flexibility of code structure, it’s possible to write the following code:

class Algo(object):
 def run(self):
   pass
 def __reduce__(self):
   import itertools
   return (list, (itertools.count(1), ))

Once the object needs to be deserialized (unpickled), a function list is called with only one argument. The function list is a list constructor in Python, and the function itertools.count produces an infinite iterator of values, starting with the passed parameter. Turning an infinite iterator into a finite list can have disastrous consequences to your system’s performance and stability.

The only real cure for this type of vulnerability is opting not to deserialize data coming from external sources. In case this is not possible, it is suggested to use a checksum or a digital signature to prevent deserialization of data that was potentially modified by a malicious user. Also, try to set up a sandbox environment decoupled from your main system to limit the effects of issues that might arise.

When using external libraries for deserializing data, for example from XML or JSON, try to pick the ones that allow you to do object type checks before an actual deserialization procedure has been executed. This could catch unexpected object types whose sole purpose is to harm your system.

As with all other actions your application performs, enforce extensive logging and monitoring. Deserializations happening often or failing more than normal are signals that something bad is happening. Catch the issues early.

Insufficient Logging and Monitoring

How much time do you spend to make sure you log all warnings and errors that happen in your application? Do you only store errors happening in the code or do you also log validation errors? What happens when your domain’s business rules aren’t met? Failure to persist all erroneous and suspicious activities in your application presents a security and data compromise.

Imagine the following scenario. Your application contains a sign-in page, as most do. The form has two fields, one for entering an email and the other one for a password. If a user tries to sign in and they provide an incorrect password, they can try again. Unfortunately, the number of incorrect attempts is not limited, so the sign-in page will not be locked after N unsuccessful attempts. An attacker can utilize this opportunity and, given one correct email, keep entering passwords from a rainbow table until one combination finally succeeds. Provided your application is secure enough and you hash the passwords before entering them into a database, this particular attack won’t work. However, do you have a mechanism for identifying intrusions?

Just because this one attempt failed to crack open your sign-in page, it doesn’t mean that some other one won’t. The sign-in page is probably not the only potential backdoor you have, either. If not for something else, someone might try to use broken access control against you. Even perfectly crafted applications should know that someone is trying to attack them, even though it might not be possible. It always it, though.

To do your best to protect yourself against such attacks, do the following steps:

• Log all failures and warnings happening in the application, be it exceptions thrown in the code or access control, validation, and data manipulation errors. All information stored must be replicated and persisted for long enough such that retrospective inspection and analysis is possible.
• Determining the format and persistence layer is important. Having a huge file with arbitrary text format is easy to accomplish; processing it later is not. Choose a storage option which makes it easy to store and read the data and a format which allows for easy and fast (de)serialization. Storing JSON in a database enabling quick access eases the usage. Keep the database small by backing it up regularly.
• If you are dealing with important and valuable data, keep a trail of actions which can be followed to audit the final state. Implement a mechanism for preventing data tampering.
• Have background systems analyze the logs and alert you if something comes up. Checks—which are as simple as testing if a user repeatedly tries to access a protected part of the application—help. Don’t overload the system with dummy checks, though. Monitoring systems must be run as separate services and must not affect the performance of the main system.

When solving the problem, take special care not to leak error logs to external users. Failing to do so makes you susceptible to sensitive information exposure. Logging and monitoring should aid you in solving problems, not attackers in doing their job more efficiently.

Next Steps

Being aware of the potential threats and vulnerabilities in web applications is important. It’s even more important to start identifying them in your applications and apply the patches to remove them.

Attention to application security is an important part of all steps of the software development project. Software architects, developers, and testers must all incorporate software testing procedures into their workflows. It is beneficial to utilize security checklists and automated tests into appropriate steps of the software development process to reduce the security risk.

Whether you are analyzing an existing application or developing a fresh new one, you should look into the OWASP Application Security Verification Standard Project (ASVS). The goal of the project is to develop a standard for application security verification. The standard enumerates tests and requirements for developing secure web applications. The tests are assigned levels from one to three, where one means the least amount of danger and three means the highest potential threat. The classification allows application managers to decide which of the threats are more likely and important. It’s not necessary to include each test within each application.

Both new and existing web application projects, especially those following Agile principles, benefit from structured planning of efforts for securing their applications. The planning of OWASP ASVS tests is easier if you decide to use OWASP Security Knowledge Framework. It’s an application for managing security-testing-oriented sprints, coming with a set of examples on how to solve common security problems, and easy-to-follow checklists based on OWASP ASVS.

If you have just started exploring web application security and need a safe sandbox playground, use a web application implementation from OWASP—WebGoat. It’s an intentionally insecure implementation of a web application. The application guides you through lessons, with each lesson being concentrated on one security threat.

During application development, make sure you:

• Identify and prioritize threats. Define which threats can realistically happen and pose a risk for your application. Prioritize the threats and decide which ones deserve the most development and testing effort. There isn’t much point in putting a lot of effort into solving insufficient logging and monitoring if you are serving a static blog.
• Evaluate the architecture and design of your application. Some vulnerabilities are very difficult to solve during the later phases of application development. For example, if you intend to execute third-party code, and have no plans of using a sandbox environment, it will be very difficult to defend against insecure deserialization and injection attacks.
• Update the software development process. Testing against web application threats must, as much as possible, be an automated process. It is beneficial to augment your CI/CD workflows with automated tests trying to find security holes. You can even utilize your existing unit testing system to develop security tests and run them periodically.
• Learn and improve. The list of issues and vulnerabilities is not static and definitely not limited to ten or fifteen threats. New functionality and ideas open the doors for new types of attacks. It is important to read about the current trends in the web application security world to stay current. Apply what you learn; otherwise, you are wasting your time.

Conclusion

Even though, as the name suggests, the OWASP Top Ten Project lists only ten security vulnerabilities, there are thousands of possible traps and backdoors threatening your applications and, most importantly, your users and their data. Make sure to be on the lookout and constantly refresh your knowledge because technologies changes and improvements have both upsides and downsides.

Oh, and, don’t forget, the world is not black and white. Security vulnerabilities don’t come alone; they are often intertwined. Being exposed to one often means that a bunch of others are around the corners, waiting to rear their ugly heads and sometime, even though it isn’t your fault, as the system security developer, you’re still meant to patch loopholes to curb cybercrime. For an example, see Hacked Credit Card Numbers Are Still, Still Google-able.