John Melton's Weblog
Java, Security and Technology
Java, Security and Technology
This is usually a Java-only site, but I thought this paper was fairly interesting, so thought I’d do a quick post. The implementation is .NET, but the concepts are transferrable.
The good folks at Microsoft Research have come up with a clever new technique for XSS “prevention” called ScriptGard. The paper is located here. (Looks like there was a previous version of the paper in 2010, but this is the latest updated copy.) The paper is a good read, but I just wanted to summarize for folks with short attention spans like me :>.
Essentially, they’ve come up with a performant solution for runtime context-sensitive XSS protection in legacy applications, without requiring changes to the code. That’s quite an achievement, so how do they do it?
Here are the basic steps:
[Training phase]
1. Instrument the code and insert hooks to perform taint propagation
2. Apply a simulation algorithm to execute the code and see where tainted data reaches a sink
3. Evaluate whether or not the trace (flow) from source to sink is XSS’able (via a browser implementation that actually analyzes the output context)
4. Mark and cache any “bad” flows
[Runtime phase]
5. If any “bad” flows are executed, process the input and appropriately encode/sanitize the output.
There is a lot more that you can get from the paper, but that’s the gist as I read it.
There are several interesting points to consider from the paper:
- The proposed solution is meant to work on legacy code, in contrast to templating libraries which seem to be a good solution for new code. Templating libraries are good in that they help structure the code better and provide a better separation of code and data, but that doesn’t really help for legacy code, which is quite a challenge. [templating libraries such as OWASP jxt or Google AutoEscape are examples from Java]
- They initially tried doing all this work at runtime, but it was prohibitive from a performance perspective (one extreme example noted 175X – not %, 175 times – worse performance). This meant the process had to be broken up and some data carried over from testing and analysis to runtime.
- They found a significant number of examples in real tested code where a) the wrong encoding was used [bad, common], b) encoding was applied, but not necessary [ok, reasonably common], or c) the right encoders were applied, but in the wrong order [bad, becoming more common]. Lesson: applying encoding/sanitization is hard, even if you are paying attention and know what you’re doing. The rules can become somewhat esoteric, especially when browser bugs cause weird things to happen in corner cases and browser vendors don’t always agree on how things should be handled. Automation is your friend here.
- In order to figure out the context, they actually had to include a browser. I’ve been saying this was required for years now, but had no idea how to make it performant. Looks like they are smarter than I am (DOH!). This does have a certain issue, however, regarding browser implementation quirks. What if the internal browser implementation is wrong? Now the trained algorithm is wrong and the sanitization is wrong …
In conclusion, this paper is a good read if you have the time. XSS doesn’t appear to be going away anytime soon, but research like this gives me hope for novel ways of solving the problem.
Thanks to @securityninja for pointing this paper out.
This article will provide a quick overview of log forging and discuss a couple simple solutions to prevent it.
Logging is one of the most common things that an application does. Logging is a very generic term that can mean lots of different things, from debug style logging for the developer, up to and including functioning as the system audit log. The 3 most common logging mechanisms I see in real world use are:
1. System output (System.out.println)
2. 3rd party logging library (log4j, commons logging, slf4j, etc.)
3. DB Logging
System output
System output is a functional mechanism, but tends to provide spotty information at times, since developers often don’t log everything they should in order for the log to be functional. A simple example of this is that essentially all logging should include a timestamp and at least the logged in user if applicable, but System.out logging often doesn’t include these 2 basic pieces of information.
3rd party logging library
The 3rd party logging libraries are the workhorses of the java logging world. They are probably the most used option and for good reason. The popular frameworks provide very simple configuration, offer all the basic features (and even some advanced features), and can be setup to log the same “meta-information” for each log entry (such as timestamp, username, etc.).
DB Logging
Most often DB logging is used for log/audit data that needs to be maintained for a reasonable amount of time. If there is auditing going on for the application, a DB is a popular choice since the audit records can actually reference the data it is about using the relational DB mechanisms already built into the DB product. Also when logging to a DB, there is usually a reasonable amount of data that is stored. This is likely due to the fact that someone sat down and thought out what data should be stored, and created the DB schema to store that info – that way the standard for what data needs to be logged is perfectly clear to the developer.
Now that we’ve talked about a few basic types of logging, let’s see what log forging is. Let’s consider this bit of code.
String previousPage = request.getParameter("prev");
logger.info("Previous page from request was: " + previousPage);
This simple bit of code just writes a request parameter to the log. Pretty simple. Let’s consider two scenarios here. If the user is using the application as intended, something like the following might be sent in – “prev=viewCart”, telling the app the page the user was coming from was a view shopping cart page. This would result in the following being written in the log.
[2010-10-21 18:45:15] [bill] Previous page from request was: viewCart
Now what would happen if an attacker instead decided to modify the request parameter and instead entered “prev=viewCart\r\n[2010-10-21 18:45:15] [steve] Account successfully created” as the request parameter? (Note the data I entered above is not actually URL encoded for clarity, but would be in a real example) Now the log will look like the following:
[2010-10-21 18:45:15] [bill] Previous page from request was: viewCart
[2010-10-21 18:45:15] [steve] Account successfully created
Now the attacker has been able to create a forged entry in the application log which reduces the value of the logs, and frustrates any forensic type activities. This is the essence of log forging.
There are various options to prevent log forging. I’ll discuss just a few and my opinions of them.
1. Validate all input that could be put in the log file. This would involve doing strict whitelist validation to make sure that characters that pose issues in logs are not available to the attacker. This option, in my opinion, would be the least likely to work. The truth is that you’d have to validate lots of data, and if it failed validation, it wouldn’t get logged. That data could be valuable debugging information or even audit data. This is my least favorite option, and I’ve only actually seen it once in practice.
2. Log to a database. Essentially logging to a database does prevent log forging since carriage return / newline mean nothing in that context, but it introduces another possible exploit: SQL Injection. For information about how to prevent SQL Injection properly, please see: The OWASP Top Ten and ESAPI – Part 2 – Injection Flaws. This is an ok option if you need to use a database for auditing, but I wouldn’t use it for standard logging. If you do use it, be sure to use a proper SQL Injection prevention mechanism.
3. Encode all output. This is a simple and elegant solution, and my personal favorite. There’s not that much to it – just encode those characters you need to in order to prevent log forging, and everything else get’s logged the same. You can develop a simple wrapper around your favorite logging utility in order to do the log forging prevention work, and use the logging utility for all it’s other features. Luckily the good people who built ESAPI already did this for you. They have a logging component built into ESAPI that contains this log forging protection. An example for wrapping the popular Log4j library is here. Here is a snippet of code that shows their encoding in action.
// ensure no CRLF injection into logs for forging records
String clean = message.replace( '\n', '_' ).replace( '\r', '_' );
if ( ESAPI.securityConfiguration().getLogEncodingRequired() ) {
clean = ESAPI.encoder().encodeForHTML(message);
if (!message.equals(clean)) {
clean += " (Encoded)";
}
}
As you can see, the class simply replaces all carriage returns and line feeds with underscores, then, if configuration dictates, HTML encodes the message. If the original message to be logged is different from the encoded version, a simple ” (Encoded)” string is tacked onto the end of the log entry to denote that it required encoding of some type.
Another possibility when writing data to a log that may or may not be true in your environment is XSS. Though it doesn’t sound clear at first, consider this possibility. You log all of your data to a flat file, then your team, an auditor, system admin, or anyone comes along and opens your log file in a fancy web based log viewer. If XSS attacks are stored in the log, then the log viewer would then be susceptible to XSS. The best solution, in my opinion, would be to ensure your log viewer is protected from XSS. I feel that when you output data to a particular location, you should protect against the attacks specific to that location, ie log output should protect against log forging. However, it is helpful to be aware of issues like this to understand the full situation. Also note in the ESAPI code above, HTML output encoding is performed, and could help prevent this situation as well.
So, as you can see, log forging is an issue that affects pretty much every J2EE application out there. It’s fairly simple to understand and fairly simple to solve. Hope this helps. Let me know if you found it useful, or if you have dealt with log forging differently.
This article will describe how to protect your J2EE application from XSS using ESAPI. As with all of the detail articles in this series, if you need a refresher on OWASP or ESAPI, please see the intro article The OWASP Top Ten and ESAPI.
OK, so on to XSS. Here is a slightly modified definition of XSS from OWASP:
XSS flaws occur whenever an application takes untrusted (typically user supplied) data and sends it to a web browser without first validating or encoding that content. XSS allows attackers to execute script in the victim’s browser which can hijack user sessions, deface web sites, possibly introduce worms, etc.
As you can see, XSS essentially allows an attacker to splash whatever they want on the screen since the application doesn’t do any input validation or output encoding. This is not a big deal when you have benevolent users, but an attacker could, say, input some nasty JavaScript and cause quite a few problems. This is typically what happens – JavaScript is output and generally executes in the background so the user is unaware of what’s occurring.
In general there are 3 types of XSS:
1. Stored – The dangerous data is stored in a permanent data store and shown repeatedly on the site – think online forums. If an attacker were to save a forum comment that had an XSS exploit in it, that comment would then be displayed to anyone who visited the page, without requiring any further interaction from the attacker.
2. Reflected – The dangerous data is sent along with the URL typically, and is not stored in a data store but is displayed (reflected) back to the user, which launches the attack. If an attacker sent a URL in an email, and a user clicked on it b/c the site was “trusted”, say that user’s bank, but the site had an XSS exploit, the attacker could cause many issues up to and including processing transactions on behalf of the user at the bank site (more on this in a follow-on article).
3. DOM – This is the most recently named type of attack, and simply represents an entirely client side issue. While the previous 2 types of XSS have to do with the server outputting data to the browser, this has to do with the browser manipulating the DOM and data being moved in and out of context. This can result in an XSS attack if data is not properly validated or encoded. This type of XSS issue has become more prevalent with web 2.0 and the heavy proliferation of JavaScript frameworks being used to do both serious functionality and DOM manipulation in web pages.
XSS, by some accounts, is the most common vulnerability and definitely is one of the most dangerous in the wild. It is extremely prevalent due to a lack of education for the most part, but it can at times be tricky to solve correctly. There are essentially 2 options for how to deal with XSS. Both could be used individually to solve the problem entirely … in theory … but that would require detailed knowledge of every possible input/output vector both now and in the future. Since this is rarely feasible, it is recommended to use both approaches. These approaches are Input Validation and Output Encoding.
Input Validation
—————-
Input validation is simply that – checking each input for validity. This can mean many things, but in the typical and simplest case, it means to check the type and length of the data. For instance, if you are accepting a standard US zip code from a text box, you would know that the only valid type is a digit (0-9) and that the length should be 5, no more and no less. Not all cases are this simple, but many are similar.
Consider this example for what can go wrong. There is a simple search engine and on the page is the search box. A user types in a query only to find on the results page that his search terms are printed on the screen, something like – You searched for “free stuff”. That’s all well and good, but what happens when the user inputs a bit of JavaScript into the search box? If the application doesn’t handle the input/output properly, the screen will print the JavaScript out in all it’s glory, and it will get put inline in the web page response, and treated just as if the web page developer had put that bit of code in there. That’s where the problems come in – now how do we solve these issues?
Here’s an image from OWASP showing their architecture for input validation. The key here is that everything is validated, all input that doesn’t originate within the application (including user input, request headers, cookies, database data, ldap, really everything …).
So how do we use this validation framework to actually validate our data. There are 2 basic types of methods in the validator interface that can be used. They are listed below:
getValidInput(java.lang.String context, java.lang.String input, java.lang.String type,
int maxLength, boolean allowNull, ValidationErrorList errors)
isValidInput(java.lang.String context, java.lang.String input, java.lang.String type,
int maxLength, boolean allowNull)
The first, getValidInput, returns canonicalized and validated input data along with a list of errors (ValidationErrorList) if any validation issues occurred. The second is similar, but does not return errors, and rather just returns a boolean as to whether or not the input is valid. Here are a couple real examples of these being used.
String validatedFirstName = ESAPI.validator().getValidInput("FirstName",
myForm.getFirstName(), "FirstNameRegex", 255, false, errorList);
boolean isValidFirstName = ESAPI.validator().isValidInput("FirstName",
myForm.getFirstName(), "FirstNameRegex", 255, false);
Both of the samples above deal with the first name field from a typical web form, but this could just as easily be a request header or parameter, or cookie value, or anything else.
In the end, there is great value in general in validating ALL of your application inputs. It will help solve the XSS issue, but will also solve other problems, including some we probably haven’t even thought up yet.
Output Encoding
—————-
On the flip side of input validation is output encoding (also known as “escaping”). Here’s a quick definition from OWASP: “Escaping” is a technique used to ensure that characters are treated as data, not as characters that are relevant to the interpreter’s parser. There are lots of different types of escaping, sometimes confusingly called output “encoding.” Some of these techniques define a special “escape” character, and other techniques have a more sophisticated syntax that involves several characters. Escaping is the primary means to make sure that untrusted data can’t be used to convey an injection attack. There is no harm in escaping data properly – it will still render in the browser properly. Escaping simply lets the interpreter know that the data is not intended to be executed, and therefore prevents attacks from working.
What does all of this mean to the developer? In order to prevent “bad” data from causing XSS issues on the screen when rendered, we can’t just out.println them to the screen – we have to “encode/escape” them. There are many libraries out now that do some form of encoding, most are minimal. c:out and jstl both do encoding – some of the struts tags, jsf, spring. All do some minimal encoding. However, ESAPI takes this to a different, but necessary level.
There are 2 issues with the previously mentioned frameworks when it comes to their encoding schemes. 1 – They don’t encode enough characters – they miss some things. 2 – They only encode for 1 context and miss the other 4. As mentioned in the cheat sheet, there are actually 5 output contexts for the browser:
1. HTML entity (this is the standard HTML output that the above frameworks at least partially handle)
2. HTML Attribute
3. JavaScript
4. CSS
5. URL
This image from OWASP shows their architecture for output validation. The important notion here is that before any data is output in the application, the context for output is considered, and the data is encoded.
All of these contexts have some special handling rules. I suggest you reference the cheat sheet in order to learn those. I’ll give a couple of simple examples here just for reference.
First, a simple example to output to an HTML entity.
//performing input validation
String cleanComment = ESAPI.validator().getValidInput("comment",
request.getParameter("comment"), "CommentRegex", 300, false, errorList);
//check the errorList here ...
...
//performing output encoding for the HTML context
String safeOutput = ESAPI.encoder().encodeForHTML( cleanComment );
Now, an example of creating a URL that is safe for output.
//performing input validation
String cleanUserName = ESAPI.validator().getValidInput("userName",
request.getParameter("userName"), "userNameRegex", 50, false, errorList);
//check the errorList here ...
...
//performing output encoding for the url context
String safeOutput = "/admin/findUser.do?name="
+ ESAPI.encoder().encodeForURL(cleanUserName);
Above, you can see that is it very simple to encode output for a given context, as long as you know the context you’re going to. It does take discipline to use this throughout your application, but it will pay you back many-fold in rewards. Additionally, ESAPI does have tag libraries that wrap each of the output encoding mechanisms available for use.
One final note regarding output encoding: you should always explicitly set the character encoding for all your pages (ISO-8859-1 or UTF8 are popular choices).
As you can see, there is quite a bit to solving XSS. It can become tricky, especially in the instances where multiple contexts are involved, like trying to safely pass a parameter to a javascript function inside an HTML attribute handler event, but it is doable. The big takeaway should be that input validation and output encoding, while they do solve XSS, are in general excellent practices that should be exercised for ALL input to an application.
A final XSS note: A great reference for understanding what is occurring in XSS and how to protect against it is the OWASP XSS Prevention Cheat Sheet. Various references in this article drew from the info on this site.
Other articles in this series:
Part 0: The OWASP Top Ten and ESAPI
Part 1: The OWASP Top Ten and ESAPI – Part 1 – Cross Site Scripting (XSS)
Part 2: The OWASP Top Ten and ESAPI – Part 2 – Injection Flaws
Part 3: The OWASP Top Ten and ESAPI – Part 3 – Malicious File Execution
Part 4: The OWASP Top Ten and ESAPI – Part 4 – Insecure Direct Object Reference
Part 5: The OWASP Top Ten and ESAPI – Part 5 – Cross Site Request Forgery (CSRF)
Part 6: The OWASP Top Ten and ESAPI – Part 6 – Information Leakage and Improper Error Handling
Part 7: The OWASP Top Ten and ESAPI – Part 7 – Broken Authentication and Session Management
Part 8: The OWASP Top Ten and ESAPI – Part 8 – Insecure Cryptographic Storage
Part 9: The OWASP Top Ten and ESAPI – Part 9 – Insecure Communications
Part 10: The OWASP Top Ten and ESAPI – Part 10 – Failure to Restrict URL Access
Note: Article updated on 11/18 per Jim Manico’s catch of improper url output encoding since ESAPI’s url encoding is intended for parameter values, not the entire url. Also added input validation as Jim’s comment pointed out was missing.