What is it and why should I care?
Unit testing is the term generally associated with the process of writing code specifically purposed for testing your application functionality. You write test code to run your functional application code and verify the results.
Note: Unit testing is actually a specific subset of this idea focused on the individual unit (generally recognized as a single class and/or method). Other testing techniques (integration, end-to-end, functional) are also extremely valuable, but unit testing is the most well-known and most of the concepts are transferrable.
So, what does unit testing provide to me and why should I want to use it? Let me try to address that in two parts.
Since unit testing is fundamentally an activity that goes on during the development of software, let me first consider what developers use it for. Considering the proper use of unit testing (I’ll address improper use below), it provides you primarily with confidence and quality (which are arguably the same thing in my constrained definition here). I’ve had discussions with people who say it offers a lot more, and certainly you can argue that point, but for me all of the myriad reasons come back to confidence and quality. You’ve exercised the code enough (and you continually do so) that you feel confident that your code works the way you expected it to work. There’s a lot that goes into this ideal world where you have good tests, but it can be practically done, and it makes an immense difference in the quality of code that you produce.
So, unit testing inside the development process produces confidence and quality. What do we get from it when applied to security? … The same thing. Security is no different from other code from the perspective of correctness, though it is arguably less well understood in many organizations. At the end of the day, though, we should treat security requirements with the same rigor we would treat performance, functionality, or any other requirement. We should specify what we expect, build to that, test to that, and produce that. Unit testing is a fantastic tool that can be applied to produce confidence and quality in the implementation of security in an application.
What should I do about it?
“You should stop everything you’re doing on the app you’re currently working on and go back and make sure you have at least 80% code coverage with your tests.” I put that in quotes because, sadly, I had a previous job (long ago) where I was informed that this was exactly what I needed to do. I thought they were joking at first, but I quickly found out that this was their real requirement. No matter that the number was seemingly arbitrary and that the tests they gave me as “good examples of what we’re looking for” didn’t actually examine the output of the test, but rather just ran the code (READ: not a real test). No, testing was all about hitting a number, and saying you did it.
If that’s the way you look at testing, you get no value from it, and it actually costs you quite a bit in development, configuration and execution.
Ok, so if not that, then what?
Here are a couple ideas I’d recommend when first starting out on the path to good unit testing.
2. Read somebody else’s (good) test code. I always recommend this when folks are getting started. I usually recommend something like the Spring framework, which has some of the highest quality code around, to use as a starting point to get good ideas for how to test. Look at the class they are testing and the tests they wrote for it. This will get you a good idea of what’s going on.
Once you’ve gotten through the initial learning phase regarding real unit testing, here are a few more ideas that I’ve seen work pretty well. Certainly add to or remove from this list to make it work in your environment, but at least consider the concepts. (Also, here is someone else’s list if you’d like some additional ideas)
1. Build a regression suite
A huge advantage of writing another full set of code to test your functional code is that the code stays around (also can have a bad result at times for maintenance, but let’s focus on the positive here). That means you have an incredible regression test suite. This is one of the most powerful things about unit testing. You’ve written some function, and a few tests to cover it. You modify the function in order to add some new functionality, and now one of your tests breaks. This happens constantly, but is a strong safety net. This is one of the single best things I’ve ever had to learn the hard way that I needed.
2. Write good tests
Writing a test that exercises a method is trivial – call the method with necessary parameters … done. This is the cheap (and useless) way to do testing. What you should do (and you’ll learn this in the book) is write positive and negative tests for the method. Think about nulls. Think about boundary conditions for your parameters. What happens if I send a negative number in or what if I send Integer.MAX_VALUE? Inspect the return values from functions and make sure they align with your expectations.
This type of “how can I break this” thinking usually comes natural to good security folks, so it’s actually a good fit for them. However, good developers certainly get this mindset as they test more, and it’s extremely beneficial for both functional and security tests. This is a cool and powerful way to get developers who aren’t security minded going with security.
3. Start small
This piece of advice is given for almost everything, but that’s because it works. Certainly begin practising on some insignificant code or even just make a sample app to get going. However, when getting good tests into your code-base, I generally recommend picking either/or a) your most important classes or b) your most buggy classes. You’ll likely be surprised at the number of bugs you find when you start the process, but just remember those are bugs your customer won’t find!
4. Capture metrics (and use them)
There are lots of tools that help you capture metrics over source code or bug repositories. It’s really a fun exercise to see the number of reported bugs go down as your number of unit tests goes up. You can also use data like the number of bugs in code in a certain part of the application, or written by a certain developer to identify where to focus your testing efforts.
5. Use tools to help you out
You could certainly write your own unit testing framework, but why when others have already done it. In the Java world, the reigning king is JUnit. Martin Fowler is quoted as saying “never in the field of software development have so many owed so much to so few lines of code” when referencing JUnit. The idea behind JUnit is simple, but it’s notable because the execution is great, and there are lots of additional tools, like IDE support and build tool support, that make using it so simple.
6. Do code coverage
A bit earlier, I knocked on code coverage because I feel it’s a pretty weak metric in and of itself. However, it is a valuable tool in combination with a quality process. If you know your tests are good, and you know that you cover 85% of your significant code, then you’re doing pretty well. Again assuming that you’re tests are good, this can be another metric in the process that points to improvement.
As far as tools go, there are several, but I generally recommend EclEmma for this purpose.
7. Code review your tests with the code they’re testing
Yes, you should code review your tests. This is part of the “writing good tests” process. You need to ensure that people are testing their code, and that they are doing it correctly. Just fold this into your code review process, and you’re golden.
8. Test different requirements
Don’t just test functionality. Test for performance. Test for security. Test for other types of requirements. All these things can be added, and they’ll improve your confidence and quality.
9. Add a test when you get a bug
When you find a bug, do this:
- add a test that covers the bug
- make sure all the existing tests pass, but the new test fails
- fix the code
- make sure all tests pass
This process allows you to make sure you don’t regress over time and that the same bug doesn’t come back to haunt you.
10. Write your tests first … at least on paper
A lot of people swear by Test Driven Development, the idea that you write all your tests for a method/class _before_ you write the class. You ensure all of them fail. You then write the class, and by the time all the tests pass (assuming you’ve written tests to cover every scenario and that the tests are accurate and “good), then you’re done with coding.
I’m personally not a stickler for this process, though there are advantages. If you’re doing a brand-new project, this can be great, but a lot of the work developers do is on legacy code (even if only a few weeks/months old), and a lot of that doesn’t have tests, so we have to have a process that allows for that.
What I will say is that you should write your tests separately from the code. All tests come from the requirements, not the code. This is an important point, and is where most new test writers fail – they write the tests to work with the code that’s there, not to test the actual code. Of course if you’re writing tests meant to work for the code you see, those tests should pass. It’s divorcing yourself from the implementation when you’re writing tests that’s important. This is something you shouldn’t sacrifice on.
11. Don’t let testing replace modelling altogether
In this superb talk, Glenn Vanderburg points out that software engineering relies quite heavily on modelling, when we can achieve many of the same desired results through testing. He talks about aerospace engineers using modelling, followed by a prototype, but says that of course they would use the prototype every time if it were as cheap as the modelling, since it’s actually testing the “real thing”. I think this point does ring true to an extent. I don’t personally like using modelling extensively because in practice it’s a) overkill and b) outdated as quickly as you can create it. However, I do think higher-level modelling adds significant value and that testing is never going to be able to effectively replace it because the value is in the mental exercise of considering the system at a higher level, which testing often doesn’t do properly, or at least effectively.
12. Continuous testing
Continuous integration platforms are common nowadays, but there are still lots of people not using them (but you SHOULD!). However, they really are extremely helpful for testing. They all include the idea that the unit tests should be run on every build. This is powerful because it forces you to find the broken tests quickly and fix them while what you changed is still fresh on your mind. If you don’t have a CI environment, your build system and configuration should at least allow you to do this every time you build, and then you should have your process include that step.
13. Get unit testing into your SDLC.
It sounds sad to say, but you have to force testing as a requirement or there will be those that won’t do it. Testing is a definite and important step in the SDLC and should be represented as such. I will note here that I fully believe your good developers will really enjoy the effect of the unit tests once they use them (even if they still don’t like writing them). Every good developer I’ve worked with has thought good tests were worth their weight in gold.
14. Add in tests for integration, functional, end-to-end, etc.
Unit testing is really a subset of the larger testing scope. There are tests at the component, business function, application, etc. levels and all of these should be tested. All of the same rules above apply to these as well. It’s really great to be able to hit a button and know that your application is being run through the gauntlet of unit, integration, functional, and end-to-end tests. It’s a pretty powerful concept.
Unit testing is one of those ideas that’s not really specifically for security, but actually does quite a bit for security if applied correctly. It’s also a great way to get developers and security people working together for a common goal.
It’s also near and dear to my heart as I think it’s been the single most important idea that’s helped me improve as a developer over the years. I hope it’s as useful for you as it has been for me.