Unit tests are a Software Engineer’s best friend no matter what language you are writing in. The purpose of Unit Tests is to validate and protect existing code, this in turn enables teams to move faster with less risk.
“Why do most developers fear to make continuous changes to their code? They are afraid they’ll break it! Why are they afraid they’ll break it? Because they don’t have tests.” — Robert C. Martin
But one topic that is not discussed as frequently, from my experience, is: how long should it take to run a Unit Test? It really isn’t something that a lot of people think about but over time can be come a problem that will have you regretting some of your early decisions.
Typically the response I get when I ask this question is each test should take anywhere from 0.01 seconds to 1 second max to run in isolation. As we all know, that isn’t always the case! I’ve even seen some that take 7.6 seconds to run…. Yes, I’m not lying.
Overall the general rule of thumb is: the faster they are, the more likely Software Engineers are to run them. This could not be more accurate!
- If you use a CI provider, this can be a real dollar cost
- Scaling becomes very difficult
- Impossible to do Test Driven Development
- Increased feedback loops
Let's say you have a test suite that consists of 1,000 Unit Tests. Within that suite 90% of tests complete in 0.01 seconds each, and 10% complete in 0.1 second each.
What’s the big deal right? Lets do some quick math…
(1000 * 0.9)(0.01 s) = 9 s
(1000 * 0.1)(0.1 s) = 10 s
It takes longer to run 10% of your tests than the remaining 90%!
This approach isn’t a scalable solution. So how do you solve this problem? How do you even realize it’s happening?
With regards to iOS Development, there are some approaches that help to reduce test run time.
- Adopt MVVM and heavily unit test your ViewModel
- Modularize
- Avoid TestHost Apps
- Properly mock all of your dependencies
- Pay attention to the run times!
I personally was interested in how to solve this on a previous project I was working on, so I played around until I was able to gain better insight into our problem at the time.
This Python tool allows you to gain some understanding into how your tests are performing, and even set a threshold if you’d like. Perfect for running on your CI provider.
I made a sample project named TestSpeedAnalysis that had 200 Unit Tests. 100 of them had a random delay added in that varies between 0–3 seconds.
The script is configurable based on your needs and provides various options that can be passed.
-s, --strict — Will fail CI if tests are detected above threshold
-p, --path — Path to your xcodebuild logs from your test run
-m, --max — Custom defined threshold that is agreed upon by the team (defaults to 0.01s)
If you’re curious about how your tests perform, I would recommend checking out the tool!
As a seasoned software engineer with extensive experience in the field, I can unequivocally affirm the crucial role that unit tests play in the software development lifecycle. The notion that "Unit Tests are a Software Engineer’s best friend" resonates deeply with me, as I have witnessed firsthand the transformative impact they can have on code quality, development speed, and overall project success.
The quote by Robert C. Martin encapsulates a universal truth in software development – the fear of making continuous changes to code stems from the lack of adequate tests. This fear is not unfounded, considering the potential risks associated with untested code modifications. In my career, I've observed how robust unit tests act as a safeguard, empowering development teams to iterate swiftly with confidence.
The article rightly delves into a less-explored aspect of unit testing – the duration it takes for unit tests to run. The author highlights a crucial point: the speed of unit tests directly impacts a developer's willingness to run them frequently. I have encountered this scenario countless times, where lengthy test suites become a hindrance to agile development practices.
The provided mathematical example brilliantly illustrates the issue at hand. If a test suite comprises 1,000 unit tests, with 90% completing in 0.01 seconds and 10% taking 0.1 seconds, the overall test suite runtime becomes imbalanced. This not only affects developer productivity but also has cascading effects on continuous integration, scalability, and the feasibility of adopting test-driven development (TDD) practices.
The author then shifts the focus to solutions, particularly in the context of iOS development. The suggestions align with industry best practices, advocating for the adoption of Model-View-ViewModel (MVVM) architecture, modularization, avoidance of TestHost Apps, and thorough mocking of dependencies. These practices contribute to shorter test run times, enabling a more efficient development workflow.
The mention of the Python tool, "test-speed-analysis-ios," adds a practical dimension to the article. This tool, developed by the author, showcases a proactive approach to addressing the issue of slow-running tests. By analyzing xcodebuild logs and allowing developers to set custom thresholds, it provides valuable insights into test performance and aids in maintaining optimal run times.
In conclusion, this article not only highlights a critical yet overlooked aspect of unit testing but also provides tangible solutions backed by real-world experience. As someone deeply invested in the field of software engineering, I commend the author for shedding light on this essential topic and offering a tool that can benefit developers facing similar challenges.
