A software regression is a type of software bug where a feature that has worked before stops working. This may happen after changes are applied to the software's source code, including the addition of new features and bug fixes.[1] They may also be introduced by changes to the environment in which the software is running, such as system upgrades, system patching or a change to daylight saving time.[2] A software performance regression is a situation where the software still functions correctly, but performs more slowly or uses more memory or resources than before.[3] Various types of software regressions have been identified in practice, including the following:[4]
- Local – a change introduces a new bug in the changed module or component.
- Remote – a change in one part of the software breaks functionality in another module or component.
- Unmasked – a change unmasks an already existing bug that had no effect before the change.
Regressions are often caused by encompassed bug fixes included in software patches. One approach to avoiding this kind of problem is regression testing. A properly designed test plan aims at preventing this possibility before releasing any software.[5] Automated testing and well-written test cases can reduce the likelihood of a regression.
Prevention and detection
Techniques have been proposed that try to prevent regressions from being introduced into software at various stages of development, as outlined below.
Prior to release
In order to avoid regressions being seen by the end-user after release, developers regularly run regression tests after changes are introduced to the software. These tests can include unit tests to catch local regressions as well as integration tests to catch remote regressions.[6] Regression testing techniques often leverage existing test cases to minimize the effort involved in creating them.[7] However, due to the volume of these existing tests, it is often necessary to select a representative subset, using techniques such as test-case prioritization.
For detecting performance regressions, software performance tests are run on a regular basis, to monitor the response time and resource usage metrics of the software after subsequent changes.[8] Unlike functional regression tests, the results of performance tests are subject to variance - that is, results can differ between tests due to variance in performance measurements; as a result, a decision must be made on whether a change in performance numbers constitutes a regression, based on experience and end-user demands. Approaches such as statistical significance testing and change point detection are sometimes used to aid in this decision.[9]
Prior to commit
Since debugging and localizing the root cause of a software regression can be expensive,[10][11] there also exist some methods that try to prevent regressions from being committed into the code repository in the first place. For example, Git Hooks enable developers to run test scripts before code changes are committed or pushed to the code repository.[12] In addition, change impact analysis has been applied to software to predict the impact of a code change on various components of the program, and to supplement test case selection and prioritization.[13][14] Software linters are also often added to commit hooks to ensure consistent coding style, thereby minimizing stylistic issues that can make the software prone to regressions.[15]
Localization
Many of the techniques used to find the root cause of non-regression software bugs can also be used to debug software regressions, including breakpoint debugging, print debugging, and program slicing. The techniques described below are often used specifically to debug software regressions.
Functional regressions
A common technique used to localize functional regressions is bisection, which takes both a buggy commit and a previously working commit as input, and tries to find the root cause by doing a binary search on the commits in between.[16] Version control systems such as Git and Mercurial provide built-in ways to perform bisection on a given pair of commits.[17][18]
Other options include directly associating the result of a regression test with code changes;[19] setting divergence breakpoints;[20] or using incremental data-flow analysis, which identifies test cases - including failing ones - that are relevant to a set of code changes,[21] among others.
Performance regressions
Profiling measures the performance and resource usage of various components of a program, and is used to generate data useful in debugging performance issues. In the context of software performance regressions, developers often compare the call trees (also known as "timelines") generated by profilers for both the buggy version and the previously working version, and mechanisms exist to simplify this comparison.[22] Web development tools typically provide developers the ability to record these performance profiles.[23][24]
Logging also helps with performance regression localization, and similar to call trees, developers can compare systematically-placed performance logs of multiple versions of the same software.[25] A tradeoff exists when adding these performance logs, as adding many logs can help developers pinpoint which portions of the software are regressing at smaller granularities, while adding only a few logs will also reduce overhead when executing the program.[26]
Additional approaches include writing performance-aware unit tests to help with localization,[27] and ranking subsystems based on performance counter deviations.[28] Bisection can also be repurposed for performance regressions by considering commits that perform below (or above) a certain baseline value as buggy, and taking either the left or the right side of the commits based on the results of this comparison.