General principles of Testing Relief

Testing Relief is designed to give the user the answer to the question whether any changes in the code of the software product affect the areas of the software product directly defined by the user (read it below about "Test Point"). It is important to understand that if the user incorrectly formulates the question, he will not get the entire information about changes in the software product.

Before starting the analysis of the software product, the user must define which areas of the software product are important concerning the quality of the software product (for example, as a rule, they include visual elements of the software product).

Testing Relief is an analytical system

As any analytical system, Testing Relief has condition limits within which the results of the analysis are reliable. These limits are described not by a set of fixed parameters, but by a set of rules meeting which guarantees the reliable analysis result.

First, the set of assemblies of one version (base or last) that should be analyzed must be the result of one and the same compilation. You cannot form sets of assemblies out of the results of several compilations - the probability of an error is too high in this case. Even if all versions, names of types, methods and the number of their parameters coincide from the point of view of Reflection, there is no guarantee that the implementation of some method will be of the correct version. The analysis will reveal no mistakes in this case, but the result of the analysis will be incorrect.

Second, it is very important to manually specify links between methods correctly. Manual linking directly influences building the call graph so the results of the analysis will be incorrect in case links are specified incorrectly. Always make sure that the list of unprocessed methods the program creates after the analysis is empty.

Assemblies

Testing Relief detects modifications in the code of the software product analyzing the MSIL (MicroSoft Intermediate Language) code of assemblies. For Testing Relief to run correctly, all assemblies (including system one) are required to be loaded into the project.

All assemblies loaded into the project are divided into two types:

• Public assemblies
• Hidden assemblies

Public assemblies are displayed in the project tree. You can examine the contents of public assemblies.

Hidden assemblies are not displayed in the project tree and cannot be examined. The depth of the analysis depends on the visibility property of an assembly in the project. Testing Relief does not analyze the code deeper than visible assemblies and hidden assemblies are not analyzed.

System assemblies, third-party assemblies and other assemblies that will not be modified during the development of the project should be specified as hidden.

Analysis process

During analysis, Testing Relief scans the code of methods in the software product for changes as compared to the base versions of these methods. After analyzing changes in the current method, Testing Relief analyzes all called methods. Thus, it builds a call graph. This cycle continues until the program reaches the last method in the chain of calls or until it completes the circle of calls.

Test Point

The point from which the analysis of methods starts is explicitly specified in the project.
The test point is a type of assembly.
Methods that are used as start points in analyzing the code of the software product are separately defined in the test point. The program starts building the call graph from these very methods marked in the test point.

Test results

After building the tree of calls, Testing Relief summarizes information about changes in each method in the test point by processing information about changes in each of the methods in the call graph. Thus, it detects not only areas that were directly modified, but also areas dependent on them.

Data about changes is divided into two groups: critical and non-critical.

Critical changes are highlighted in red in the user interface of Testing Relief. Critical changes include:

• Method Definition Changed - the code of this method has been modified
• Reference Method Definition Changed - the code of the method that is lower in the chain of calls has been modified

Non-critical changes are highlighted in yellow in the user interface of Testing Relief. Non-critical changes include:

• Assembly Added - the assembly the method is defined in has been added
• Assembly Manifest Changed - the parameters of the assembly where the method is defined have been modified
• Assembly Members List Changed - the list of members in the assembly the method is defined in has been modified
• Type Added - the type the method is defined in has been added
• Type Definition Changed - the list of members in the type the method is defined in has been modified
• Method Added - this method has been added
• Reference Assembly Added - the assembly the method that is lower in the chain of calls is defined in has been added
• Reference Assembly Manifest Changed - the parameters of the assembly the method that is lower in the chain of calls is defined in have been modified
• Reference Assembly Members List Changed - the list of members in the assembly the method that is lower in the chain of calls is defined in has been modified
• Reference Type Added - the type the method that is lower in the chain of calls is defined in has been added
• Reference Type Definition Changed - the list of members in the type the method that is lower in the chain of calls is defined in has been modified
• Reference Method Added - the method that is lower in the chain of calls has been added

.NET Framework limitations

Since .NET Framework supports integration with COM and also supports calling methods from other libraries containing unmanaged code, it is generally impossible to track the entire chain of calls.

It is also impossible to determine which of the overloaded virtual methods should be called in the code of the software product without executing the code of the application in case such a virtual method is called.

Besides, Testing Relief analyzes only public assemblies in the project so all callbacks from hidden assemblies also cannot be detected automatically. That is why Testing Relief cannot detect calls of events.

Due to the above limitations, there is a problem of building an incomplete graph of method calls. To solve this problem, Testing Relief has a mechanism of manual method linking.

Workaround for .NET Framework limitations

A workaround for .NET Framework limitations and limitations imposed by the conceptual design of Testing Relief comes down to specifying links between methods explicitly. Information about links between methods can be saved both in the project settings and in the code of the software product as attributes.

During analysis, after Testing Relief processes all MSIL call or callvirt instructions, it analyzes all specified additional links in the method code.

Thus, specifying additional links between methods whenever necessary, you can get a complete and correct graph of calls.

To help you find out if it is necessary to create additional links, after analysis the program creates a list of methods links to which were obtained with the MSIL ldftn instruction, but which were not included in the method tree.

Since, in case a method is virtually called with the MSIL callvirt instruction, one of its implementations is selected depending on the status of memory of the running program, it is impossible to surely determine during the analysis of the software product code which of the implementations of the virtual method will be called in this part of the code.

That is why all implementations of the method are analyzed when a virtual method is called. Thus, all implementations of the virtual method are added to the call tree.