Archive for the 'Creating tests' Category

“Coverage Closure” is the process used to reach 100% of your coverage goals. In a directed test methodology, it is simply the process of writing all of the testcases outlined in the verification plan. In a constrained-random methodology, it is the process of adding constraints, defining scenarios or writing directed tests to hit the uncovered areas in your functional and structural coverage model. In the latter case, it is a process that is time-consuming and challenging: you must reverse-engineer the design and verification environment to determine why specific stimulus must be generated to hit those uncovered areas.

Note that your first strategy should be to question the relevance of the uncovered coverage point. A coverage point describes an interesting and unique condition that must be verified. If that condition is already represented by another coverage point, or it is not that interesting (if no one on your team is curious about nor looking forward to analyzing a particular coverage point, then it is probably not that interesting), then get rid of the coverage point rather than trying to cover it.

Something that is challenging and time-consuming is an ideal candidate for automation. In this case, the Holy Grail is the automation of the feedback loop between the coverage metrics and the constraint solver. The challenge in automating that loop is correlating those metrics with the constraints.

For input coverage, this correlation is obvious. For every random variable in a transaction description, there is usually a corresponding coverage point, then cross coverage points for various combinations of random variables. VCS includes automatic input coverage convergence. It automatically generates the coverage model based on the constraints in a transaction descriptor, then will automatically tighten the constraints at run-time to reach 100% coverage in very few runs.

For internal or output coverage, the correlation is a lot more obscure. How can a tool determine how to tweak the constraints to reach a specific line in the RTL code, trigger a specific expression or produce a specific set of values in an output transaction? That is where newly acquired technology from Nusym will help. Their secret sauce traces the effect of random input values on expressions inside the design. From there, it is possible to correlate the constraints and the coverage points. Once this correlation is known, it is a relatively straightforward process to modify the constraints to target uncovered coverage points.

A complementary challenge to coverage closure is identifying coverage points that are unreachable. Formal tools, such as Magellan, can help identify structural coverage points that cannot be reached and thus further trim your coverage space. For functional coverage, that same secret sauce from Nusym can also be used to help identify why existing constraints are preventing certain coverage points from being reached.

Keep in mind that filling the coverage model is not the goal of a verification process: it is to find bugs! Ultimately, a coverage model is no different than a set of directed testcases: it will only measure the conditions you have thought of and consider interesting. The value of constraint-random stimulus is not just in filling those coverage models automatically, but also in creating conditions you did not think of. In a constrained-random methodology, the journey is just as interesting—and valuable—as the destination.

Pratish Kumar KT  & Sachin Sohale, Texas Instruments
Tushar Mattu,  Synopsys

In an ideal world, one would want maximum automation and minimal effort. The same holds true when you would want to uncover bugs in your DUV. You would want to provide your team members with an environment whereby they can focus their efforts on debugging issues in the design rather than spend time writing hundreds of lines of testcase code. Here we want to share some simple techniques to make the test writer’s life easier by enabling him to achieve the required objectives with minimum lines of test code, and by providing more automation and reuse. Along with automation, it is important to ensure that the test logic remains easy to understand. In one recent project, where the DUV was an image processing block, the following were some of the requirements:

- The relevant configuration for each block had to be driven first on a configuration port before driving the frame data on another port

- The configuration had to be written to the registers in individual blocks and each block had its own unique configuration.

- Several such blocks together constituted the overall sub-system which required an integrated testbench

- Because all these blocks had to be verified in parallel, the requirement was to have one a generic Register Model which could work not only with the blocks in parallel but also be reused at the sub-system level

- Also, another aspect was to provide as much of reuse in the testcases as possible from block to system level

Given the requirements, we decided to go ahead in the manner described below using a clever mix of the MSS scenarios , named constraints blocks and RAL:

Here is the brief description of the different components and the flow which is represented in the pictorial representation above:

Custom RAL model

As the RAL register model forms the configuration stimulus space, we decided to put all test-specific constraints for register configuration in the top level RAL block itself and had them switched off by default. As seen below, the custom RAL block extended from the generated model has all the test-specific constraints which are switched off.

Basic Multistream Scenario

For each block, one basic scenario (“BASIC SCENARIO”) was always registered with the MSSG. This procedural scenario governs the test flow, which is:

- Randomize the RAL configuration with default constraints,

- Drive the configuration to the configuration port

- Put the desired number of frame data into the driver BFM input channel

The following snippet shows the actual ‘BASIC SCENARIO”

Scenario Library

With the basic infrastructure in place, the strategy for creating a scenario library for the test-specific requirements was simple. Each extended scenario class was meant to change the configuration generation through the RAL model of the different blocks. Thus, for the scenario library for each block, each independent scenario extended the basic scenario and overloaded only the required virtual method for configuration change as shown below:

Test

Each test would then select the scenario corresponding to the test and register it with the MSSG.

At the block level, we were able to reuse this approach across all block level tests consistently and effectively.

Later we were able to reuse all block level tests in the sub-system level testbench , as tests were written in terms of constraints in the RAL model itself. At the sub-system level, we would then turn on specific constraint per block using corresponding block level scenarios. This ensures that there is maximum reuse and that the test flow is consistent across levels.

At higher levels, the scenarios extend the “basic_system_scenario”. The test flow managed by the system level scenarios have a slightly different execution flow than block level tests.But the ‘configuration generation’ is reused consistently and efficiently from block to system. That means modifications of block level test constraints would not require any modification for subsystem level tests using that configuration.

And voila !! Scenarios and test dumping steps were automated at block and sub-system level using simple perl scripting. Test-specific constraint per block were written in an XL sheet which a script will take that as input to generate the scenario and test at block/sub-system level. The scenario execution flow is predetermined and defined in the block/sub-system basic scenario . Furthermore, the flexibility was given to user within this automation process to create multiple such scenarios and reuse existing test constraints.

Dr. Ambar Sarkar, Chief Verification Technologist, Paradigm Works Inc.

Have you been frustrated by how long it takes to run the initial phases of a test before it gets to those that really matter? And it gets worse when you need to run a number of these tests. I often see projects where about 80% of the test execution time is used up in these initial phases.

While you do need to execute these initial phases, repeating them for each test does not contribute to any additional verification. Ideally, you would like to run the initial phases for a group of related tests just once, just to a point where you are ready to send real traffic or transactions. The phases that follow should be executed once per test, with the environment rolled back to an appropriate state before the next test starts executing.

The diagram below shows how you may want chain two testsTEST1 and TEST2, where you want to use the same configuration for both.

VMM1.2 supports this model of test execution for implicitly phased environments.

First the steps.

Step 1. Tag a test as concatenable and specify the rewind-to phase.?

If you are using implicit phasing of your environment, this is done simply by using the macro that announces the tests as concatenable, along with the phase to roll_back to.

class TEST1 extends vmm_test;
//Macro to indicate which phase to rollback to
`vmm_test_concatenate(configure_test)
…

endclass

class TEST2 extends vmm_test;
//Macro to indicate which phase to rollback to
`vmm_test_concatenate(configure_test)
…

endclass

Step 2. Invoke it.

This is done by simply specifying the command line parameters:

//Command line arguments to run the example
./simv +vmm_test=TEST1+TEST2

You also have the option of specifying all tests in a file or just chain all the tests.

So what are the gotchas?

The underlying VMM implementation (and you can read the gory details of how this is done using the three timelines: pre, top, and post in the VMM user guide) takes care of synchronizing all the test phases so that the tests can be effectively chained. However, the user still has to implement any application specific cleanup in between tests to restore the environment. This seems fair enough, and the cleanup phase is the perfect phase to implement that.

However, there is one limitation when considering chained tests. You cannot use vmm_test::set_config() in concatenated tests, which is the place to perform class-factory overrides for the environment components. This is not a limitation of the implementation however, but just how class factory overrides work. Note that you can still use class factory to override your traffic scenarios, just not the environment components.

First, a test can only run after the environment is created. Second, a class factory override for the environment can work only if it is called before the environment is created. This means that after the first test is executed, you cannot override the environment setting using class factory override, unless you are willing to rebuild the environment from scratch in between. Clearly, this rebuild per test run will negate any benefits of chaining. Since vmm_test::set_config is explicitly used to reconfigure environments using the class factory method, it is not allowed when tests are chained.

This article is the 5^th in the Verification in the trenches series. Hope you found this article useful. If you would like to hear about any other related topic, please comment or drop me a line at ambar.sarkar@paradigm-works.com. Also, if you are starting out fresh, please check out the free VMM1.2 environment generator at http://resourceworks.paradigm-works.com/svftg/vmm .

Fabian Delguste, Snr CAE Manager, Synopsys

VMM 1.1 comes with a new base class called vmm_test that can be used for all tests. The main motivation for developing this base class was to enable a single compile-elaboration-simulate step for all tests rather than multiple ones. It is recommended to implement tests in a program thread as this provides a good way of encapsulating a testbench, and reduces races between design and testbench code. All test examples showed the tests implemented directly in program blocks. The drawback of this technique is that the user needs to recompile, elaborate and simulate each test individually. When dealing with large regressions consisting of 1000s of tests, multiple elaborations can waste a significant amount of time whereas tests using vmm_test only requires one elaboration. A given test can be selected at run-time using a command-line option. Switches like +ntb_random_seed can be used in conjunction with these tests.

To understand better how this base class works, let’s look at the example that ships with the VMM 1.1 release in the sv/examples/std_lib/vmm_test directory. This example shows how to constrain some ALU transactions. These transactions, modelled by the alu_data class , are randomly generated by an instance of a vmm_atomic_gen and passed to an ALU driver using a vmm_channel.

Before digging in the gory details of vmm_test, let’s see how tests are traditionally written:

1. class add_test_data extends alu_data;

2.   constraint cst_test {

3.      kind == ADD;

4.    }

5. endclass

6.

7. program alu_test();

8.    alu_env env;

9.    initial begin

10.     add_test_data tdata = new;

11.     env.build();

12.     env.gen.randomized_obj = tdata;

13.     env.run();

14.   end

15.endprogram

· In lines 1-4, the alu_data class is extended to the new class add_test_data which contains test-specific constraints. In this test, only ADD operations are carried modelled by this class.

· In line 7, a program block is used to instantiate the environment alu_env based on vmm_env.

· In line 10-13, the environment is built and the new transaction add_test_data is used as the factory instance for our vmm_atomic_gen transactor.

Of course, this test is very specific and a user clearly needs to create a similar program block with other constraints to fulfill the corresponding test plan. For instance, this test can be duplicated many times to send {MUL, SUB, DIV} ALU operations to the ALU driver. In this case, multiple program blocks are required, so are multiple elaborations and binaries to simulate these tests.

VMM 1.1 provides a way to include all test files in a single compilation. The previous test can now be written like this:

1. class add_test_data extends alu_data;

2.   constraint cst_test {

3.     kind == ADD;

4.   }

5. endclass

6.

7.  `vmm_test_begin(test_add, alu_env, “Addition”)

8.  env.build();

9.   begin

10.   add_test_data tdata = new;

11.   env.gen.randomized_obj = tdata;

12. end

13. env.run();

14. `vmm_test_end(test_add)

· In line 7, the vmm_test short hand macro `vmm_test_begin is used to declare the test name (test_add in our example), the name of the vmm_env where all transactors reside (alu_env in our example) and a label that is used to tag this particular test.

· In lines 8-13, users can build the environment, insert the factory instance and kick off the test.

· In line 14, the vmm_test short hand macro `vmm_test_end is used to terminate this test declaration.

Of course, other tests with variations in constraints can be written in the same way. Since the environment is exposed after the `vmm_test_begin short hand macro, it is possible to register callbacks, replace generators or do any other operation which is traditionally done in the VMM program block.

An important aspect of these tests is that whenever they are included, they become statically declared and visible to the environment.

Now let’s see how to include these tests in VMM program block:

1. `include “test_add.sv”

2. `include “test_sub.sv”

3. `include “test_mul.sv”

4. `include “test_ls.sv”

5. `include “test_rs.sv”

6.

7. program alu_test();

8.    alu_env env;

9.    initial begin

10.    vmm_test_registry registry = new;

11.    env = new(alu_drv_port, alu_mon_port);

12.    registry.run(env);

13.  end

14. endprogram

· In line 1-5, all tests are simply included

· In line 10, registry, which is an instance of vmm_test_registry,is constructed. This object contains all tests implemented using vmm_test_begin that have been previously included

· In line 12, registry can be run and a handle to the environment is passed as an argument to this class. This is how all vmm_tests can have access to the environment

Running a specific test is achieved by providing the test name on the command line, for example:

simv +vmm_test=test_add

simv +vmm_test=test_sub

Note that calling simv without the +vmm_test switch returns a FATAL error and lists all registered tests. This is a good way to document the available tests.