Using vmm_test Base Class
Posted by Fabian Delguste on April 24th, 2009
Fabian Delguste, Snr CAE Manager, Synopsys
VMM 1.1 comes with a new base class called vmm_test that can be used for all tests. The main motivation for developing this base class was to enable a single compile-elaboration-simulate step for all tests rather than multiple ones. It is recommended to implement tests in a program thread as this provides a good way of encapsulating a testbench, and reduces races between design and testbench code. All test examples showed the tests implemented directly in program blocks. The drawback of this technique is that the user needs to recompile, elaborate and simulate each test individually. When dealing with large regressions consisting of 1000s of tests, multiple elaborations can waste a significant amount of time whereas tests using vmm_test only requires one elaboration. A given test can be selected at run-time using a command-line option. Switches like +ntb_random_seed can be used in conjunction with these tests.
To understand better how this base class works, let’s look at the example that ships with the VMM 1.1 release in the sv/examples/std_lib/vmm_test directory. This example shows how to constrain some ALU transactions. These transactions, modelled by the alu_data class , are randomly generated by an instance of a vmm_atomic_gen and passed to an ALU driver using a vmm_channel.
Before digging in the gory details of vmm_test, let’s see how tests are traditionally written:
1. class add_test_data extends alu_data;
2. constraint cst_test {
3. kind == ADD;
4. }
5. endclass
6.
7. program alu_test();
8. alu_env env;
9. initial begin
10. add_test_data tdata = new;
11. env.build();
12. env.gen.randomized_obj = tdata;
13. env.run();
14. end
15.endprogram
· In lines 1-4, the alu_data class is extended to the new class add_test_data which contains test-specific constraints. In this test, only ADD operations are carried modelled by this class.
· In line 7, a program block is used to instantiate the environment alu_env based on vmm_env.
· In line 10-13, the environment is built and the new transaction add_test_data is used as the factory instance for our vmm_atomic_gen transactor.
Of course, this test is very specific and a user clearly needs to create a similar program block with other constraints to fulfill the corresponding test plan. For instance, this test can be duplicated many times to send {MUL, SUB, DIV} ALU operations to the ALU driver. In this case, multiple program blocks are required, so are multiple elaborations and binaries to simulate these tests.
VMM 1.1 provides a way to include all test files in a single compilation. The previous test can now be written like this:
1. class add_test_data extends alu_data;
2. constraint cst_test {
3. kind == ADD;
4. }
5. endclass
6.
7. `vmm_test_begin(test_add, alu_env, “Addition”)
8. env.build();
9. begin
10. add_test_data tdata = new;
11. env.gen.randomized_obj = tdata;
12. end
13. env.run();
14. `vmm_test_end(test_add)
· In line 7, the vmm_test short hand macro `vmm_test_begin is used to declare the test name (test_add in our example), the name of the vmm_env where all transactors reside (alu_env in our example) and a label that is used to tag this particular test.
· In lines 8-13, users can build the environment, insert the factory instance and kick off the test.
· In line 14, the vmm_test short hand macro `vmm_test_end is used to terminate this test declaration.
Of course, other tests with variations in constraints can be written in the same way. Since the environment is exposed after the `vmm_test_begin short hand macro, it is possible to register callbacks, replace generators or do any other operation which is traditionally done in the VMM program block.
An important aspect of these tests is that whenever they are included, they become statically declared and visible to the environment.
Now let’s see how to include these tests in VMM program block:
1. `include “test_add.sv”
2. `include “test_sub.sv”
3. `include “test_mul.sv”
4. `include “test_ls.sv”
5. `include “test_rs.sv”
6.
7. program alu_test();
8. alu_env env;
9. initial begin
10. vmm_test_registry registry = new;
11. env = new(alu_drv_port, alu_mon_port);
12. registry.run(env);
13. end
14. endprogram
· In line 1-5, all tests are simply included
· In line 10, registry, which is an instance of vmm_test_registry,is constructed. This object contains all tests implemented using vmm_test_begin that have been previously included
· In line 12, registry can be run and a handle to the environment is passed as an argument to this class. This is how all vmm_tests can have access to the environment
Running a specific test is achieved by providing the test name on the command line, for example:
simv +vmm_test=test_add
simv +vmm_test=test_sub
Note that calling simv without the +vmm_test switch returns a FATAL error and lists all registered tests. This is a good way to document the available tests.
April 27th, 2009 at 4:32 am
Nice Explanation.
May 19th, 2009 at 12:13 am
Hi Fabian,
The vmm_test feature added in VMM1.1 is really helpful and saves lot of compilation time. Thanks for the good explanation !!
I have used vmm_test feature in my verification environment and it really helps while running the regression. But albeit I want to run signal testcase, will it still compile all the testcase which I have included in program file? I can always make a script to modify the include file as per the needs, but is there any other work around which I can use, when I just want to run a single testcase?
Regards,
Puja
May 19th, 2009 at 4:00 am
Hi Puja,
Well, as long as you modify a SV test you need to recompile and elaborate your testbench.
Note that include files are not dynamic.
Thanks
Fabian
May 25th, 2009 at 3:44 am
Thanks Fabian for the prompt reply.
I agree that whenever SV test is modified, we need to recompile the same again. But my question is that, say for example I have included 5 tests in the program file as you have shown in the example and I just want to run single testcase say “test_add” and not regression, will it still compile all the 5 included tests, which is not required when I am running single testcase and not regression?
Thanks,
Puja
May 25th, 2009 at 8:10 am
Hi Puja,
You’re right, all included tests are compiled each time. That’s the way VMM 1.1 works, all tests are included and statically appended to a list of possible tests. Then at run-time user can pick the ones he wants to run.
Overhead for putting all tests together is very small and we haven’t experiences any significant slowdown so far.
For sure, you need to draw the line on how many tests should really be compiled and available for a given regression. For instance, say that you have 1000 tests to regress, this may yield to huge binary in the end if all them are appended. In this case, it would be better to group these tests by categories and have separate binaries, like 10 buckets containing 100 tests each.
Thanks,
Fabian
June 3rd, 2009 at 2:24 am
Hi Fabian,
VMM short hand macros and test feature helps a lot. (Reduce manual coding error and time too)
However, in VMM 1.1 Test feature, everytime I need to make sure to give naming convention of extended class is different as it compiles all testcase in one shot.
Let say I have 1000 testcases. To overwrite default constraint of configuration class,
I cant write class alu_cfg_ow extends alu_cfg in all 1000 testcases as it causes compile error.(when testcase created by multiple users)
Well, benefits of VMM 1.1 features are more than these minor compile issues.
Thanks
Falguni
August 5th, 2009 at 11:22 am
Hi Fabian,
Very good explaination! I would surely use this in my upcoming project. But I was wondering, will this test strategy still let us do the regression grading, because during the regression phase we often look at the functional coverage and code coverage number contributed by each testcase to qualify their value.
-Nitin
February 17th, 2010 at 12:39 pm
Hi Fabian,
The article is informative and useful.
The VMM_TEST solution looks much better than the other alternative (separate compile, that is).
Are there any limitations to what sorts of things you can put inside a vmm_test? for example – external constraints or functions? Seems like that would be a problem.
Yaron
October 22nd, 2010 at 3:34 am
Hi Fabian,
It is really a very good instructive article .
I have a question for regression use of vmm_test .
Here come one problem that in our test same class name is used in different testcase .
This is not valid in VCS compile process .
Is there any ways to solve it except for rename the class one by one manually?
Thanks
Zephyr
October 25th, 2010 at 4:34 am
Hi Zephyr,
Actually all new tests that extends from vmm_test should have a unique name.
I mean to say:
// test1
class test1 extends vmm_test;
…
endclass;
// test2
class test2 extends vmm_test;
…
endclass;
The main reason is that vmm_test is able to compile all tests and only pick one or several ones at run time.
For that their name should be unique.
You can keep same names for tests but should instantiate them in different scope, which can be achieved with different program blocks.
Thanks
Fabian
July 20th, 2011 at 4:43 am
Hi Fabian
Thanks for quick response!!
I meet another problem when using vmm_test
In your example
8. alu_env env;
9. initial begin
10. add_test_data tdata = new;
11. env.build();
12. env.gen.randomized_obj = tdata;
13. env.run();
In lines 8-13, users can build the environment, insert the factory instance and kick off the test.
In my test case I would like to use task to update ral or feedback some variables to the generator
I instance and declare the task in the area as line 8-13 in your example .
It will cause compile Error !!
Is there any trick to solve this ?
Thanks