Archive for August, 2009

Andrew Piziali, Independent Consultant

You are using coverage, along with other metrics, to measure verification progress as part of your verification methodology.^{1 2} Yet, lurking in the flow are the seeds of a bug escape that will blindside you. How so?

Imagine you are responsible for verifying an in-order, three-way x86 superscalar processor in the last millennium, before the Constrained Random Generation. Since your management wouldn’t spring for an instruction set architecture (ISA) test generator, you hired a team of new college grads to write thousands of assembly language tests. Within the allocated development time, the tests were written, they were functionally graded and achieved 100% coverage, and they all finally passed. Yeah! But, not so fast …

When first silicon was returned and Windows was booted on the processor, it crashed. The diagnosis revealed a variant of one of the branch instructions was misbehaving. (This sounds better than “It had a bug escape.”) How could this be? We reviewed our branch instruction coverage models and confirmed they were complete. Since all of the branch tests passed, how could this bug slip through?

Further analysis revealed this branch instruction was absent from the set of branch tests yet used in one of the floating point tests. Since the floating point test was aimed at verifying the floating point operation of the processor, we were not surprised to find it was insensitive to a failure of this branch instruction. In other words, as long as the floating point operations verified by the test behaved properly, the test passed, independent of the behavior of the branch instruction. From a coverage aspect, the complete ISA test suite was functionally graded rather than each sub-suite graded according to its functional requirements. Hence, we recorded full coverage.

The problem was now clear: the checking and coverage aspects of each test were not coupled, conditioning coverage recording on passing checked behavior. If we had either (1) functionally graded each test suite only for the functionality it was verifying or (2) conditionally recorded each coverage point based upon a corresponding check passing, this bug would not have slipped through. Using either approach, we would have discovered this particular branch variant was absent from the branch test suite. In the first case, that coverage point would have remained empty for the branch test suite. Likewise, in the second case we would not have recorded the coverage point because no branch instruction check would have been activated and passed.

Returning to the 21^st century, the lesson we can take away from this experience is that coverage—functional, code and assertion—is suspect unless, during analysis, you confirm that for each coverage point a corresponding checker was active and passed. From the perspective of implementing your constrained random verification environment, each checker should emit an event (or some other notification), synchronous with the coverage recording operation, indicating it was active and the functional behavior was correct. The coverage code should condition recording each coverage point on that event. If you are using a tool like VMM Planner to analyze coverage, you may use its “-feature” switch to restrict the annotation of feature-specific parts of your verification plan to the coverage database(s) of that feature’s test suite.

You might ask if functional qualification³ would address this problem. Functional qualification answers the question “Will my verification environment detect, propagate and report each functional bug?” As such, it provides insight into how well your environment detects bugs but says nothing about the quality of the coverage aspect of the environment. I will address this topic in a future post if there is sufficient interest.

Remember, make your coverage count by coupling checking with coverage!

1Metric Driven Verification, 2008, Hamilton Carter and Shankar Hemmady, Springer

2Functional Verification Coverage Measurement and Analysis, 2008, Andrew Piziali, Springer

3“Functional Qualification,” “EDA Design Line,” June 2007, Mark Hampton

Srinivasan Venkataramanan, CVC

Pawan Bellamkonda, Brocade

During our recent VMM training at CVC, we learned about VMM data member macros, and our engineers liked it. Some of our teams at Brocade have started adopting it in their projects right away! We see that we can avoid much of the lengthy code and increase readability with these new macros. We will surely avoid making silly mistakes which might be hard to debug later.

However as with any built-in automation, there are always scenarios where-in user level customization of some or all of the methods is required. VMM provides this flexibility for overriding the default behavior of virtual methods of vmm_data class. In one of our blocks we needed to tweak the constructor of the transaction. One question that perplexed us was:

“I have built a transaction class extending from vmm_data. We have used the short hand macro `vmm_data_member…..  to get all the functions automatically. But while creating an object of this transaction, we want to pass a configuration class object as argument in the new function. How should we override the new() function alone when we use the short hand macros? When we tried using do_new() (like overriding other functions), it did not work.”

As we explored a bit, we found another macro specifically meant for this:

`vmm_data_new(<class_name>)

This macro should be used before the beginning of data-member macros. This lets the succeeding macros do all the work except the “new” function implementation.

class s2p_xactn extend vmm_data;
rand bit [7:0] pkt_len, pkt_pld;

`vmm_data_new(s2p_xactn)
function new(int my_own_arg = 2);
`vmm_note (log, $psprintf (“my val is %0d”, my_own_arg));
endfunction : new

`vmm_data_member_begin(s2p_xactn)
`vmm_data_member_scalar(pkt_len, DO_ALL)
`vmm_data_member_scalar(pkt_pld, DO_ALL)
`vmm_data_member_end(s2p_xactn)
endclass : s2p_xactn

As with traditional martial arts, functional verification too has some slightly different styles/requirements that makes each project interesting and unique. To its credit, we feel that VMM is as proven as traditional martial arts: it can be tailored to different requirements while providing a standardized means of combat.

Weihua Han, CAE, Synopsys

As a well-known Object-Oriented technique, class factory has actually been applied in VMM since inception. For instance, in the vmm atomic and scenario generators, by assigning different blueprints to randomized_obj and scenario_set[] properties, these generators can generate transactions with user specified patterns. Using the class factory pattern, users create an instance with a pre-defined method (such as allocate() or copy()) instead of the constructor. This pre-defined method will create an instance from the factory not just the type of the variable being assigned.

VMM1.2 now simplifies the application of the class factory pattern within the whole verification environment so that users can easily replace any kind of object, transaction, scenario and transactor by a similar object. Users can easily follow the steps below to apply the class factory pattern within the verification environment.

1. define “new”, “allocate”, “copy” methods for a class and create the factory for the class.

class vehicle_c extends vmm_object;

//defines the new function. each argument should have default values

function new(string name=”",vmm_object parent=null);

super.new(parent,name);

endfunction

//defines allocate and copy methods

virtual function vehicle_c allocate();

vehicle_c it;

it = new(this.get_object_name,get_parent_object());

allocate = it;

endfunction

virtual function vehicle_c copy();

vehicle_c it;

it = new this;

copy = it;

endfunction

//these two macros will define necessary methods for class factory and create factory for the class

`vmm_typename(vehicle_c);

`vmm_class_factory(vehicle_c);

endclass

`vmm_typename, `vmm_class_factory will implement the necessary methods to support the class factory pattern, like get_typename(), create_instance(), override_with_new(), override_with_copy(), etc.

Users can also use `vmm_data_member_begin and `vmm_data_member_end to implement the “new”, “copy”, “allocate” methods conveniently.

2. create an instance using the pre-defined “create_instance()” method

To use the class factory, the class instance should be created with pre-defined create_instance() method instead of the constructor. For example:

class driver_c extends vmm_object;

vehicle_c myvehicle;

function new(string name=”",vmm_object parent=null);

super.new(parent,name);

endfunction

task drive();

//create an instance from create_instance method

myvehicle = vehicle_c::create_instance(this,”myvehicle”);

$display(“%s is driving %s(%s)”, this.get_object_name(),

myvehicle.get_object_name(),

myvehicle.get_typename());

endtask

endclass

program p1;

driver_c Tom=new(“Tom”,null);

initial begin

Tom.drive();

end

endprogram

For this example, the output is:

Tom is driving myvehicle(class $unit::vehicle_c)

3.  define a new class

Let’s now define the following new class which is derived from the original class vehicle_c:

class sedan_c extends vehicle_c;

`vmm_typename(sedan_c);

function new(string name=”",vmm_object parent=null);

super.new(name,parent);

endfunction

virtual function vehicle_c allocate();

sedan_c it;

it = new(this.get_object_name,get_parent_object());

allocate = it;

endfunction

virtual function vehicle_c copy();

sedan_c it;

it = new this;

copy = it;

endfunction

`vmm_class_factory(sedan_c);

endclass

And we would like to create myvehicle instance from this new class without modifying driver_c class.

4. override the original instance or type with the new class

VMM1.2 provides two methods for users to override the original instances or type.

• override_with_new:(string name, new_class factory, vmm_log log,string fname=”",int lineno=0)

With this method, when create_instance() is called, a new instance of new_class will be created through facory.allocate() and returned.

• override_with_copy(string name, new_class factory,vmm_log log, string fname=”", int lineno=0)

With this method, when create_instance() is called, a new instanced of new_class will be created through factory.copy() and returned.

For both methods, the first argument is the instance name, as specified in the create_instance() method, which users hope to override with the type of new_class. Users can use powerful name matching mechanism defined in VMM to specify the override happens on dedicated instance or all the instances of one class in the whole verification environment.

The code below will override all vehicle_c instances with sedan_c type in the environment:

program p1;

driver_c Tom=new(“Tom”,null);

vmm_log log;

initial begin

//override all vehicle_c instances with type of sedan_c

vehicle_c::override_with_new(“@%*”,sedan_c::this_type,log);

Tom.drive();

end

endprogram

And the output of the above code is:

Tom is driving myvehicle(class $unit::sedan_c)

If users only want to override one dedicated instance with a copy of another instance, users can call override_with_copy using the following code:

vehicle_c::override_with_copy(“@%Tom:myvehicle”,another_sedan_c_instance,log);

As the above example shows, with the class factory pattern short-hand macros provided with VMM1.2, users can easily use class factories patterns to replace transactors, transactions and other verification components without modifying the testbench code. I find this very useful for increasing the reusability of verification components.

Nasib Naser, Phd, CAE, Synopsys

In this blog I will discuss the Use Model demonstrated in Figure 1 where a VMM layered testbench is used to verify an RTL DUT against a SystemC transaction level model. SystemVerilog allows for the creation of a reusable layered testbench architectures. The VMM methodology provides the basis for such a layered architecture. With a layered approach, transaction-level reference models can be easily integrated at the appropriate level to provide self-checking functions. In this Use Model the VMM Function layer is communicating with SystemC model using the TLI mechanism to perform read/write transactions, and using the same testbench scenarios the VMM command layer is driving the DUT at pin level.

Figure 1 – VMM driving TLM and RTL with checking

Synopsys’ VCS functional verification solution addresses the challenge of this use model with its SystemC-SystemVerilog Transaction-Level Interface (TLI). Using TLI SystemC interface methods can be invoked in SystemVerilog and vice versa. The **value add** for using TLI is that the SystemVerilog DPI based communication code that synchronizes both domains is automatically generated.

Let’s take a look at the various code components in SystemC and SystemVerilog based on the VMM methodology that enables such a verification use model. In the following example we define the read and write transactions as SC Interface methods.

class Buf_if: virtual public sc_interface {
public:

// do the pure virtual function read()/write() declarations here
virtual void read(unsigned int addr, unsigned int* data) = 0;

virtual void write(unsigned int addr, unsigned int data) = 0;
};

Following code shows VMM Transactor invoking SystemC transactions read and write at function layer. VMM transactor tb_mast is communicating to SystemC TLM using vmm channel tb_mast_out_ch1 and with the RTL model using the vmm channel tb_mast_out_ch2 channel, as shown in the following code:

class tb_master extends vmm_xactor;
virtual tb_if.master ifc;
tb_data_channel tb_master_in_ch;
tb_data_channel
tb_master_out_ch1;
tb_data_channel
tb_master_out_ch2;
…
extern virtual task main();
endclass: tb_master

task tb_master::main();
tb_data tr, tr_out;
super.main();
forever begin
...
// Send the Instruction to SC Reference Model
tb_master_out_ch2.put(tr_out);
// Send the Instruction to RTL
tb_mast_out_ch1.put(tr_out);
end
endtask: main

The following code shows VMM reference Transactors calling the SystemC transaction functions via the adaptor alu_tl_if_adpt_vlog which was automatically generated by VCS TLI.

class tb_ref extends vmm_xactor;

tb_data_channel     tb_ref_in_ch;

alu_tl_if_adpt_vlog alu_tl_if_adpt_vlog_inst0;

extern function new (string instance,
integer stream_id = -1,

tb_data_channel tb_ref_in_ch = null);

extern virtual task main();

endclass: tb_ref

task tb_ref::main();

super.main();

forever begin

…

case(tr_out.tb_data_type)

SA_SB_OP_GO : begin

alu_tl_if_adpt_vlog_inst0.write(addrA,a);

alu_tl_if_adpt_vlog_inst0.write(addrB,b);

alu_tl_if_adpt_vlog_inst0.write(addrOP,op);

alu_tl_if_adpt_vlog_inst0.read(addrOut,d);

tr_out.data_out = d;
end
…
endcase

…

end

endtask

This VCS TLI use model provides a complete and easy way to integrating blocking and non-blocking SystemC reference models into a VMM based multi-layer verification environment.

Srinivasan Venkataramanan & Bagath Singh, CVC Pvt. Ltd. Jaya Chelani, Quartics Technologies

Consider an SoC with several interfaces being verified.  It is quite common to have each interface report its activity via display messages.  Now let’s take up a case where-in a specific user is debugging say the AXI interface for failure, performance analysis etc.  While the full log file is providing the overall picture, it is quite possible to get lost quickly in the sea of messages.

Won’t it be nice to have AXI report all its activities to its own log file? This would greatly reduce the analysis/debug time for a given task.  However doing a change inside the testbench is not a good practice, if one were to do it via `ifdef, config etc.  Also, it may be restricted to only few tests/runs, and not for entire regression.

Huh, such a common scenario, you wonder.  Yes, and that’s why vmm_log has that capability built-in.  By default, loggers direct messages to STDOUT, but can be easily asked to direct them to a specific file. The function vmm_log::log_start(file_pointer) performs just this and can be used at will during run time.

1. program automatic sep_log_files;

2.   `include “vmm.sv”

3.   vmm_log axi_logger, cvc_prop_if_logger;

4.   int axi_fp;

5.   initial begin : b1

6.     axi_fp = $fopen (“axi.log”, “w”);

7.     axi_logger = new (“AXI Log”, “0″);

8.     cvc_prop_if_logger = new (“CVC Log”, “0″); // Defaults to STDOUT

9.     axi_logger.log_start(axi_fp); // AXI alone is being sent to a separate LOG file

10.    `vmm_note (axi_logger, “Message from AXI Interface”);

11.    `vmm_note (cvc_prop_if_logger, “Messages from CVC Proprietary Interface”);

12.   end : b1

13.  endprogram : sep_log_files

Line 6 opens a file named “axi.log” for writing. Line 9 ensures that all messages form axi_log to the new log file pointer (created in Line 6). Note that one can point to any logger inside the env via hierarchical path and hence this can be done at a testcase level, if desired.

A few additional notes

1. As these methods work on specific log instance, they ought to be used once the log instance is constructed, typically after the vmm_env::build() phase.

2. There is also a counterpart to stop logging vmm_log::log_stop(file_pointer) to a separate file (it continues sending messages to STDOUT).

3. By design, the vmm_log::log_start() sends a copy of the message to the specified file in addition to STDOUT. This is done so that the complete log (sent to STDOUT) is intact from all loggers in the environment.

4. If it is desired by the user to avoid the duplication of log messages, the user can use an explicit call to vmm_log::log_stop(STDOUT).

5. These two methods can also be useful in performance measurement during a specific time window in a simulation run. One can setup notifiers to indicate the start & stop of the time window, and run a parallel thread to correspondingly perform log_start()  log_stop(). More on this in another blog post.

Andrew Piziali, Independent Consultant

A question I am often asked is “When and where should I use functional coverage and code coverage?” Since the purpose of coverage is to quantify verification progress, the answer lies in understanding the coverage spaces implemented by these two kinds of coverage.

A coverage space represents a subset of the behavior of your DUV (design under verification), usually of a particular feature. It is defined by a set of metrics, each a parameter or attribute of the feature quantified by the space. For example, the coverage space for the ADD instruction of a processor may be defined by the product of the absolute values of ranges of the operands (remember “addends?”) and their respective signs. In order to understand the four kinds of coverage metrics, we need to discuss the coverage spaces from which they are constructed.

A coverage metric is determined by its source—implementation or specification—and its author—explicit or implicit. An implementation metric is derived from the implementation of the DUV or verification environment. Hence, the width of a data bus is an implementation metric, as is the module defining a class. Conversely, a specification metric is derived from the DUV functional or design specification. A good example is the registers and their characteristics defined in a specification.

The complementary coverage metric classification is determined by whether the metric is explicitly chosen by an engineer or implicit in the metric source. Hence, an explicit metric is chosen or invented by the verification engineer in order to quantify some aspect of a DUV feature. For example, processor execution mode might be chosen for a coverage metric. Alternatively, an implicit metric is inherent in the source from which the metric value is recorded. This means things like module name, line number and Boolean expression term are implicit metrics from a DUV or verification environment implementation. Likewise, chapter, paragraph, line, table and figure are implicit metrics from a natural language document, such as a specification.

Combining the two metric kinds—source and author—leads to four kinds of coverage metrics, each defining a corresponding kind of coverage space:

1. Implicit implementation metric

2. Implicit specification metric

3. Explicit implementation metric

4. Explicit specification metric

An example of an implicit implementation metric is a VHDL statement number. The register types and numbers defined by a functional specification are an implicit specification metric. Instruction decode interval is an explicit implementation metric. Finally, key pressed-to-character displayed latency is an example of an explicit specification coverage metric.

Each metric kind may be used to define an associated kind of coverage space. The astute reader may also wonder about coverage spaces defined by a mix of the above metric kinds. If such a hybrid space more precisely quantifies the verification progress of a particular feature, use it! To the best of my knowledge, you’d have to design and implement this space in much the same way as any functional coverage space because no commercial tool I am aware of offers this kind of coverage.

With an understanding of the kinds of coverage spaces, we can now classify functional and code coverage and figure out where they ought to be used. Functional coverage, making use of explicit coverage metrics—independent of their source—defines either an explicit implementation space or an explicit specification space. Code coverage tools provide a plethora of built-in implicit metric choices. Hence, it defines implicit implementation spaces. Where you want to measure verification progress relative to the DUV functional specification, where features are categorized and defined, functional coverage is the appropriate tool. Where you want to make sure all implemented features of the DUV have been exercised, you should use code coverage. Lastly, when your code coverage tool does not provide sufficient insight, resolution or fidelity into the behavior of the DUV implementation, functional coverage is required to complement the implicit spaces it does offer.

Functional coverage can tell you the DUV is incomplete, missing logic required to implement a feature or a particular corner case, whereas code coverage cannot. On the other hand, code coverage can easily identify unexercised RTL, while functional coverage cannot. Functional coverage requires a substantial up-front investment for specification analysis, design and implementation yet relieves the engineer of much back-end analysis. Code coverage, on the other hand, may be enabled at the flip of a switch but usually requires a lot of back-end analysis to sift the false positives from the meaningful coverage holes. Both are required—and complementary—but their deployment must be aligned with the stage of the project and DUV stability.

Some smart alec will point out that you can’t measure verification progress using coverage alone, and you’re right! Throughout this discussion I assume each feature, with its associated metrics, has corresponding checkers that pipe up when the DUV behavior differs from the specified behavior. (I’ll leave the topic of concurrent behavior recording and checking for another day.)

If you’d like to learn much more about designing, implementing, using and analyzing coverage, the following books delve much more deeply into verification planning, management and coverage model design:

• Functional Verification Coverage Measurement and Analysis, A. Piziali, 2008, Springer, ISBN 978-0-387-73992-2

• ESL Design and Verification, B. Bailey, G. Martin, A. Piziali, 2007, Elsevier, ISBN 0-12-373551-3, chapters 10 and 13

Janick Bergeron, Synopsys Fellow

The VMM Memory Allocation Manager (vmm_mam) can be used to manage a shared address space. It does not physically allocate memory but dishes out address ranges that are guaranteed to be previously unallocated.

By default, the only constraints on the Memory Allocation Manager are 1) the entire address range must not be already allocated, 2) the starting offset must be greater than or equal the base offset of the memory and 3) the ending offset must be less than or equal to the offset limit.

That’s it.

So randomly allocating memory will result in memory regions starting at random positions. For example, the following code allocates ten 4-byte regions in a 256-byte memory:

program test;

`include “vmm.sv”
`include “vmm_ral.sv”

initial
begin
vmm_mam_cfg    cfg;
vmm_mam        mgr;
vmm_mam_region bfr;

cfg = new;
cfg.n_bytes      = 1;
cfg.start_offset = 8′h00;
cfg.end_offset   = 8′hFF;

mgr = new(“Mem Mgr”, cfg);

repeat (10) begin
bfr = mgr.request_region(4);
$write(“%s\n”, bfr.psdisplay());
end
endprogram

And produce the following results:

['h0000000000000080:'h0000000000000083]
['h0000000000000015:'h0000000000000018]
['h00000000000000bd:'h00000000000000c0]
['h00000000000000b9:'h00000000000000bc]
['h00000000000000e6:'h00000000000000e9]
['h000000000000005f:'h0000000000000062]
['h000000000000008d:'h0000000000000090]
['h0000000000000025:'h0000000000000028]
['h0000000000000051:'h0000000000000054]
['h0000000000000087:'h000000000000008a]

but what if you needed the region to be aligned on quad-word boundaries?

Simple!

You can add constraints by extending the vmm_mam_allocator object and supplying the new allocator to the request_region() call or making it the default allocation policy on the Memory Allocation Manager instance:

program test;

`include “vmm.sv”
`include “vmm_ral.sv”

class qword_aligned_allocator extends vmm_mam_allocator;
constraint qword_aligned {
start_offset[1:0] == 0;
}
endclass

initial
begin
vmm_mam_cfg    cfg;
vmm_mam        mgr;
vmm_mam_region bfr;

cfg = new;
cfg.n_bytes      = 1;
cfg.start_offset = 8′h00;
cfg.end_offset   = 8′hFF;

mgr = new(“Mem Mgr”, cfg);

begin
qword_aligned_allocator alloc = new;

repeat (5) begin
bfr = mgr.request_region(4, alloc);
$write(“%s\n”, bfr.psdisplay());
end

mgr.default_alloc = alloc;

repeat (5) begin
bfr = mgr.request_region(4);
$write(“%s\n”, bfr.psdisplay());
end
end

end

endprogram

The resulting regions are now QWORD aligned:

['h00000000000000b4:'h00000000000000b7]
['h0000000000000064:'h0000000000000067]
['h00000000000000c0:'h00000000000000c3]
['h0000000000000008:'h000000000000000b]
['h00000000000000b8:'h00000000000000bb]
['h0000000000000048:'h000000000000004b]
['h0000000000000010:'h0000000000000013]
['h0000000000000018:'h000000000000001b]
['h00000000000000d4:'h00000000000000d7]
['h00000000000000fc:'h00000000000000ff]

Other kind of constraints can you add via a customized allocator include:

• Region must be within a 1-k address page
• 75% of regions must be in the upper half of the address space
• Regions must be allocated from a pool of pre-allocated regions

Remember that regions are allocated by randomizing the allocator object. That means that a highly directed and procedural allocation policy (such as the last one mentioned above) can be implemented using the post_randomize() method.

Once you have a region, if you have associated the Memory Allocation Manager with a memory RAL abstraction class, you may access it as if it were a tiny memory in and of itself. The integration of MAm and RAL effectively implements a mini virtual-memory system.