Negative Verification

The objective of negative verification is to check that an error is detected when present. It is actually one of the most important aspect of verification because undetected errors do not “exists”. It is thus important that you make sure that your monitor properly issues an error message when the errors it is supposed to detect actually occur.

But how do you distinguish between an error that is supposed to occur (and is not an error) and one that is unexpected (and actially is an error)? How do you prevent the former from causing your test to fail but not the latter? And how do you make sure that the expected error message indeed came out? That’s where the message demotion and catching capabilities of the VMM message service come into play. And we use these capabilities in verifying the VMM library itself!

Take the VMM Data Stream Scoreboard, for example. It has three pre-defined methods for checking an observed response against the expected response. How would you know if your integration of it worked properly unless you injected an error in your testcase? Or if you decide to write your own “expect()” method, how would you verify that it detects errors?

Let me illustrates how that could be done using the APB bus examples located in $VMM_HOME/sv/examples/sv/apb_bus in the OpenSource distribution.

The example provides a simple positive test that does not expect any errors. If you invoke “make”, you should see the following message:

Simulation PASSED on /./ (/./) at 6010 (0 warnings, 0 demoted errors & 0 demoted warnings)

To check that the scoreboard would indeed identify a bad transaction, should it see one, we need to break the correct functionality of the environment. So let’s create a test that will corrupt a transaction in each of the streams. That can be easily accomplished by registering a callback after the scoreboard integration callback that will modify the transaction before it is executed. For example, let’s corrupt the address of the N^th transaction targetting slave #N, for each of the three slaves:

class apb_master_to_sb_inj extends apb_master_cbs;
   int n[3] = '{0, 0, 0}; 

   virtual task pre_cycle(apb_master xactor,
                          apb_rw     cycle,
                          ref bit    drop);
      // Corrupt the address on the Nth transaction on slave #N
      if (this.n[cycle.addr[9:8]]++ == cycle.addr[9:8]) begin
         cycle.addr[7:0] = ~cycle.addr[7:0];
      end
   endtask: pre_cycle 

endclass: apb_master_to_sb_inj

We then inject the error in a new tests:

initial
begin
   tb_env env = new;
   env.build();
   begin
      apb_master_to_sb_inj inj = new;
      env.mst.append_callback(inj);
   end
   env.run();
   $finish();
end

and voila! Three streams, three failures:

Simulation *FAILED* on /./ (/./) at 6010: 3 errors, 0 warnings

It is a good idea to include this test in your regression suite. But how to make it pass when it really needs to fail? The new VMM message catcher to the rescue! Simply define a handler that will trap and mask the following error messages:

!ERROR![FAILURE] on Data Stream Scoreboard(Master->Slave) at                   70:
    In-order packet was not expected:
       Actual: APB WRITE @ 0x000000db = 0xd612ef36
       Expect: APB WRITE @ 0x00000024 = 0xd612ef36.

First, let’s define a message catcher that will turn all messages into normal note messages. For safety reasons, it is a good practices to count the number of such demotions to ensure that only the expected messages where demoted:

class handler extends vmm_log_catcher; 

   int n = 0; 

   virtual function void caught(vmm_log_msg msg);
      msg.effective_typ = vmm_log::NOTE_TYP;
      msg.effective_severity = vmm_log::NORMAL_SEV;
      this.n++;
      this.throw(msg);
   endfunction 

endclass

then register this message catcher with the appropriate message service interface instance, specifying the appropriate message content. At the end of the test, simply check that only three messages were demoted:

initial
begin
   tb_env env = new;
   handler hdlr = new;
   env.build();
   env.m2s.log.catch(hdlr, .text("/In-order packet was not expected/"));
   begin
      apb_master_to_sb_drop inj = new;
      env.mst.append_callback(inj);
   end
   env.run();
   if (hdlr.n != 3) begin
      `vmm_error(log, "The scoreboard did not properly detect errors");
   end
   $finish();
end

And now the negative test will succesfully report that the scoreboard detects errors:

Normal[NOTE] on Data Stream Scoreboard(Master->Slave) at                   70:
    In-order packet was not expected:
       Actual: APB WRITE @ 0x000000db = 0xd612ef36
       Expect: APB WRITE @ 0x00000024 = 0xd612ef36.
Normal[NOTE] on Data Stream Scoreboard(Master->Slave) at                 1090:
    In-order packet was not expected:
       Actual: APB WRITE @ 0x000000ca = 0xe6aa5052
       Expect: APB WRITE @ 0x00000035 = 0xe6aa5052.
Normal[NOTE] on Data Stream Scoreboard(Master->Slave) at                 2030:
    In-order packet was not expected:
       Actual: APB READ @ 0x000000bf = 0xxxxxxxxx
       Expect: APB READ @ 0x00000040 = 0x57879754.
Simulation PASSED on /./ (/./) at 6010 (0 warnings, 0 demoted errors & 0 demoted warnings)