Automating Coverage Closure
Posted by Janick Bergeron on July 5th, 2010
“Coverage Closure” is the process used to reach 100% of your coverage goals. In a directed test methodology, it is simply the process of writing all of the testcases outlined in the verification plan. In a constrained-random methodology, it is the process of adding constraints, defining scenarios or writing directed tests to hit the uncovered areas in your functional and structural coverage model. In the latter case, it is a process that is time-consuming and challenging: you must reverse-engineer the design and verification environment to determine why specific stimulus must be generated to hit those uncovered areas.
Note that your first strategy should be to question the relevance of the uncovered coverage point. A coverage point describes an interesting and unique condition that must be verified. If that condition is already represented by another coverage point, or it is not that interesting (if no one on your team is curious about nor looking forward to analyzing a particular coverage point, then it is probably not that interesting), then get rid of the coverage point rather than trying to cover it.
Something that is challenging and time-consuming is an ideal candidate for automation. In this case, the Holy Grail is the automation of the feedback loop between the coverage metrics and the constraint solver. The challenge in automating that loop is correlating those metrics with the constraints.
For input coverage, this correlation is obvious. For every random variable in a transaction description, there is usually a corresponding coverage point, then cross coverage points for various combinations of random variables. VCS includes automatic input coverage convergence. It automatically generates the coverage model based on the constraints in a transaction descriptor, then will automatically tighten the constraints at run-time to reach 100% coverage in very few runs.
For internal or output coverage, the correlation is a lot more obscure. How can a tool determine how to tweak the constraints to reach a specific line in the RTL code, trigger a specific expression or produce a specific set of values in an output transaction? That is where newly acquired technology from Nusym will help. Their secret sauce traces the effect of random input values on expressions inside the design. From there, it is possible to correlate the constraints and the coverage points. Once this correlation is known, it is a relatively straightforward process to modify the constraints to target uncovered coverage points.
A complementary challenge to coverage closure is identifying coverage points that are unreachable. Formal tools, such as Magellan, can help identify structural coverage points that cannot be reached and thus further trim your coverage space. For functional coverage, that same secret sauce from Nusym can also be used to help identify why existing constraints are preventing certain coverage points from being reached.
Keep in mind that filling the coverage model is not the goal of a verification process: it is to find bugs! Ultimately, a coverage model is no different than a set of directed testcases: it will only measure the conditions you have thought of and consider interesting. The value of constraint-random stimulus is not just in filling those coverage models automatically, but also in creating conditions you did not think of. In a constrained-random methodology, the journey is just as interesting—and valuable—as the destination.
July 15th, 2010 at 10:18 am
Hey quite a cool article….
Even I have expressed my views on this topic…..
http://www.nishantverma.com/2010/06/regression-testing.html
July 22nd, 2010 at 4:00 pm
I’ll believe in an automatic solution for creating and closing input coverage. There is a 1-1 relationship between a random variable and a coverage point.
I am very skeptical about the automation of input to internal or output coverage. Any reasonably complex design will have a VERY complex relationship between the inputs and outputs that includes fixed design configuration (pins and registers), the current state of the design and the applied inputs. I would challenge this tool to automate closure on even something as simple as a PCIE DMA controller. Can coverage and constraints be written in a natural way or does the tool require some special assistance? What about capacity issues that pure formal verification tools still struggle with? Your diagram also shows coverage on the DUT. Does this imply RTL assertions? In many cases collecting coverage from a reference model is the most efficient way to collect coverage at a higher level of abstraction. Using this example this would be coverage on a complete DMA operation VS the individual register operations to configure that operation. Would the tool be able to utilize coverage in this form?
Arguably the reality of formal tools is falling well short of the marketing promises. I see lots of hype regarding nusym in deepchip (http://www.deepchip.com/items/0475-02.html) but not much in the way of actual users saying it really saved time. Is there any customer feedback on real designs?
July 22nd, 2010 at 8:04 pm
Andrew,
Your skepticism is understandable and you certainly raise some very legitimate points, let me try to address them in order.
1. Coverage Target Complexity
As you correctly pointed out not all coverage points are created equal. Automatic convergence on stimulus coverage points
is relatively easy whereas on coverage points that sample say checker signals, and capture end to end design
functionality is extremely difficult. However, there is a wide of coverage targets that lie in between so the question
becomes how far along this continuum can the technology be pushed.
2. Coverage/Constraints Description
There are no restrictions on how either the class constraints or the coverage targets can be described. From the get-go
the technology was designed to be used on existing designs and verification environments without the need
to capture constraints/coverage targets in different formats. Currently, SystemVerilog and Vera are supported.
3. Capacity
Although it makes use of formal techniques, the technology is not based on pure formal analysis and does therefore not
exhibit the same memory behavior as traditional formal tools. The memory scalability profile is closer to that of a
simulator although in absolute terms is will obviously use more memory.
4. RTL assertions
There is nothing intrinsic to the technology that would prevent it from targeting cover properties and assertions.
5. Customer feedback
There is another interesting article on Deepchip (http://deepchip.com/items/0479-05.html), not sure if you had a chance
to read it.
Ultimately, the success of the technology will be determined by its ability to improve the customer’s coverage closure
process. That process involves not only targeting coverage points on low probability paths but also identifying
over-constrained randomization, un-reachable coverage points, testbench bugs, etc. Even if this technology won’t be able
to hit any cover point in any design regardless of complexity, it will surely provide its user with a wealth of useful
information that would otherwise require long hours of manual effort to collect. This by itself will significantly improve
what is now arguably the most painful verification task.
July 28th, 2010 at 11:24 am
Hi Alex,
thanks for the additional information. The deepchip link and an India Snug 2010 article about Echo both seem to be about closing configuration coverage. To be honest, that is under direct user (environment) control so is a pretty simple problem.
I’m more interested in how the tool fares in complicated situations. I recently went through coverage closure for an NPU block the old fashioned way and would like to describe one particularly difficult coverage point.
An instruction would cause a local ring buffer to move through a packet. This instruction specified how far to move in the packet and the minimum data available in the buffer. The instruction could have both immediate and register operands. Zero, one or two requests for more data could be issued by the DUT depending on the instruction and current state of the buffer. The state of this internal buffer was affected by previously executed instructions _and_ direct environment randomization at safe times in the instruction stream (to provide good coverage). If a request for more data was issued, a response was randomized by the environment. The “simple” coverage point we were interested in was ensuring both the first and second response had started and ended at all words in the buffer (i.e. responses had been written starting and ending at all buffer locations).
If this technology can identify the constraints that need to be modified to close this coverage point I would be impressed. This alone is difficult (manually) given the interconnectedness and non-liner relationships. If the tool could _automate_ the closure of this coverage point I would be sold! Do you have any comments about this problem?
Closing _code_ coverage can also be a painfully manual and time consuming process. Do you have any comments about applying this technology? A user could always write an RTL internal coverage point that _directly_ correlates to unexecuted code but if the technology automated this process that would also be impressive.
So, I’m still skeptical but hoping to give this a try….