As is usually the case, staying ahead of the appetite of a high performance processor with the memory technology of the day was a major challenge. This processor consumed instructions and data far faster than current PC memory systems could supply. Fortunately, spatial and temporal locality–the tendency for memory accesses to cluster near common addresses and around the same time–were on our side. These could be exploited by a cache that would present a sufficiently fast memory interface to the processor while dealing with the sluggish PC memory system behind the scenes. However, this cache would require a three level, on-chip memory hierarchy that had never been seen before in a microprocessor. Could it be done?
The system level designer responsible the cache design–let’s call him “Ambrose”–managed to meet the performance requirements, yet with an exceedingly complex cache design. It performed flawlessly as a C model running application address traces, rarely stalling the processor on memory accesses. Yet, when its RTL incarnation was unleashed to the verification engineers, it stumbled … and stumbled badly. Each time a bug was found and fixed, performance took a hit while another bug was soon exposed. Before long we finally had a working cache but it unfortunately starved the processor. Coupled with the processor not making its clock frequency target and schedule slips, this product never got beyond the prototype stage, after burning through $35 million and 200 man-years of labor. Ouch! What can we learn about the role of the system level designer from this experience?
The system level designer faces the challenge of evaluating all of the product requirements, choosing implementation trade-offs that necessarily arise. The architectural requirements of a processor cache include block size, hit time, miss penalty, access time, transfer time, miss rate and cache size. It shares physical requirements with other blocks such as area, power and cycle time. However, of particular interest to us are its verification requirements, such as simplicity, limited state space and determinism.
The cache must be as simple as possible while meeting all of its other requirements. Simplicity translates into a shorter verification cycle because the specification is less likely to be misinterpreted (fewer bugs), fewer boundary conditions to be explored (smaller search space and smaller coverage models), less simulation cycles required for coverage closure and fewer properties to be proven. A limited state space also leads to a smaller search space and coverage model. Determinism means that from the same initial conditions and given the same cycle-by-cycle input the response of the cache is always identical from one simulation run to the next. Needless to say, this makes it far easier to isolate a bug than an ephemeral glitch that cannot be produced on demand. These all add up to a cost savings in functional verification.
Ambrose, while skilled in processor cache design, was wholly unfamiliar with the design-for-verification requirements we just discussed. The net result was a groundbreaking, novel, high-performance three level cache that could not be implemented.
“Coverage Closure” is the process used to reach 100% of your coverage goals. In a directed test methodology, it is simply the process of writing all of the testcases outlined in the verification plan. In a constrained-random methodology, it is the process of adding constraints, defining scenarios or writing directed tests to hit the uncovered areas in your functional and structural coverage model. In the latter case, it is a process that is time-consuming and challenging: you must reverse-engineer the design and verification environment to determine why specific stimulus must be generated to hit those uncovered areas.
Note that your first strategy should be to question the relevance of the uncovered coverage point. A coverage point describes an interesting and unique condition that must be verified. If that condition is already represented by another coverage point, or it is not that interesting (if no one on your team is curious about nor looking forward to analyzing a particular coverage point, then it is probably not that interesting), then get rid of the coverage point rather than trying to cover it.
Something that is challenging and time-consuming is an ideal candidate for automation. In this case, the Holy Grail is the automation of the feedback loop between the coverage metrics and the constraint solver. The challenge in automating that loop is correlating those metrics with the constraints.
For input coverage, this correlation is obvious. For every random variable in a transaction description, there is usually a corresponding coverage point, then cross coverage points for various combinations of random variables. VCS includes automatic input coverage convergence. It automatically generates the coverage model based on the constraints in a transaction descriptor, then will automatically tighten the constraints at run-time to reach 100% coverage in very few runs.
For internal or outputcoverage, the correlation is a lot more obscure. How can a tool determine how to tweak the constraints to reach a specific line in the RTL code, trigger a specific expression or produce a specific set of values in an output transaction? That is where newly acquired technology from Nusym will help. Their secret sauce traces the effect of random input values on expressions inside the design. From there, it is possible to correlate the constraints and the coverage points. Once this correlation is known, it is a relatively straightforward process to modify the constraints to target uncovered coverage points.
A complementary challenge to coverage closure is identifying coverage points that are unreachable. Formal tools, such as Magellan, can help identify structural coverage points that cannot be reached and thus further trim your coverage space. For functional coverage, that same secret sauce from Nusym can also be used to help identify why existing constraints are preventing certain coverage points from being reached.
Keep in mind that filling the coverage model is not the goal of a verification process: it is to find bugs! Ultimately, a coverage model is no different than a set of directed testcases: it will only measure the conditions you have thought of and consider interesting. The value of constraint-random stimulus is not just in filling those coverage models automatically, but also in creating conditions you did not think of. In a constrained-random methodology, the journey is just as interesting—and valuable—as the destination.
Here is a quick collection of various VMM updates from SNUG India 2010 – as seen by TeamCVC. Expect to hear more on VMM1.2 soon from us as now I have a young team all charged up with VMM 1.2 (thanks to Amit @SNPS). All the papers and presentations can be accessed through: http://www.synopsys.com/Community/SNUG/India/Pages/IndiaConferenceataGlance.aspx
TI’s usage of VMM 1.2 & RAL
In one of the well received papers, TI Bangalore talked about “Pragmatic Approach for Reuse with VMM1.2 and RAL “. The design is a complex digital display subsystem involving numerous register configurations. Not only handling the register configurations is a challenge, but also the ability to reuse of block level subenvs at system level with ease, and with minimal rework and reduced verification time. The author presented their success with VMM 1.2 & RAL to address these challenges.
Key elements touched up on advanced VMM are:
· TLM 2.0 communication mechanism
· VMM Multi-Stream Scenario gen (MSS)
· VMM RAL
Automated Coverage Closure with ECHO
It is real and live – automated coverage closure is slowly becoming reality atleast in select designs & projects. Having been attempted by various vendors for a while (see: http://www.cvcblr.com/blog/?tag=acc) VCS has added this under ECHO technology. At SNUG, TI presented their experience with ECHO & VMM-RAL. In their paper titled “Automating Functional Coverage Convergence and Avoiding Coverage Triage with ECHO Technology” TI described how an ECHO based methodology in a VMM RAL based environment, can in an automated manner close the feedback loop in targeting coverage groups involving register configuration combinations resulting in significant reduction in verification time.
WRED Verification with VMM
In her paper on “WRED verification with VMM”, Puja shared her usage of advanced VMM capabilities for a challenging verification task. Specifically she touched upon:
· VMM Multi-Stream Scenario gen
· VMM Datastream Scoreboard with its powerful “with_loss” predictor engine
· VMM RAL to access direct & indirect RAMs & registers
What we really liked is to see real application of some of these advanced VMM features – we were taught all of these during our regular CVC trainings and we even tried many of them on our own designs. It feels great to hear form peers on similar usage and to appreciate the value we derive out of VMM @CVC and the vibrant ecosystem that CVC creates around the same.
System-Level verification with VMM
Ashok Chandran, of Analog Devices presented their use of specialized VMM components in a system-level verification project. Specifically he touched upon specialized VMM base classes like vmm_broadcast and vmm_scheduler
At the end the audience learnt what are some of the unique challenges a SoC verification project can present. Even more interesting was the fact that the ever growing VMM seems to have solution for a wide variety of such problems, well thought-out upfront – Kudos to the VMM developers!
Ashok also briefed on his team’s usage of relatively new features in VMM such as vmm_record and vmm_playback and how it helps us to quickly replay streams.
On the tool side, a very useful feature for regressions is the usage of separate compile option in VCS.
VMM 1.2 for VMM users
Amit from SNPS gave a very useful and upto-the-point update on VMM 1.2 for long time VMM users. It was rejuvenating to listen to the VMM 1.2 run_tests feature and the implicit phasing techniques. Though look like little “magic” these features are bound to improve our productivity as there are lesser things to code-debug and move-on..
Amit also touched upon the use of TLM 2.0 ports and how they can be nicely used for integrating functional coverage, instead of using the vmm_callbacks.
The hierarchical component creation and configurations in VMM 1.2 puts us on track for the emerging UVM and is very pleasing to see how the industry keeps moving to more-n-more automation.
A truly vibrant ecosystem enabled by CVC -VMM Catalyst member
A significant addition to this year’s SNUG India was the DCE – Designer Community Expo – a genuine initiative by Synopsys to bring in partners to serve the larger customer base better all under one roof. CVC (www.cvcblr.com) being the most active VMM catalyst member in this region was invited to setup a booth showcasing its offerings. We gave away several books including our popular VMM adoption bookhttp://systemverilog.us/?p=14 and all the new SVA Handbook 2nd edition http://systemverilog.us/?p=16 .
Here is a snapshot of CVC’s booth with our VMM and other offerings.
I was recently talking with a friend about a subject near and dear to my heart: functional coverage. Our topic was block level or subsystem coverage models that are not ultimately satisfied until paired with corresponding system level context. The question at hand was “Is it possible to use conditional block level coverage as a proxy for system level coverage by recording coverage points at the block level, conditioned upon subsequent observation of system level context?”
For example, let’s say I have an SoC DUV (design under verification) that contains an MPEG-4 AVC (H.264) video encoder with a variety of features enabled by configuration registers. I need to observe the encoder operating in a subset of modes—referred to as “profiles” in H.264 parlance—defined by permutations of the features. In addition, each profile is only stressed when the encoder is used by an application for which the profile was designed. For example, the “high profile” is intended for high definition television while the “scalable baseline profile” was designed for mobile and surveillance use. The profiles may be simulated at the block level but the applications are only simulated at the system level. How do I design, implement and fill this H.264 coverage model so that the model is populated using primarily block level simulations, with their inherent performance advantage, while depending upon system level application context? May I even use the block level coverage results as a proxy for the corresponding system level coverage?
I think this nut is tougher to crack than the black walnuts that recently fell from my tree. The last time I tried to crack those nuts I resorted to vise grips from my tool chest, leading to a number of unintended, but spectacular, results, but I digress. My friend and I were able to define a partial solution but it wasn’t at all clear whether or not a viable solution exists that leverages the speed of block level simulations. After you finish reading this post, I’d like to hear your thoughts on the matter. This is how far we got.
As with the top level design of any coverage model, we started with its semantic description (sometimes called a “story line”):
Record the encoder operating in each of its profiles while in use by the corresponding applications.
Next, we identified the attributes required by the model and their values:
Attribute
Values
Profile
BP, CBP, Hi10P, Hi422P, Hi444PP, HiP, HiSP, MP, XP
Then, we finished the top level design using a second order model (simplified somewhat for clarity):
The left column of the table serves as row headings: Attribute, Value, Sampling Time and Correlation Time. The remaining cells of each row contain values for the corresponding heading. For example, the “Attribute” names are “Profile,” “Application” and “Feature.” The values of attribute “Profile” are BP, CBP, Hi10P, etc. The time at which each attribute is to be sampled is recorded in the “Sampling Time” row. The time that the most recently sampled attribute values are to be recorded as a set is when the “application [is] started,” the correlation time. Finally, the magenta cells define the value tuples that compose the coverage points of the model.
The model is implemented in SystemVerilog, along with the rest of the verification environment and we begin running block level simulations. Since no actual H.264 application is run at the block level, we need to invent an expected application value whenever we record a coverage point defined by an attribute value set, lacking only an actual application. Why not substitute a proxy application value, to be replaced by an actual application value when a similar system level simulation is run? For example, if in a block level simulation we observe profile BP and feature ASO ({BP, *, ASO}), we could substitute the value “need_low_cost” for “low_cost” for the application value, recording the tuple {BP, need_low_cost, ASO}. This becomes a tentative proxy coverage point. Later on, when we are running a system level simulation with an H.264 application, whenever we observe a {BP, low_cost, ASO} event, we would substitute the much larger set of {BP, need_low_cost, ASO} block level events for this single system level event, replacing “need_low_cost” with “low_cost.” This would allow us to “observe” the larger set of system level {BP, low_cost, ASO} events from the higher performance block level simulations. How can we justify this substitution?
We could argue that a particular system level coverage point is an element of a set of block level coverage points because (1) it shares the same subset of attribute values with the system level coverage point and (2) the block level simulation is, in a sense, a superset of the system level simulation because it implicitly abstracts away the additional detail available in the system level simulation. Is there any argument that the profile value BP and feature value ASO are the same in the two environments? The second reason is clearly open to discussion.
This brings us back to the opening question, can we use conditional block level coverage as a proxy for system level coverage by recording coverage points at the block level, conditioned upon subsequent observation of system level context? If so, is this design approach feasible and reasonable? If not, why not? Have at it!
Remember the kick-off of your last new project, when the road to tape-out was paved with good intentions? Architects were brainstorming novel solutions to customer requirements? Designers were kicking around better implementation solutions? Software engineers were building fresh new SCM repositories? And you, the verification engineer, were excitedly studying the new design and planning its verification? Throughout all of this early excitement, all sorts of good intentions were revealed. Addressing the life story of each intention would make a good short story, or even a novel! Since we don’t have room for that, let’s just focus on the design intentions.
Design intent, how the architect intended the DUV (design under verification) to behave, originates in the mind’s eye of the architect. It is the planned behavior of the final implementation of the DUV. Between original intent and implementation the DUV progresses through a number of representations, typically referred to as models, wherein intent is unfortunately lost. However, intent’s first physical representation, following conception, is its natural language specification.
We may represent the space of design behaviors as the following Venn diagram:1
Each circle—Design Intent (AEGH), Specification (BEFH) and Implementation (CFGH)—represents a set of behaviors. AEGH represents the set of design requirements, as conveyed by the customer. BEFH represents the intent captured in the specification(s). CFGH represents intent implemented in the final design. The region outside the three sets (D) represents unintended, unspecified and unimplemented behavior. The design team’s objective is to bring the three circular sets into coincidence, leaving just two regions: H (intended, specified and implemented) and D. By following a single design intention from set Design Intent to Specification to Implementation, we learn a great deal about how design intent is lost.
An idea originally conceived appears in set AEGH (Design Intent) and, if successfully captured in the specification, is recorded in set EH. However, if the intent is miscommunicated or not even recorded and lost, it remains in set A. There is the possibility that a designer learns of this intent, even though it not recorded in the specification, and recaptures it in the design. In that case we find it in set G: intended, implemented, but unspecified.
Specified intent is recorded in set BEFH. Once the intent is captured in the specification it must be read, comprehended and implemented by the designer. If successful, it makes it to the center of our diagram, set H: intended, specified and implemented. Success! Unfortunately some specified requirements are missed or misinterpreted and remain unimplemented, absent from the design as represented by set E: intended, specified but unimplemented. Sometimes intent is introduced into the specification that was never desired by the customer and (fortunately) never implemented, such as set B: unintended, unimplemented, yet specified. Unfortunately, there is also unintended behavior that is specified and implemented as in set F. This is often the result of gratuitous embellishment or feature creep.
Finally, implemented intent is represented by set CFGH, all behaviors exhibited by the design. Those in set G arrived as originally intended but were never specified. Those in set H arrived as intended and specified. Those in set F were introduced into the specification, although unintended, and implemented. Behaviors in set C were implemented, although never intended nor specified! In order to illustrate the utility of this diagram, let’s consider a specific example of lost design intent.
We can think of each part of the development process as building a model. Teams write documentation as a specification model, such as a Microsoft Word document. System architects build an abstract algorithmic system model that captures the specification model requirements, using SystemC or C++. Designers build a synthesizable RTL model in Verilog or VHDL. Verification engineers build an abstract problem space functional model in SystemVerilog, SVA and/or e.
If any member of the team fails to implement an element of the upstream, more abstract model correctly (or at all), design intent is lost. The verification engineer can recover this lost design intent by working with all members of the team and giving the team observability into all models.
Consider an example where the system model (ex. C++) uses a finite state machine (FSM) to control the data path of the CPU whereas the specification model (ex. MS Word) implies how the data path should be controlled. This could be a specification ambiguity that the designer ignores, implementing the data path controller in an alternate manner, which he considers quite efficient.
Some time later the system architect may tell the software engineers that they do not need to implement exclusive locking because the data path FSM will handle concurrent writes to same address (WRITE0, WRITE1). However, the designer’s implementation is not based on the system model FSM but rather the specification model. Therefore, exclusive locking is required to prevent data corruption during concurrent writes. We need to ask: How can the verification engineer recover this lost design intent by observing all models?
Synopsys illustrates a complete solution to the problem in a free verification planning seminar that dives deep into this topic. However, for the purposes of this blog we offer a simplified example, using the design and implementation of a coverage model:
Analyze the specification model along with the system model
Identify the particular feature (ex. mutex FSM) and write its semantic description
Determine what attributes contribute to the feature behavior
Identify the attribute values required for the feature
Determine when the feature is active and when the attributes need to be sampled and correlated
This top-level design leads to:
Feature
CPU_datapath_Ctrl
Description
Record the state transitions of the CPU data path controller
Attribute
Data path controller state variable
Attribute Values
IDLE, START, WRITE0, WRITE1
Sample
Whenever the state variable is written
The verification engineer can now implement a very simple coverage model to explicitly observe the system model, ensuring entry to all states:
The verification engineer can link the feature “CPU_datapath_Ctrl” in his verification plan to the cntlr_cov covergroup. Running the system model with the verification environment and RTL implementation will reveal that bin “t4″ is never visited, hence state transition WRITE0 to WRITE1 is never observed. The team can review the verification plan to determine if the intended FSM controller should be improved in the design to conform to all design intent.
Although there are many other subsets of the design intent diagram we could examine, it is clear that a design intention may be lost through many recording and translation processes. By understanding this diagram and its application, we become aware of where intent may be lost or corrupted and ensure that our good intentions are ultimately realized.
1The design intent diagram is more fully examined in the context of a coverage-driven verification flow in chapter two of the book Functional Verification Coverage Measurement and Analysis (Piziali, 2004, Springer, ISBN 978-0-387-73992-2).
Have you ever been reviewing a specification and asked yourself “Say what?!” Then you’re not alone! One of the most challenging tasks we face as verification engineers is understanding design specifications. What does the architect mean when she writes “The conflabulator remains inoperative until triggered by a neural vortex?” Answering that question is part of specification analysis, the first step in planning the verification of a design, the subsequent steps being coverage model design, verification environment implementation, and verification process execution.
The specifications for a design—DUV, or “design-under-verification” for our purposes—typically include a functional specification and a design specification. The functional specification captures top level, opaque box, and implementation- independent requirements. Conversely, the design specification captures internal, clear box, implementation dependent behaviors. Each is responsible for conveying the architect’s design intent at a particular abstraction level to the design and verification teams. Our job is to ultimately comprehend these specifications in order to understand and quantify the scope of the verification problem and specify its solution. This comprehension comes through analyzing the specifications.
In order to understand the scope of the verification problem, the features of the DUV and their relationships must be identified. Hence, specification analysis is sometimes referred to as feature extraction. The features are described in the specifications, ready to be mined through our analysis efforts. Once extracted and organized in the verification plan, we are able to proceed to quantifying the scope and complexity of each by designing its associated coverage model. How do we tackle the analysis of specifications ranging from tens to hundreds of pages? The answer depends upon the size of the specification and availability of machine-guided analysis tools. For relatively small specifications, less than a hundred pages or so, bottom-up analysis ought to be employed. Specifications ranging from a hundred pages and beyond require top-down analysis.
Bottom-up analysis is the process of walking through each page of a specification: section-by-section, paragraph-by-paragraph, and sentence-by-sentence. As we examine the text, tables and figures, we ask ourselves what particular function of the DUV is addressed? What behavioral requirements are imposed? What verification requirements are implied? Is this feature amenable to formal verification, constrained random, or a hybrid technology? If formal is applicable, how might I formulate a declarative statement of the required behavior? What input, output and I/O coverage is needed? If this feature is more amenable to constrained random simulation, what are the stimulus, checking and coverage requirements?
Each behavioral requirement is a feature to be placed in the verification plan, in either the functional or design requirements sections, as illustrated below:
1 Introduction ………………………………… what does this document contain?
Bottom-up analysis is amenable to machine-guided analysis, wherein an application presents a specification before the user. For each section of the spec, perhaps for each sentence, the tool asks if this describes a feature, what are its property, stimulus, checking and coverage requirements, and records this information so that it may be linked to the corresponding section of the verification plan. This facilitates keeping the specifications and the verification plan synchronized. The verification plan is incrementally constructed within a verification plan integrated development environment (IDE).
The alternative to bottom-up analysis is analyzing a specification from the top down, required for large specifications. Your objective here is to bridge the intent abstraction gap between the detail of the specification and the more abstract, incrementally written verification plan. Behavioral requirements are distilled into concise feature descriptions, quantified in their associated coverage models. Top-down analysis is conducted in brainstorming sessions wherein representatives from all stakeholders in the DUV contribute. These include the systems engineer, verification manager, verification engineer, hardware designer and software engineer. After the verification planning methodology is explained to all participants, each engineer contributing design intent explains their part of the design. The design is explored through a question-and-answer process, using a whiteboard for illustration. In order to facilitate a fresh examination of the design component, no pre-written materials should be used.
Whether bottom-up or top-down analysis is used, each design feature should be a design behavioral requirement, stating the intended behavior of the DUV. Both the data and temporal behaviors of each feature should be recorded. In addition to recording the name of each feature, the behavior should be summarized in a sentence or two semantic description. Optionally, design and verification responsibilities, technical references, schedule information and verification labor estimates may be recorded. If the verification plan is written in Microsoft Word, Excel or in HVP1 plain text, it may drive the downstream verification flow, serving as a design-specific verification user interface.
The next time you ask “Say what?!,” make sure you are methodically analyzing the specification using either of the above approaches and don’t hesitate to contact the author directly. Many bugs discovered during these exchanges are the least expensive of all!
You are using coverage, along with other metrics, to measure verification progress as part of your verification methodology.12 Yet, lurking in the flow are the seeds of a bug escape that will blindside you. How so?
Imagine you are responsible for verifying an in-order, three-way x86 superscalar processor in the last millennium, before the Constrained Random Generation. Since your management wouldn’t spring for an instruction set architecture (ISA) test generator, you hired a team of new college grads to write thousands of assembly language tests. Within the allocated development time, the tests were written, they were functionally graded and achieved 100% coverage, and they all finally passed. Yeah! But, not so fast …
When first silicon was returned and Windows was booted on the processor, it crashed. The diagnosis revealed a variant of one of the branch instructions was misbehaving. (This sounds better than “It had a bug escape.”) How could this be? We reviewed our branch instruction coverage models and confirmed they were complete. Since all of the branch tests passed, how could this bug slip through?
Further analysis revealed this branch instruction was absent from the set of branch tests yet used in one of the floating point tests. Since the floating point test was aimed at verifying the floating point operation of the processor, we were not surprised to find it was insensitive to a failure of this branch instruction. In other words, as long as the floating point operations verified by the test behaved properly, the test passed, independent of the behavior of the branch instruction. From a coverage aspect, the complete ISA test suite was functionally graded rather than each sub-suite graded according to its functional requirements. Hence, we recorded full coverage.
The problem was now clear: the checking and coverage aspects of each test were not coupled, conditioning coverage recording on passing checked behavior. If we had either (1) functionally graded each test suite only for the functionality it was verifying or (2) conditionally recorded each coverage point based upon a corresponding check passing, this bug would not have slipped through. Using either approach, we would have discovered this particular branch variant was absent from the branch test suite. In the first case, that coverage point would have remained empty for the branch test suite. Likewise, in the second case we would not have recorded the coverage point because no branch instruction check would have been activated and passed.
Returning to the 21st century, the lesson we can take away from this experience is that coverage—functional, code and assertion—is suspect unless, during analysis, you confirm that for each coverage point a corresponding checker was active and passed. From the perspective of implementing your constrained random verification environment, each checker should emit an event (or some other notification), synchronous with the coverage recording operation, indicating it was active and the functional behavior was correct. The coverage code should condition recording each coverage point on that event. If you are using a tool like VMM Planner to analyze coverage, you may use its “-feature” switch to restrict the annotation of feature-specific parts of your verification plan to the coverage database(s) of that feature’s test suite.
You might ask if functional qualification3 would address this problem. Functional qualification answers the question “Will my verification environment detect, propagate and report each functional bug?” As such, it provides insight into how well your environment detects bugs but says nothing about the quality of the coverage aspect of the environment. I will address this topic in a future post if there is sufficient interest.
Remember, make your coverage count by coupling checking with coverage!
A question I am often asked is “When and where should I use functional coverage and code coverage?” Since the purpose of coverage is to quantify verification progress, the answer lies in understanding the coverage spaces implemented by these two kinds of coverage.
A coverage space represents a subset of the behavior of your DUV (design under verification), usually of a particular feature. It is defined by a set of metrics, each a parameter or attribute of the feature quantified by the space. For example, the coverage space for the ADD instruction of a processor may be defined by the product of the absolute values of ranges of the operands (remember “addends?”) and their respective signs. In order to understand the four kinds of coverage metrics, we need to discuss the coverage spaces from which they are constructed.
A coverage metric is determined by its source—implementation or specification—and its author—explicit or implicit. An implementation metric is derived from the implementation of the DUV or verification environment. Hence, the width of a data bus is an implementation metric, as is the module defining a class. Conversely, a specification metric is derived from the DUV functional or design specification. A good example is the registers and their characteristics defined in a specification.
The complementary coverage metric classification is determined by whether the metric is explicitly chosen by an engineer or implicit in the metric source. Hence, an explicit metric is chosen or invented by the verification engineer in order to quantify some aspect of a DUV feature. For example, processor execution mode might be chosen for a coverage metric. Alternatively, an implicit metric is inherent in the source from which the metric value is recorded. This means things like module name, line number and Boolean expression term are implicit metrics from a DUV or verification environment implementation. Likewise, chapter, paragraph, line, table and figure are implicit metrics from a natural language document, such as a specification.
Combining the two metric kinds—source and author—leads to four kinds of coverage metrics, each defining a corresponding kind of coverage space:
Implicit implementation metric
Implicit specification metric
Explicit implementation metric
Explicit specification metric
An example of an implicit implementation metric is a VHDL statement number. The register types and numbers defined by a functional specification are an implicit specification metric. Instruction decode interval is an explicit implementation metric. Finally, key pressed-to-character displayed latency is an example of an explicit specification coverage metric.
Each metric kind may be used to define an associated kind of coverage space. The astute reader may also wonder about coverage spaces defined by a mix of the above metric kinds. If such a hybrid space more precisely quantifies the verification progress of a particular feature, use it! To the best of my knowledge, you’d have to design and implement this space in much the same way as any functional coverage space because no commercial tool I am aware of offers this kind of coverage.
With an understanding of the kinds of coverage spaces, we can now classify functional and code coverage and figure out where they ought to be used. Functional coverage, making use of explicit coverage metrics—independent of their source—defines either an explicit implementation space or an explicit specification space. Code coverage tools provide a plethora of built-in implicit metric choices. Hence, it defines implicit implementation spaces. Where you want to measure verification progress relative to the DUV functional specification, where features are categorized and defined, functional coverage is the appropriate tool. Where you want to make sure all implemented features of the DUV have been exercised, you should use code coverage. Lastly, when your code coverage tool does not provide sufficient insight, resolution or fidelity into the behavior of the DUV implementation, functional coverage is required to complement the implicit spaces it does offer.
Functional coverage can tell you the DUV is incomplete, missing logic required to implement a feature or a particular corner case, whereas code coverage cannot. On the other hand, code coverage can easily identify unexercised RTL, while functional coverage cannot. Functional coverage requires a substantial up-front investment for specification analysis, design and implementation yet relieves the engineer of much back-end analysis. Code coverage, on the other hand, may be enabled at the flip of a switch but usually requires a lot of back-end analysis to sift the false positives from the meaningful coverage holes. Both are required—and complementary—but their deployment must be aligned with the stage of the project and DUV stability.
Some smart alec will point out that you can’t measure verification progress using coverage alone, and you’re right! Throughout this discussion I assume each feature, with its associated metrics, has corresponding checkers that pipe up when the DUV behavior differs from the specified behavior. (I’ll leave the topic of concurrent behavior recording and checking for another day.)
If you’d like to learn much more about designing, implementing, using and analyzing coverage, the following books delve much more deeply into verification planning, management and coverage model design:
Functional coverage plays an essential role in Coverage Driven Verification. In this blog, I’ll explain a modular way of modeling and implementing functional coverage models.
SystemVerilog users can take the advantage of the “covergroup” construct to implement functional coverage. However this is not enough. The VMM methodology provides some important design-independent guidelines on how to implement functional coverage in a reusable verification environment.
Here are a few guidelines from the VMM book that I consider very important for implementing a functional coverage model:
“Stimulus coverage shall be sampled after submission to the design” because in some cases it is possible that not all generated transactions will be applied to DUT.
“Stimulus coverage should be sampled via a passive transactor stack” for the purpose of reuse so that the same implementation can be used in a different stimulus generation structure. For example in another verification environment, the stimulus may be generated by a design block instead of testbench component.
“Functional coverage should be associated with testbench components with a static lifetime” to avoid creating a large number of coverage group instances. So “Coverage groups should not be added to vmm_data class extensions”. These static testbench components include monitors, generators, self-checking structure, etc.
“The comment option in covergroup, coverpoint, and cross should be used to document the corresponding verification requirements”.
You can find some more details on these and other functional coverage related guidelines in the VMM book, pages 263-277.
In an earlier post “Did you notice vmm_notify?”, Janick showed how vmm_notify can be used to connect a transactor to a passive observer like a functional coverage model. Here, let me borrow his idea to implement the following VMM-based functional coverage example.
There is some random property defined in the transaction class. We would like to collect the coverage information for these properties in our coverage class, for example “format” and “src”.
2. A generic subscribe class
1. class subscribe #(type T = int) extends vmm_notify_callbacks;
2. local T obs;
3. function new(T obs, vmm_notify ntfy, int id);
4. this.obs = obs;
5. ntfy.append_callback(id, this);
6. endfunction
7. virtual function void indicated(vmm_data status);
8. this.obs.observe(status);
9. endfunction
10. endclass
The generic subscribe class specifies the behavior for “indicated”method which will be called when vmm_notify “indicate” method is called. Then with using this generic subscribe class, functional coverage and other observer models only need to implement the desired behavior with “observe” method. Please note that in line 8 the vmm_data object is passed to “observe” method through vmm notification status information.
The coverage group is implemented in coverage class eth_cov. And this coverage class is registered to one vmm_notify service through scubscribe class. The coverage group is sampled in “observe” method so when the notification is indicated the interesting properties will be sampled.
4. Monitor
1. class eth_mon extends vmm_xactor;
2. int OBSERVED;
3. eth_frame tr; // Contains reassembled eth_frame transaction
4. function new(…)
5. OBSERVED = this.notify.configure();
6. …
7. endfunction
8. protected virtual task main();
9. forever begin
10. tr = new;
11. //catch transaction from interface
12. …
13. this.notify.indicate(OBSERVED,tr);
14. …
15. end
16. endtask
17. endclass
The monitor extracts the transaction from the interface then it indicates the notification with the transaction as the status information (line 13).
In the verification environment, the coverage object is created with the monitor notification service interface and the proper notification ID.
There are other ways to implement functional coverage in VMM based environments. For example a callback-based implementation is used in the example located under sv/examples/std_lib/vmm_test in the VMM package which can be downloaded from www.vmmcentral.org.
I haven’t discussed assertion coverage in this post, which are another important type of “functional coverage”. If you are interested in using assertions for functional coverage please check out chapter 3 and chapter 6 in the VMM book for its suggestions.
Andrew Piziali, Independent Consultant
Andrew Piziali, Independent Consultant
Weihua Han, CAE, Synopsys