Simple Mailbox usage: woes, solution, and debug-automation thoughts
Posted by Shankar Hemmady on October 5th, 2009
Ramanathan Sambamurthy, Cisco Systems
Ajeetha Kumari & Srinivasan Venkataramanan, Contemporary Verification Consultants
We recently experienced a memory blow-up when an environment used for typical constrained random verification was used to apply a stress test. A quick background on the environment: it had a complex mix of C reference models, SystemVerilog generator, monitor and some checkers. The usual Memory Leak suspects of C /HDL boundary and dangling pointers were suggested. We tried using DVE-CBug (C-debugger). Not so lucky this time
Then we tried using the dynamic memory profiler within VCS, it provided vital clues pointing to the SV-TestBench code. After some more debug, we finally narrowed it down to a few Mailboxes on the monitor side. Two had recently been added for enhanced checking purposes with their consumers (i.e. checkers) still maturing. While the typical random tests didn’t show the issue, with stress tests with large number of packets, unsunk mailboxes created a memory leak!
The Mailboxes were constructed in default fashion as:
this.out_mbx = new();
endfunction : new
endclass : new_mon
Spot anything wrong above? The Mailbox is UNSIZED and hence never blocked. With no “consumer”, the monitor pushed packets into this Mailbox tirelessly leading to an eventual memory blow-up.
Interestingly, we addressed this exact topic in our VMM adoption book (http://www.systemverilog.us/vmm_adoption) in Chapter-8 Advanced Topics (Section 8.3). Let’s do some reality check – first and foremost, this is bad code, let us admit it to ourselves. It is obvious in hindsight. We should have done possibly one of the following:
· Used sized mailboxes
· Used a wrapper class around plain SV Mailbox to aid in debug around the “put”
· Even better, used a VMM_Channel, with tons of built-in features to aid in such cases (will be able to debug if not do it correct by construction). Refer to Chapter-2 http://www.systemverilog.us/vmm_adoption/vmm_isbn_0970539495.pdf
· Added an “artificial sink” for every such monitor (Something that we at CVC recommend during our regular SV/VMM trainings & consulting).
Also, with VMM 1.2, there are enhanced features built-in to navigate around the whole environment, locate all channels, and probe them at will – all without touching the existing code! It will be fun to debug such a scenario again in near future with such sophistication.