Archive for April, 2009

Badri Gopalan, Principal Engineer, Synopsys

There are several situations where RTL-level performance measurements are needed to validate micro-architectural and architectural assumptions, as well as to tune the RTL for optimal performance.

A few examples of these low-level measurement requirements are:

· Throughput, latency and effects of configuration parameters of a memory or DMA controller under different traffic scenarios

· The statistical distribution from a prediction scheme for various workloads

· Latency of a complex multi-level arbiter under different conditions

· End-to-end latency, throughput, QoS of a network switch for various types of data, control traffic

The VMM Performance Analyzer application provides a flexible and powerful framework to collect, analyze and visualize such performance statistics from their design and verification environments. It consists of a few base classes which allows you to define the performance metrics to be tracked, and collect run-time information for these performance metrics into tables in an SQL database. The data can be collected over multiple simulation runs. You then interact with the database with your favorite database analysis tool or spreadsheet. The SQL language itself offers simple yet powerful data query capabilities, which can be run either interactively or scripted for batch-mode. Alternatively, you can load the data into a spreadsheet and perform your analysis and visualization there.

At a conceptual level, you first identify the different atomic performance samples to be collected for analysis. These are referred to as “tenures”. For example, a memory transaction on a bus (from a specific master to a slave) is a tenure. The VMM-PA does the work of assigning an ID, collecting and tracking attributes such as the start time, end time, initiator and target IDs, and other associated information (suspended states, aborts, completions etc.,) as rows in a table. Each table corresponds to an instance of vmm_perf_analyzer object. You can (and probably will) have multiple tables (and thus multiple instances of the vmm_perf_analyzer object) in your simulation, dumping performance data into the database.

Here is a code snippet which illustrates the process of creating a vmm_perf_tenure tenure (a row in a table), a vmm_perf_analyzer table, and a vmm_sql_db_ascii (or vmm_sql_db_sqlite) database (a collection of tables), with some explanations following the code:

1. class my_env extends vmm_env;

2.    vmm_sql_db_sqlite db;                     //the database itself

3. vmm_perf_analyzer mem_perf_an; //One table in the database

4.    virtual function void build;

5.       super.build();

6. this.db = new(“perf_data.db”); //”perf_data.db” created on disk

7. this.mem_perf_an = new(“Mem_performance”, this.db);

8.    endfunction: build

9. endclass: my_env

10.

11. //Now, start a thread which will dump performance data to

12. // Mem_performance table in the database. Any event can be used to

13. // start or terminate tenures: it is left to the user control

14. initial begin

15. vmm_perf_tenure mem_perf_tenure = new();

16.    forever begin: mem_perf_thread

17.       this.mem_mon.notify.wait_for(mem_monitor::STARTED);

18. this.mem_perf_an.start_tenure(mem_perf_tenure);

19.       this.mem_mon.notify.wait_for(mem_monitor::ENDED);

20. this.mem_perf_an.end_tenure(mem_perf_tenure);

21.    end: mem_perf_thread

22. end

23. virtual task my_env::report;            //report is part of the environment class.. only the PA relevant code is presented

24. this.mem_perf_an.save_db(); //write any buffered data to disk

25. this.mem_perf_an.report();    //simple pre-defined performance report

26. endtask: report

· In lines 2 and 7, the SQLite database is created using vmm_sql_db_sqlite base class. You could create a different flavor of database, for instance, a plain text database, in which case, a list of SQL commands is created, which could then be replayed on your SQL data engine of choice. (See the Reference guide for more details). Typically you have one SQL database per test, however, you certainly can open multiple databases in the same test.

· In lines 3 and 8, a table in the database is created using vmm_perf_analyzer base class, which will help track statistics related to resource, in this case, a memory interface. Typically you will have multiple tables in a test, which correspond to tracking of statistics of multiple resources in the DUT or environment. This would correspond to multiple instances of the vmm_perf_analyzer base class.

· In lines 15, 18 and 20, one transaction item (“tenure”) is created and stored in the table. The transaction item is created by the vmm_perf_tenure base class. The tenure management methods such as start_tenure(), end_tenure(), suspend_tenure(), resume_tenure(), abort_tenure() allow you to express the state of the monitored tenure and reflect those in the performance tables. You can of course control when to execute these methods from your test, either timing control statements, events, callbacks, or whatever. Callbacks registered to the various vmm_xactor classes in your environment are the most scalable way to hook these into your environment, but it is your choice.

· In lines 24 and 25, data is flushed into the database at the end of simulation (in the vmm_env::report phase, to be more precise), and a basic / sample report is generated. It is important to note that you will in all likelihood be generating custom reports from the SQL database itself. That is explored further below.

Now that you have a code snippet showing you the process of monitoring statistics on shared resources in the design, you want to be able to write your custom queries, reports, and charts off the database. One could do this in a few ways:

1. Connect a spreadsheet to the database, and use the spreadsheet capabilities to generate statistics, charts etc. There was an earlier blog post on how to accomplish this: refer to “Analyzing results of the Performance Analyzer with Excel” (http://www.vmmcentral.org/vmartialarts/?p=23)

2. Use a SQL engine such as SQLite, MySQL, PostGreSQL, or any others to read in the SQL commands and generate custom query scripts which can then be used in batch mode. SQLite (http://www.sqlite.org), for example, has various plugins, such as Perl, TCL, C/C++ etc., so you can write scripts or queries in your favorite languages. There are several publicly available and commercial front ends you can use to read in the SQL data and perform your analyses (I’ve used SqliteSPY http://www.yunqa.de/delphi/doku.php/products/sqlitespy/index in the past). There are several quick-start tutorials for the SQL syntax available on the internet which should get you up and running with SQL in short order. To generate plots, one could use applications such as gnuplot, R, octave etc. It is probably more convenient to use spreadsheets to create graphs of various kinds.

In the next blog item related to the VMM Performance Analyzer, I will discuss some other aspects of the Performance Analyzer application (all of which is available by reading the User Guide: http://vmmcentral.com/resources.html#docs). I will also provide some examples of SQL code which demonstrate the analyses you can perform.

Fabian Delguste, Snr CAE Manager, Synopsys

VMM 1.1 comes with a new base class called vmm_test that can be used for all tests. The main motivation for developing this base class was to enable a single compile-elaboration-simulate step for all tests rather than multiple ones. It is recommended to implement tests in a program thread as this provides a good way of encapsulating a testbench, and reduces races between design and testbench code. All test examples showed the tests implemented directly in program blocks. The drawback of this technique is that the user needs to recompile, elaborate and simulate each test individually. When dealing with large regressions consisting of 1000s of tests, multiple elaborations can waste a significant amount of time whereas tests using vmm_test only requires one elaboration. A given test can be selected at run-time using a command-line option. Switches like +ntb_random_seed can be used in conjunction with these tests.

To understand better how this base class works, let’s look at the example that ships with the VMM 1.1 release in the sv/examples/std_lib/vmm_test directory. This example shows how to constrain some ALU transactions. These transactions, modelled by the alu_data class , are randomly generated by an instance of a vmm_atomic_gen and passed to an ALU driver using a vmm_channel.

Before digging in the gory details of vmm_test, let’s see how tests are traditionally written:

1. class add_test_data extends alu_data;

2.   constraint cst_test {

3.      kind == ADD;

4.    }

5. endclass

6.

7. program alu_test();

8.    alu_env env;

9.    initial begin

10.     add_test_data tdata = new;

11.     env.build();

12.     env.gen.randomized_obj = tdata;

13.     env.run();

14.   end

15.endprogram

· In lines 1-4, the alu_data class is extended to the new class add_test_data which contains test-specific constraints. In this test, only ADD operations are carried modelled by this class.

· In line 7, a program block is used to instantiate the environment alu_env based on vmm_env.

· In line 10-13, the environment is built and the new transaction add_test_data is used as the factory instance for our vmm_atomic_gen transactor.

Of course, this test is very specific and a user clearly needs to create a similar program block with other constraints to fulfill the corresponding test plan. For instance, this test can be duplicated many times to send {MUL, SUB, DIV} ALU operations to the ALU driver. In this case, multiple program blocks are required, so are multiple elaborations and binaries to simulate these tests.

VMM 1.1 provides a way to include all test files in a single compilation. The previous test can now be written like this:

1. class add_test_data extends alu_data;

2.   constraint cst_test {

3.     kind == ADD;

4.   }

5. endclass

6.

7.  `vmm_test_begin(test_add, alu_env, “Addition”)

8.  env.build();

9.   begin

10.   add_test_data tdata = new;

11.   env.gen.randomized_obj = tdata;

12. end

13. env.run();

14. `vmm_test_end(test_add)

· In line 7, the vmm_test short hand macro `vmm_test_begin is used to declare the test name (test_add in our example), the name of the vmm_env where all transactors reside (alu_env in our example) and a label that is used to tag this particular test.

· In lines 8-13, users can build the environment, insert the factory instance and kick off the test.

· In line 14, the vmm_test short hand macro `vmm_test_end is used to terminate this test declaration.

Of course, other tests with variations in constraints can be written in the same way. Since the environment is exposed after the `vmm_test_begin short hand macro, it is possible to register callbacks, replace generators or do any other operation which is traditionally done in the VMM program block.

An important aspect of these tests is that whenever they are included, they become statically declared and visible to the environment.

Now let’s see how to include these tests in VMM program block:

1. `include “test_add.sv”

2. `include “test_sub.sv”

3. `include “test_mul.sv”

4. `include “test_ls.sv”

5. `include “test_rs.sv”

6.

7. program alu_test();

8.    alu_env env;

9.    initial begin

10.    vmm_test_registry registry = new;

11.    env = new(alu_drv_port, alu_mon_port);

12.    registry.run(env);

13.  end

14. endprogram

· In line 1-5, all tests are simply included

· In line 10, registry, which is an instance of vmm_test_registry,is constructed. This object contains all tests implemented using vmm_test_begin that have been previously included

· In line 12, registry can be run and a handle to the environment is passed as an argument to this class. This is how all vmm_tests can have access to the environment

Running a specific test is achieved by providing the test name on the command line, for example:

simv +vmm_test=test_add

simv +vmm_test=test_sub

Note that calling simv without the +vmm_test switch returns a FATAL error and lists all registered tests. This is a good way to document the available tests.

Janick Bergeron, Synopsys Fellow

The VMM Performance Analyzer stores its performance-related data in a SQL database. SQL was chosen because it is an IEEEANSI/ISO standard with a multiplicity of implementation, from powerful enterprise systems like Oracle, to open source versions like MySQL to simple file-based like SQLite. SQL offers the power of a rich query and analysis language to generate the reports that are relevant to your application.

But not everyone knows SQL. You need an SQL-aware application to do fancier stuff. And guess what! Excel can import SQL data! And everyone knows Excel!

In this post, I will show how you can get VMM Performance Analyzer data from a SQLite database into an Excel spreadsheet. A similar mechanism can be used if you are using MySQL or any other SQL implementation that offers an ODBC (Open Database Conduit).

First, download and install the SQLite OBDC from http://www.ch-werner.de/sqliteodbc/sqliteodbc.exe on your PC and install.

Next, create a new OBDC Data source by opening the OBDC Data Source Administrator by navigating to Start -> Settings -> Control Panel -> Administrative Tools -> Data Sources (OBDC).

Click the “Add…” button to add a new data source. Scroll down to select the “SQLite3 OBDC Driver” entry.

C lick the “Finish” button. This will then open the OBDC data source configuration window. Specify a name for the data source and browse to the SQLite database file. A SQLite ODBC data source connects to a single database file. It is therefore a good idea to name this new data source according to the performance data it contains. Leave all other options to their default value.

Click “OK” to complete the creation of the new data source on your computer.

Repeat these steps for each SQLite database that will need to be analyzed using Microsoft Excel on your computer. Click “OK” when all data connections have been defined.

Start Excel with a blank workbook, then Select Data -> From Other Sources -> From Microsoft Query.

This will bring up a dialog box to select a data source. Select the data source corresponding to the desired SQLite database.

Click “OK”. The following error pops up. Don’t worry! The next step is to correct that error.

Click “OK” to close the error pop-up. This will bring up the Microsoft Query Wizard.

Click on “Options”. Modify any of the check boxes but make sure the “Tables” option is (or remains) checked. You must make a modification to at least one check box.

Click OK.  The Microsoft Query Wizard should now be populated with the list of tables found in the SQLite database file. From this point on, which table and which columns you select depend on the analysis you wish to perform.

For example, to perform an arbiter fairness analysis, you would select the “InitiatorID” and “active” columns of the “Arb” table.

Click “Next >” three times then return to Microsoft Excel by clicking “Finish”. The Import Data dialog box will appear.

Click “OK” to insert the data as a table and voila! You now have a table populated from the data dynamically extracted from the SQLite database.

From there, a PivotChart can be used to display the average and maximum arbiter response time for each initiator. As you can see, the arbiter appears to be fair, given the small number of samples collected.

I hope you’ll find this step-by-step guide useful.

What cool thing have you done with the Performance Analyzer?