Coverage Cookbook/Executable Testplan Format

A testplan is a document which captures the important features of a design and how they will be verified.

Motivation for Creating a Testplan

Functional Verification has been described as a major challenge in SoC Designs with many citing inadequate visibility into the verification process as a major factor contributing to this challenge. This lack of visibility impacts design quality, schedule predictability and cost (resource/tools/infrastructure). Typical questions that verification managers need answers to are: When are we done with verification? Are all critical requirements of the system verified? Are we using all verification resources adequately to meet project schedules?

By putting an executable plan in place that captures a prioritized list of our verification intent, we are able to do the following:

• Organize ourselves better. It is not possible to completely verify all states of a design however we can identify all critical, important and nice to have features, then put a plan in place where we ensure that all critical and some important features are fully verified within schedule. If schedule permits, other features can be verified as well
• During Day to Day execution of the Verification project, an executable plan can help give you insight into current status. Progress can now be tracked instantly because you can see exactly where you are relative to what you intended to verify. You can visualize this progress against Milestones, priority of feature, or even resource
• Helps manage function coverage closure as the project is being executed. Verification Engineers can focus their coverage closure efforts on the critical features that have been identified as holes, followed by important features and then by the nice to have features.

Creating a Testplan

In many cases, the features will be verified in simulation and recorded as verified using either code coverage or functional coverage. The testplan can also include information about lab validation and firmware/hardware integration testing. For testplans which include code coverage and functional coverage, the connection between the testplan and simulations can be automated. To make the testplan executable a certain document format must be followed. The format which Questa's Verification Management solution uses is described below.

Capturing the Testplan

Flexibility is key when it comes to connecting a testplan to a simulator. In Questa Verification Management's case, several different source document formats are valid. The only requirement is that the source document must be able to be exported to an XML document. For example, Microsoft Word, Microsoft Excel, OpenOffice Writer, OpenOffice Calc and Adobe FrameMaker can all easily output an XML document which can be processed and made executable. In fact, it is perfectly acceptable to capture one section of your testplan in Word and another in Excel. These different sections can then be combined together as described below in the Re-Using Existing Testplans section.

Of the different input options, spreadsheets are the most commonly used and lend themselves very nicely to the type of data which is captured in a testplan. The spreadsheet format provides a clean and concise means of visualizing verification intent. The rest of this article will describe both required information needed in a spreadsheet and how to flexibly add additional information for usage throughout the testplan's life cycle.

Contents of the Testplan

Capturing the requirements for testing into the plan is a process. That process is defined in the Specification to Testplan article. The process involves rigorous analysis of specifications, collaboration between architecture, design and verification teams and reviews to ensure nothing is missed. All of the requirements which come out of this process need to be captured in the testplan with some required structure. The required structure allows the testplan to become executable and to become part of the verification cycle.

Required Plan Structure

Questa's Verification Management solution requires four distinct pieces of information for each requirement captured. They are Section, Title, Link and Type. Additionally, other information is understood out of the box. The additional information for each requirement includes a Description, a Weight and a Goal. While these additional fields are not required, they are used the majority of the time. Questa's Verification Management solution also has the flexibilty to allow user defined fields.

If we look at the typically used fields in a spreadsheet format, each field is represented by a column in the spreadsheet.

Plan Structure

Each row in the spreadsheet corresponds to a requirement captured during the testplan creation process. Each column has specific meaning in Questa's Verification Management solution.

Section and Title

The Section and Title columns work in conjunction to create the naming and hierarchy within the testplan.

Section

The Section column, usually a number, is used to create hierarchy within the testplan and group related testplan items together in a parent / child relationship. In spreadsheets, the user is responsible for entering this information. Typically you will start numbering sections with the number '1' and continuing sequentially. A sub-section beneath that section would then be numbered "1.1" and so on, where each additional level of hierarchy would be represented by the addition of a "." between section numbers.

Title

The title column captures the name of the requirement or design feature to be verified. This is the name of the testplan section that will appear within Questa. The name chosen here should have meaning as it will be visible through the tool flow.

When the testplan is extracted by Questa, it uses the information in the Section and Title columns to create a hierarchical name and unique tag for each row of the testplan. For example, given the simple testplan example above, Section 1.1 would have a hierarchical name /testplan/Parent_1/Child_1.

Description

The description column allows for more detail to be added to the spreadsheet. This could include references to other documentation to allow engineers to gather more information or it could be a simple explanation as to why the requirement exists. Any text can be captured in this column. It is technically optional, but in practice a requirement captured in a testplan should have an entry in the description column.

Link and Type

The Link and Type columns are used to specify the code or functional coverage items that will be linked to the requirement. A requirement can be linked to multiple different coverage metrics, including metrics of different types. Questa supports linking to Covergroups, Coverpoints, Crosses, Assertions, Cover Directives, Directed tests and code coverage metric types. These columns also allow other testplans to be imported as described in the Re-Using Existing Testplans section.

Link

This column is where you specify the name of the actual coverage object(s) in the coverage database that is linked to this respective testplan item. This could include a specific covergroup instance, an assertion, etc.

Type

This column is where you specify the type of the coverage object you specified in the Link column.

Together, the Link and Type column information is used to find the corresponding coverage objects in the coverage database efficiently and create the links between the testplan and the specified coverage objects. These columns enable the testplan to become executable.

Weight

The weight column captures an integer number that reflects the relative importance of the current testplan item amongst its siblings, to its parent testplan section. The default is 1 if not specified. When coverage for the testplan is being calculated by Questa, which uses a "weighted-average" calculation algorithm, these weights are taken into account. For more information about how Questa calculates tesplan coverage please see Questa documentation on Verification Management.

Additionally, the weight column can be used to exclude portions of a testplan by specifying a value of 0 for the testplan section / item row that need to be excluded.

Goal

This column specifies the verification objective for a particular testplan section. Legal values range from 1 to 100, with the default being 100 if not specified. Questa uses this information to determine the point at which a testplan section / item is deemed to be covered. It does not alter how coverage is calculated.

Other Columns Understood By Questa

There are other special purpose columns which can be used in a coverage testplan. These columns are all optional however when they are present Questa leverages their content.

Path

When linking to a specific instance of a covergroup, two options exist. Optionally, the full hierarchical path to the covergroup could be added into the Link column. This works very well if only one coverge item is being accessed at that level of design hierarchy. If however, multiple coverage items are being linked to in the same level of design hierarchy, the Path column allows for the specification of the design path which will be prepended to the entry in the link column to create a fully qualified reference.

Unimplemented

As testplans are being defined, it is common for requirements to be captured where corresponding coverage items don't yet exist in a testbench or design. To handle this situation, a requirement can be maked as unimplemented by either adding a value of 'yes' or a number greater than zero to the Unimplmented column. This will cause testplan coverage calculations to accurately reflect that a requirement exists which is not yet covered by showing zero coverage for that requirement. By default, it is assumed that coverage for a requirement is implemented unless this column is specified.

User Defined Columns

To ensure users have the flexibility to record other vital information related to a requirement, Questa's Verification Management solution allows user defined columns to be created as well. This allows for the specification of anything that needs to be tracked. It can also give you better visibility into the verification process. Some columns which are routinely added include which engineer is responsible for testing a requirement and what the priority of the requirement or design feature is.

These added columns help to answer questions such as:

• How am I doing on my high priority testplan items?
• Where am I with regards to meeting my current project milestones?
• How are my doing in terms of my resources?

The spreadsheet used in the System Level Functional Coverage Example adds Owner and Priority user defined columns.

Re-Using Existing Testplans

Often it is useful to be able to make use of existing testplan's in other contexts. A few common situations where this becomes useful are:

• Testplans were developed at the block level and need to be incorporated into the chip or system level testplan.
• A testplan is delivered with a piece of IP or verification IP. For example, protocol specific Verification IP should contain a complete protocol testplan to ensure a protocol is completely verified.
• Incorporate auto generated testplans into our sub-system or SoC level plan. For example, for power aware simulations simulators should be able to create a testplan to ensure the power scenarios are completely
• Combine testplans captured by different editing tools into one master testplan.

In these situations, existing testplans can be hierarchically imported into the top-level testplan being created, allowing for visualization of the full depth of the verification effort.

When creating such a testplan, the Type column will be given a value of "XML" to inform Questa that this section does not reference a coverage object, but rather it refers to another testplan document. The Link column will be used to specify the actual testplan file that contains the testplan being imported (via either a relative or full-path to the file).