Open Source VHDL Verification Methodology/Better Coverage in VHDL

Ссылки сюда (26) →

Подстраницы (1) →

Проект OS-VVM

• Руководство пользователя (eng)
• Описание пакета CoveragePkg (eng)
• Описание пакета RandomPkg (eng)
  • Примеры распределений

---

Исходные коды

• CoveragePkg.vhd
• RandomPkg.vhd
  • RandomBasePkg.vhd
  • SortListPkg_int.vhd
• Примеры

---

Описание примеров

• Пример использования OS-VVM
  • Пример 2 (Интеллектуальный ТБ)

---

Презентации

• Верификация в VHDL
• Презентация (eng)
• Better Coverage in VHDL (eng)

---

Coverage

• Code Coverage (en)
• Functional Coverage (en)

---

• Словарь терминов
• Дневник
• История проекта

* VHDL * PSL *

Introduction (Why testbenches are not enough)

Testbench vs. Coverage

• When we deal with small designs or small blocks of larger designs, we can create testbench that will:
  • Produce correct simulation results,
  • Exercise all behaviors of the design.
• In case of modern large designs with complex functionality, even most knowledgeable designer may miss some design behaviors while creating testbench.
• We need confirmation that:
  • Testbench tested all design behaviors,
  • Design behaviors are the intended behaviors.

We need Coverage!

What Is Coverage?

• Coverage is an extremely ambiguous term when used on its own.

We can say that (in electronic design):

Coverage is a kind of design metrics that tells us if certain aspects of the design were properly exercised during testing procedure.

• We usually qualify coverage term by saying what is the aspect that was tested:
  • Code quality testing yields code coverage,
  • Property checking yields property coverage,
  • Collection of design variable values yields functional coverage.

Structure vs. Functionality

• When we simulate HDL code of our design with code coverage enabled, we can monitor if all elements of code structure were exercised. If they were, we assume that design quality is high.
• Fully tested HDL code may still not meet design specification: lack of errors means that actual, not intended functionality of the code was tested...
• Functionality testing (represented by property coverage and functional coverage) gives us better information than structure testing.

Are structure and functionality tests mutually exclusive or do they overlap?

Code Quality vs. Functionality

Our Challenge

• We agree that we have to have good design code. Bad code can sometimes surprise us with good functionality, but it is not manageable.

We need code coverage!

• We know that good code can represent bad functionality.

We can simulate a lot to verify that it really represents desireddesign behaviors.

Property and functional coverage can do it faster and better!

• Let’s see what are the advantages and disadvantages of all three kinds of coverage...

Code Coverage Flavors (When and How to Use Them)

Code Coverage Origins

• Code Coverage was invented to improve quality of programs written in “regular” programming languages (like C).

• By nature, code coverage analyses structural correctness of the code, not functional correctness! Structurally incorrect code usually malfunctions, so correcting bad code usually corrects functional errors.

• Code coverage developed into wide variety of flavors to address obvious and well-hidden problems that can sneak into the code.

Basic Code Coverage Flavors

• The most basic flavors of code coverage are:
  • Statement Coverage – checking if all statements in the code were executed at least once,
  • Branch Coverage – checking if all branches of conditional statements (if, case, etc.) were executed.

• Statement Coverage is closely related to Line Coverage, results of both differ only if the coder squeezes more than one statement in one line (not a good practice).

• The results of Branch Coverage could be extracted from the statement coverage results, but in large designs it would be very tedious task...

Additional Code Coverage Flavors

• Statement and branch coverage can yield false positive results when they deal with non-trivial expressions:

  • Trig := SensA or SensB;

There are 4 different combinations of values of SensA and SensB, but statement coverage will be OK after just one execution of the assignment.

We need Expression Coverage here.

• if Load='1' and Enable='1' then ...

Branch coverage will just tell that the branch was not executed without specifying if Load, Enable or both are to blame.

We need Condition Coverage here.

Special Coverage Flavor

if pol='1' then 1 2 3 4 a := buf1; 1 3 else a := not buf1; 2 4 end if; if act='1' then 1 2 3 4 res := a or b; 1 4 else res := a and b; 2 3 end if;

If we run simulation of the code presented to the left, then two executions are needed to achieve statement and branch coverage (they are marked with 1 and 2). But there are two additional execution paths that were not exercised yet – marked with 3 and 4. We need two more executions and Path Coverage enabled to fully test this code...

HDL Aspects of Code Coverage

• Original code coverage was created for sequential code and ignores concurrency and modularity of VHDL.

• Modularity issue can be addressed by per instance data collection in addition to per unit data collection:
  • If you have D-flipflop component instantiated 4 times and per unit data collection is enabled, execution of one instance signals complete coverage.
  • If you enable per instance data collection in the same design, all 4 instances must be executed to signal complete coverage.

• There are no ways of verifying (with code coverage) concurrent execution of two processes residing in one unit...

Code Coverage in the Tools

• Code coverage is not a language feature – it is a tool feature...

• As such, it usually requires more advanced version of the simulator or additional license features.

• Always check which flavors of code coverage are supported in the tool.

• All tools should generate textual coverage reports;

look for additional coverage viewers that make analysis of code coverage results easier.

• Do not enable all coverage flavors at the same time;

it will slow down your simulation and usually will not yield significantly better results...

Summary

• Code coverage checks structural integrity of the code directly and only indirectly suggests functional correctness.

• Code coverage lets you improve quality of your verification procedures.

• 100% code coverage never gives guarantee that design functionality is correct.

• Less than 100% code coverage gives strong indication that there are also functional problems in the design.

• Code coverage will not check concurrency and timing!

Property Coverage (Close Relative of Assertions)

Basic Terms

• Boolean logic tests general statements about the design.

• Temporal logic adds time dimension to Boolean logic and tests absence, presence, persistence, sequence, etc. of certain events.

• Property is the formalized description of design behavior taken from the design specification and expressed in temporal logic terms.

• Properties can be expressed in PSL (Property Specification Language – now native to VHDL) or in SVA (SystemVerilog Assertions subset – works with VHDL via checker modules).

• Sequences are the simplest form of properties, representing sequences of events/conditions.

Clocking Properties

• Both PSL and SVA use discrete time version of temporal logic, i.e. they sample values of design signals at certain clocking events.

• PSL lets you set default clock using:

default clock is rising_edge(CLK1);

or clock for a given property (or sequence) using:

my_prop_or_seq @falling_edge(CLK2);

• You can also use rose() and fell() functions to detect signal edges for the purpose of “internal clocking”.

Simple Sequence Examples

Concatenation: A is true in the current clock cycle and B is true in the next cycle:

{A ; B}

Concatenation with Consecutive Repetition: A is true for 2 clock cycles and B is true for 2 clock cycles (starting at the next sample after A sequence ended):

{A[*2] ; B[*2]}

Consecutive Repetition With Range: A is true for 2 clock cycles and B is true for 2 to 3 clock cycles (starting at the next sample after A sequence ended):

{A[*2] ; B[*2 to 3]}

Fusion with Repetition: A is true for 2 clock cycles and B is true for 2 to 3 clock cycles (starting at the same sample where A sequence ended):

{A[*2] : B[*2 to 3]}

More Sequence Examples

Inclosure: A is true for 2 clock cycles within 5 clock period time when B is true :

{A[*2] within B[*5]}

Non-consecutive Repetition: A is true for 2 clock cycles (not necessarily consecutive) within 5 clock period time when B is true :

{A[=2] within B[*5]}

Go-to Repetition: A is true for 2 clock cycles (not-consecutive, but including last sample in sequence) within 5 clock period time when B is true :

{A[->2] within B[*5]}

Property Examples

Operator always: Signal POWER must always be high:

always (POWER='1')

Operator never: Signal Z should never be true right after Y:

never {Y ; Z}

Operator eventually!: Signal DONE should be true before end of simulation:

eventually! DONE

Non-overlapping implication: If X is true for 2 cycles, then Y is true for 3 cycles starting at the last sample of X:

X[*2] |-> Y[*3]

Overlapping implication: If X is true for 2 cycles, then Y is true for next 3 cycles after the last sample of X:

X[*2] |=> Y[*3]

Property Coverage vs. Assertions

• Properties checked with assert directive alarm us only when something went wrong (not good for coverage).

• We must use the property with cover directive to get confirmation that the behavior it represents was tested.

• Properties used in cover directive enable property coverage;

we get messages when property evaluation succeeded and report listing all properties that were not covered.

• Properties with implication (|=>, |->) are very convenient in assertions, but not so much in covers.

(Vacuous pass — situation when implication antecedent is false — avoids false alarms in assertions, but triggers false positives in covers)

Design Example

• Quote from the design specification:

Signal DVALID should be activated 3 to 5 clock cycles after activation of ACK and 2 to 4 clock cycles before deactivation of ACK.

DVALID should remain active at least 2 clock cycles after deactivation of ACK.

• First phase of the property:

{ rose(ACK) : (ACK='1')[*3 to 5] }

• Second phase:

{ rose(DVALID) : ((ACK and DVALID)='1')[*2 to 4] }

or even

{ rose(DVALID) : (DVALID='1')[*2 to 4] }

• Third, final phase:

{ fell(ACK) : (DVALID='1')[*2 to inf] }

Design Example – cont.

• We can inject PSL code into special comments in VHDL;

this way it will not interfere with synthesis and similar applications.

• We can define 3 sequences to represent 3 phases of our property.
• Final property definition concatenates those 3 sequences.
• Labeled cover directive tells simulator how to check property.

--@clk  rising_edge(CLK);
--@psl  sequence ack35_s is { rose(ACK) : (ACK='1')[*3 to 5] } ;
--@psl  sequence dvalid24_s is { rose(DVALID) : (DVALID='1')[*2 to 4] } ;
--@psl  sequence dvalid2inf_s is { fell(ACK) : (DVALID='1')[*2 to inf] } ;
--@psl  property ackdvalid_p is { ack35_s ; dvalid24_s ; dvalid2inf_s } ;
--@psl  ackdvalid_c: cover(ackdvalid_p) report "Sequence ACK/DVALID covered!";

Property Coverage Hints

• Additional simulation resources are needed to check properties, so do not exaggerate with adding them.

• Do not add properties describing obvious behaviors, e.g. counter counting. Malfunctions in such areas will be easy to notice in simulation.

• Add properties checking critical behaviors, especially if they do not demonstrate immediately as the top level signal changes.

• Add properties testing interaction of data coming from different blocks of the design (especially when dealing

with IP blocks).

Summary

• Property coverage checks functional correctness of the design by comparing its behavior against design properties.

• Property coverage requires some coding in PSL (or SVA) and proper simulator license.

• 100% property coverage says that all properties you have specified were exercised, so choose them wisely.

• Less than 100% property coverage means that your design behavior does not meet design specification requirements!

• Properties work in formal verification, too.

Smart Functional Coverage

How OS-VVM Does It Better in VHDL

Why Functional Coverage

• Sometimes coverage goal is hard to express in terms of behavior;

it may be easier to specify using values of design variables.

• Let’s consider the following example of climate controller reading data from 8-bit humidity sensor:
  • 0 to 100 readouts can trigger 5 different actions,
  • Readouts above 100 are illegal.
• If we define all possible values of sensor variable as below, how can we verify that all ranges were tested, but not over-tested?

Value Range	0 to 9	10 to 29	30 to 70	71 to 90	91 to 100	101 to 255
Status	Too_Low	Low	Normal	High	Too_High	Illegal

We need Functional Coverage!

Coverage Bins

Value Range	0 to 9	10 to 29	30 to 70	71 to 90	91 to 100	101 to 255
Status	Too_Low	Low	Normal	High	Too_High	Illegal

• Functional coverage counts values of design variable according to previously defined bins – groups of values that have special meaning in the design.

• Bins using just one variable create data structure we can call coverage item or coverage point, e.g. 6 bins for our humidity sensor variable.

• We can create cross coverage bins combining values of multiple variables to get cross coverage item.

• If we have action variable in our climate control design with values Idle, Humidify, Dehumidify, Alarm we can create bins like this: (Low, Humidify), (Normal, Idle), (High, Dehumidify), (Illegal, Alarm), etc.

• Bins can count hits, trigger errors or be ignored during simulation.

FC and Randomization

• Functional coverage can work alone, but it really shines when combined with constrained random stimulus.

• In our humidity sensor example, we can randomly generate readouts and stop simulation when each bin was hit at least once; this way we complete tests much faster than in the case of scanning through all possible sensor readouts.

• Some languages have elaborate constraint system for random variables that requires dedicated constraint solver and works independently from functional coverage.

• What if we could have live feedback coming from the FC engine that controls randomization?

Smart Functional Coverage

• Open Source VHDL Verification Methodology (OS-VVM*) introduced to VHDL the concept of smart functional coverage.

• OS-VVM contains both randomization and functional coverage packages that can work independently, but also create coverage → randomization feedback that enables smarter, more efficient verification.

• Smart coverage controls randomization process to ensure that values belonging to already covered bins are no longer randomly generated.

• * To learn more about OS-VVM, please watch recording of our previous webinar or visit osvvm.org.

Smart Coverage Benefits

• When random number generator produces uniformly distributed values from the range 1..n, it takes approximately n log(n) trials to cover all values in the range. This number is even higher for non-uniform distributions.

• Smart coverage reduces the number of trials to precisely n.

• In plain language, smart coverage gives you guarantee that each value in the randomized range will show up only once.

• You can fine tune smart coverage by using different randomization weights for different coverage bins.

Smart Coverage Example

• Let’s say that we 100x100 display matrix that we want to test randomly so that each pixel is tested at least once.

variable XYCov : CovPType;       -- coverage item object declared using
                                 -- CovPType from the OS-VVM package
variable X,Y : integer;          -- index data buffers
  . . .
XYCov.AddCross(GenBin(0,99), GenBin(0,99));  -- 100x100 bin matrix
XYCov.InitSeed(XYCov'instance_name);         -- seed for smart coverage
  . . .
while not XYCov.IsCovered loop   -- loop runs until indices covered
  wait until rising_edge(CLK);   -- sampling event detection
  (X, Y) := XYCov.RandCovPoint;  -- randomize uncovered indices
  XYCov.ICover((X, Y));          -- collect cross data
end loop;

while loop will have 10000 iterations – several thousands less than in case of ‘dumb’ randomization...

Summary

• Functional coverage checks correctness of the design by collecting values (or sets of values) of design variables during simulation.

• Bin definition requires sound knowledge of the design structure and functionality.

• Functional coverage requires some coding in VHDL — minimal if you use OS-VVM.

• 100% property coverage says that all bins you have specified were hit during simulation.

• Smart coverage speeds up simulation by controlling randomization dynamically.

Conclusion

Summary, Contact Data, Q&A

Which Coverage?

Coverage Kind	Coding Required?	Tool Setup Required?	Special License?	Checks Code Structure	Checks Design Behavior
Code Coverage	No	Yes	Yes
Property Coverage	Yes	No	Yes
Functional Coverage	Yes	No	No

• As we see in the comparison table, there is no single kind of coverage that has advantages only.

• Code coverage checks code structure, but may be expensive.

• Property coverage is excellent for checking behavior, but may be expensive, too.

• Functional coverage with OS-VVM has only one drawback – it requires some coding...

• Balanced use of those three coverage kinds should dramatically increase design quality and reduce verification time.