Open Source VHDL Verification Methodology/Дневник
- Словарь терминов
- Дневник
- История проекта
Содержание |
Challenges in Verification of Modern Designs
Relative Cost of Finding bugs
Typical Verification Flow
- Manual correlation of Requirements, Verification plan and Coverage results to make sure verification is comprehensive and complete.
- This is tedious, error-prone & time consuming process.
- Leads to delays, missed functionality & low quality.
Modern Design Verification
- Rising complexity of SoC designs
- Approaching 1 Billion logic gates
- 100s of IP blocks
- 1000s of requirements and features
- 100 – 1000s of bugs
- 1000s of regression runs with 1000s of tests
- Multiple distributed design/verification teams
- 70% of the effort goes into verification
- 89% of designs go over deadline by an average 44%*
Verification Tool Box
- Arsenal of Technologies
- Directed tests
- Random tests
- Constrained Random tests
- Coverage driven verification
- Code coverage
- Functional coverage
- Assertion based verification
- Emulation
- Formal verification
- But something is missing!
- Acceleration/Emulation
- Assertions with Formal Property Checking
- Assertions with Simulation (Simulation-based ABV)
- Clock-Domain Crossing (CDC) verification
- Code Coverage
- Constrained-Random Simulation
- Directed Test Simulation
- FPGA PrototypingFunctional coverage
- Intelligent Testbench Automation
- LintingSimulation-based Power Aware Functional Verification with UPF (or CPF)
- Stimulus Generated from an Software Running on an Embedded Processor
Verification Management
- Management of verification information
- Requirements
- Create or import requirements
- Plans
- Identify verification metrics that will verify requirements
- Assign development of metrics to individuals
- Bugs
- Link bugs with verification metric
- Regression runs
- Link regression results with verification metric
- Design and verification source
- Versions, configurations etc.
- Requirements
- Monitoring and Analysis
- Support the verification toolbox, methodology, process
COVERAGE METRICS
Measuring progress is one of the most important tasks in verification and is the critical element that enables the designer to decide when to end the verification effort. Several methods are commonly used:
- Toggle testing: verifies over a series of tests that all nodes toggled at least once from 1 to 0 and back
- Code coverage: demonstrates over a series of tests that all the source lines were exercised; in many cases, there is also an indication as to whether branches in conditional code were executed; sometimes an indication of state-machine transitions is also available
- Tracking how many bugs are found each week: possibly the most common metric used to measure progress; after a period of a few weeks with very few or zero bugs found, the designer assumes that the verification process has reached a point of diminishing returns
Оценка полноты функциональных тестов
по Полякову стр. >93
Эвристические метрики
Основаны на текущей статистике обнаружения ошибок в проекте
- календарное время между моментами обнаружения ошибок проекта (к концу процесса верификации оно увеличивается);
- общее количество промоделированных тактов работы проектируемого устройства;
- общее число обнаруженных ошибок в проекте и т.д.
Программные метрики
- покрытие кода
- покрытие строк
- покрытие переходов
- покрытие путей
- покрытие выражений
- покрытие переключений
Coverage and Verification Overview
Every project starts with a design specification. The specification contains elaborate details on the design construction and its intent.
A verification team uses the design specification to create a verification plan. The verification plan contains a list of all questions that need to be answered by the verification process (the golden reference). The verification plan also serves as a functional spec for the test bench.
Once the test bench is built and the designers succeed in implementing the design, you simulate the design to answer the question: “Does it work?”. If the answer is no, the verification engineer gives the design back to designers to debug the design. If yes, it is time to ask the next question: “Are we done yet?”. Answering this question involves investigating how much of the design has been exercised by looking at the coverage data and comparing it against the verification plan.
Overview of Code Coverage Types
Questa SIM code coverage provides graphical and report file feedback on the following:
- Statement coverage — counts the execution of each statement on a line individually, even if there are multiple statements in a line.
Statement coverage is the most basic form of coverage supported by Questa SIM. The metric for statement coverage is the count of how many times a given statement is executed during simulation. Multiple statements may be present on a single line of HDL source code. Each such statement is processed independently of other statements on the same line. Statement coverage counts are prominently displayed in the Source window. They are present in most of the other coverage windows as well.
Statement coverage statistics for “for” loops are presented using two separate entries relating to one line of code: the first entry is the number of times the “for” statement was entered, while the second is the number of times the loop was repeated. Consider the following statement displayed in a coverage report:
31 1 ***0*** for i in SETS-1 downto 0 loop
31 2 ***0***
The statement on line 31 displays counts in two entries: the count “1” refers to how many times the loop was entered, the count “2” refers to how many times the loop was repeated.
- Branch coverage — counts the execution of each conditional “if/then/else” and “case” statement and indicates when a true or false condition has not executed.
Branch coverage is related to branching constructs such as “if” and “case” statements. True branch and “AllFalse” branch execution are measured (see “AllFalse Branches”). In order to achieve 100% branch coverage, each branching statement in the source code must have taken its true path, and every AllFalse branch must have been taken.
Verilog and SystemVerilog 'if … else if … [else]' chains are analyzed with a coverage model as shown in the example below.
module top; integer i=10; initial begin #3 i = 18; #3 i = 2; #1 $finish(); end always @ (i) begin if (i == 16) $display("sweet"); else if (i == 2) $display("terrible"); else if (i == 10) $display("double digits at last"); else if (i == 18) $display("can vote"); else $display("just another birthday"); end endmodule
When this example is run to completion and the branch coverage is collected and saved to top.ucdb, the “vcover report top.ucdb -details” command produces the following report:
Example 20‑2. Coverage Report for Branch
Coverage Report by file with details
File: top.v
Branch Coverage:
Enabled Coverage Active Hits Misses % Covered
---------------- ------ ---- ------ ---------
Branches 5 3 2 60.0
==============================Branch Details=============================
Branch Coverage for file top.v --
------------------------------------IF Branch----------------------------
12 3 Count coming in to IF
12 1 ***0*** if (i == 16)
14 1 1 else if (i == 2)
16 1 1 else if (i == 10)
18 1 1 else if (i == 18)
20 1 ***0*** else
Branch totals: 3 hits of 5 branches = 60.0%
The 60% coverage number is derived, in that five bins under the initial 'if' have been inferred — 1 `if' branch, 3 'else if' branches, and 1 'else' branch. Three of these branches were executed.
If the final 'else' had not been present in the example, the coverage score would remain the same, but instead of listing an 'else' count, the report would list an 'All False Count' value of 0.
For “case” statements, the case expression itself is not considered a branch. Rather, each case item is considered a separate and independent branch. To achieve 100% branch coverage in a “case” statement, each case item must have been executed during simulation.
In order to gain more coverage detail on the HDL expressions used in branching statements, the Condition and Expression Coverage features may be used (see “Condition and Expression Coverage”).
- Condition coverage — analyzes the decision made in “if” and ternary statements and can be considered as an extension to branch coverage.
- Expression coverage — analyzes the expressions on the right hand side of assignment statements, and is similar to condition coverage.
- Toggle coverage — counts each time a logic node transitions from one state to another.
- FSM coverage — counts the states, transitions, and paths within a finite state machine.
For details related to each of these types of coverage, see “Code Coverage Types”.
Эволюция языков описания аппаратуры
- Язык SystemVerilog. Проектирование СБИС и систем // ЭЛЕКТРОНИКА: Наука, Технология, Бизнес 4/2006
