This particular example involves development of an APB master BFM. APB isn’t the most complicated of bus protocols but it’s a very good subject for an example like this because the code and tests are easy to understand (there’s an APB example in the latest UVM release also, presumably for the same reason).

UPDATE: for people looking at UVM Express that was announced by Mentor Graphics on Feb 22, this is my interpretation of how TDD can be applied at the lowest layer development of the BFM. Examples showing the addition of functional coverage and completion of an agent that includes sequence generation are still to come. You can see UVM Express on the Mentor website by going here.

In terms of requirements and documentation for our BFM, I’ll describe what’s going on with a couple of user stories:

There’s 2 stories that aptly describe this APB master verification IP. The ability to write/read memory of a peripheral and that’s pretty much it.

Here’s the acceptance criteria for both stories:

Because there’s 101 ways to build an APB BFM, the acceptance criteria here is important for narrowing things down. We’re looking for something equally useful for designers and verifiers so we’re not going to use anything fancy like UVM and we’re not going to use a transaction interface or anything constrained-random-like (which always seems to be the impulse of verification engineers). We’re just going to have IP with a simple method user interface on 1 side and pins on the other that could be connected to a peripheral. Arguably, a systemverilog interface is best for this so that’s the direction I went.

I was missing some necessary features for SVUnit so before I got started, I had to add support for generating unit test templates for systemverilog interfaces. I based the template off what I had already done for the module unit test template so it didn’t take long to get that done. With 1 test to verify the template output and 2 others to verify it compiled and simulated in both Questa and VCS, I could move on to the actual example.

The first tests I added to the unit test template were synchronous and asynchronous reset tests. No, they weren’t present on a story card but I thought this would be a good place to start anyway because they’re useful and would require all the pins of the interface but next to no functionality. Here’s one of those tests (fyi… `MST_EXPECT is a macro that checks each of the output ports on the interface. I expect all to be 0 after the reset):

The first time I ran this test, it failed – as I expected – in compilation because none of the pins nor the reset method existed in the interface yet. I added the pins and an empty method and ran it again. That ended in a runtime failure – again as I expected – because the reset method was empty and my FAIL_IF assertions were firing (here, FAIL_IF is hidden in the MST_EXPECT but you’ll see it in the next test). Add the reset functionality in the interface method, run the test a 3rd time and voila… my first passing test and my first complete loop through the TDD cycle:

Add a test, confirm the test fails, add just enough design code, confirm the test passes.

The next test I wrote had to do with testing the functionality of the select pin for a write, which takes me into implementation of our first user story. Here’s that test:

You see here that I’m calling the write method on the interface and in parallel I’m checking the functionality of the select pin. Running this the first time ends of course with a compilation error because the write method doesn’t exist yet. I add an empty write method, run it again and see the failures from my FAIL_IF assertions again… which is exactly where I want to be. Now I add the code required to control the select. That means adding a state machine that takes the interface from IDLE to SETUP to ENABLE and back to IDLE while the select pin is active in SETUP and ENABLE but inactive in IDLE. Here’s the code I added to do that:

You’ll notice that my definition of just enough includes the write method, the FSM, the logic to control the select for a single write and some synchronization. I have nothing more than that because that’s all I need to pass my test. Run SVUnit again and now I have 2 passing tests: I know I can reset my interface and I know the select works properly for a single write.

One last example to drive it home… the next test I wrote was still focused on the select pin but was verifying it would remain active for 4 cycles in the case of back-to-back write transactions. Here’s the test:

That failed because the BFM couldn’t yet handle the transition from ENABLE back to SELECT. Here’s the update to the BFM that I had to make to pass the test:

Several more baby steps through the TDD cycle eventually led to a completed ABP BFM. Instead of boring you with the rest of the code, I’ll leave you with some statistics of what I ended up with after I was finished.

• My APB BFM ended up being ~200 lines long
• It includes methods for writing and reading that designers and verifiers would both find easy to use
• I ended up with 26 tests in a unit test template that was ~550 lines long
• I had 2 tests for each port on the interface to verify it behaved correctly for single read/write transactions and back-to-back read/write transactions
• SVUnit tells me in about 4 seconds (of which about 3.5 sec is spent in compilation) whether or not my APB BFM is working properly
• Including the addition of interface support to SVUnit, this whole thing took me about 5 hours… from 8pm to 1:30am with a half hour break for tea at midnight

I’ll summarize by saying that as a beginner, I still find it a little tricky to get into TDD mode where I think in terms of tests first. Once I’m there though, I find it’s a very practical way to write code. And the best part is that you know the code is working as you write it as opposed to debugging it after the fact.

If you’re interested in seeing and running the results, this APB BFM example is currently available to SVUnit early adopters in the examples directory. If you’re not an SVUnit early adopter but you’re interested in becoming one, you can get a hold of me at neil.johnson@agilesoc.com.

-neil

Share/Bookmark

In the 3rd installment of the SVUnit Demo Series, I take people through a simple – yet complete – example of what’s required to test a UVM component within the SVUnit framework (if you haven’t seen the video yet, I’d recommend watching it here before reading on). The example I put together, and more importantly the plumbing under the hood required to make it work took me quite a while to put together for reasons that I don’t really get into in the video… but I will talk a bit about them here.

The usage model for SVUnit involves sequentially running a series of classes or modules through a corresponding list of unit test methods. In UVM however, due to the tight coupling between the phase methods in all uvm components and the instance of the uvm_root that ultimately drives the invocation of each phase method, uvm components are run in parallel.

One usage model wanting to be sequential and the other coded for parallelism gives us two models that are fundamentally at odds… so for unit testing UVM components, I had to get creative .

I wouldn’t consider myself a UVM expert so the best way forward wasn’t immediately obvious. First I attempted to disable the components I wasn’t interested in by temporarily removing them from the component hierarchy. That was a dead end however because of the local access restrictions. My second idea was to replace components with a dud that effectively had no implementation… which means if the dud ran in place of an actual component, nothing would end up happening. That too was a dead end because of the same local access restrictions of the component hierarchy (in hindsight I should have known right away this was a no-go but I was still learning!).

The last implementation that ended up working involved creating and adding a new uvm_domain that ran in parallel with the existing UVM domain (where the run phases are stored). In the video, I talk about idle components that do nothing and a single unit under test that is driven through the run phases, iteratively if necessary. The idle components end up being idle because the newly created domain they’re assigned to – after of course going through the common phases of build, connect, end of elaboration and start of simulation – ends up raising an objection in the pre_reset_phase. That objection effectively means that components assigned to the domain never advance further. That’s what idle means… never advancing further than pre_reset.

I was pleasantly surprised to see that components could be assigned to different domains at any time which means I could activate and deactivate components whenever I wanted simply by changing which domain they’re assigned to.

When a component is deactivated, it is assigned to the new idle domain by…

function void svunit_deactivate_uvm_component(uvm_component c);
  c.set_domain(svunit_idle_uvm_domain::get_svunit_domain(), 1);
endfunction

Similarly, where I show a component being activated, it is being assigned to the normal uvm domain by…

function void svunit_activate_uvm_component(uvm_component c);
  c.set_domain(uvm_domain::get_uvm_domain(), 1);
endfunction

That seems to be the trick to being able to run uvm components sequentially in SVUnit without adding any new control or requirements to the components themselves, something I was looking to avoid from the outset.

If you want to take a closer look at the code, you can become an SVUnit project member and early adopter by getting a hold of me at neil.johnson@agilesoc.com.

Bonus marks go to the person that looks at the code and can suggest a better way of doing the same thing!

-neil

Share/Bookmark

We need to find a way to work smarter (not harder).

The ultimate bit of useless advice… work smarter, not harder – or some variation thereof.

Thanks. Will do.

From managers (sorry for picking on managers but I think they’ll probably agree with me on this), the “work smarter not harder” imperative never seems to have the intended effect. It comes across more like “maybe if you thought about things a little bit we’d be done by now, we’d be getting rich off our stock options and instead of wasting your evenings in the office, you could be out waterskiing behind your new boat or golfing in Hawaii”.

What’s wrong with “work smarter”? It’s vague, there’s no method or actual goal, it devalues the hard work you are putting in, it always seems to come at a time of intense stress, and it injects panic instead of hope. Not exactly helpful. It is logical advice though and for some reason, I spent my weekend thinking about options for work smarter, not harder that teams can use to challenge themselves before the steady 12 hour work days kick in.

Here’s what I came up with…

• What would we have to do to cut our tools budget by 50% while maintaining our productivity and quality?
• How could we cut our regression times in half without increasing the size of our server farm?
• How could we get 5 days worth of work done in 4 days?
• How would we organize ourselves if we had no one to manage us?
• How could we limit the number of known defects to 1 or less (where less == 0)?
• How could we cut the size of our product by 25% without losing customers?
• How could we condense our design documentation by 75%?
• How could we lower our defect rate by 90%?

You’ll notice that these are pretty aggressive ideas and that’s on purpose. Aggressive goals like these would force teams to think outside the box (there’s another useful cliche ) to find new/interesting/revolutionary changes to the way they work instead of looking for the evolutionary optimizations that never seem to get us much further ahead. These ideas are probably a good way to recognize the wasted time and effort in your development process that you’ve been accepting without realizing it.

So the challenge is… get your team together, pick a seemingly outrageous goal from the list I have or think of one on your own, brainstorm a list of ways to make it happen (no idea is too stupid) and then go do it.

Small optimizations are not an option. Go big or go home .

-neil

Q. What “work smarter” goals would you add to the list? How would you (or have you) reached those goals?

Share/Bookmark

From page 31 of The Machine That Changed The World, this is part of the last paragraph in a section that describes specialization of the skilled workforce in a mass production system:

As time went on and engineering branched into more and more subspecialties, these engineering professionals found they had more and more to say to their subspecialists and less and less to say to engineers with other expertise. As cars and trucks became ever more complicated, this minute division of labor within engineering would result in massive dysfunctions…”

What if we mangle that last sentence to apply to hardware development?

“As [systems become] ever more complicated, this minute division of labor within engineering [will] result in massive dysfunctions.”

Does that comment make sense when we take count all the people required to put a piece of hardware together? I can think of systems engineers, design engineers-digital and analog, functional verification engineers, physical design engineers, software developers, support and application specialists, tech-writers and various layers of management. I know I’m missing others.

Some specialization is necessary, but how much is too much? Do you see massive dysfunctions arise from specialization in hardware development?

-neil

Share/Bookmark

This post looks at some ideas for future directions for SVUnit.  Both Neil and I put forth our views.

Bryan’s Views

I’m putting these ideas out there with a hope that you’ll tell me that my thoughts are good, impractical, ridiculous, or feel free join in the discussion and share your own ideas on where we should go with the SVUnit.   Now that the source code is open-source (Apache License) and pushed into a sourceforge project we can all contribute and use the framework.

But first, I’ll throw my ideas onto the wall to see what sticks!

UVM

This framework was created pre-UVM so you can probably guess that I think it might be a good idea to either convert SVUnit over to be an UVM compliant framework, or at worst fork this library off and create a new project e.g., uvm_unit.  That is, the non-UVM SVUnit will continue so that people not using UVM can still get value from the work, while the UVM compliant version can also develop in parallel.

I think creating a UVM-compliant library for unit-testing is necessary.  The SystemVerilog world is standardizing on the UVM base class library and creating something that is eventually not compliant seems pointless.  I do appreciate that there are still plenty of projects that are using pre-UVM base class libraries and can still benefit from a methodology-independent TDD framework — which is what SVUnit currently provides.

As well, if it is a UVM-compliant library, it is conceivable that it could be discussed, and eventually rolled into the UVM “standard” — giving a consistent way for all of us to do TDD.

What do I mean by ‘converting’ to UVM?  At the very least, the UVM Unit would have the following features:

• Move all messaging to use the `uvm_info, `uvm_error  etc. macros.
• Deriving each from the uvm_component so that it can benefit from the standard phasing and objection mechanisms.

The last one is bit open for speculation i.e., should I derive from uvm_component, or is there a more appropriate class like the seemingly appropriate uvm_test.  From my perspective, the SVUnit framework needs to be fairly light-weight.  (I find it difficult to describe UVM as ‘light-weight’).  Comprehensive yes; light-weight, no.  Meaning that while it clearly defines a very nice structure to build a verification environment, it does require a lot of classes and a lot of ‘plumbing’ code to get running.  Albeit true, once you get all that plumbing code finished, you have a fine structure to work from to add the application specific code i.e., what we get paid for.

This post is already long, and I have some other ideas to cover so my next post will start to put out some further ideas on creating a UVM-compliant SVUnit library.  Here’s a teaser UML diagram of what I’m thinking.  More on that in a later post…

Generator

The second direction that needs to be considered is to improve the current batch of scripts that are available to generate all the ‘plumbing’ code required by the framework itself.  Neil has done a great job updating these scripts to be more robust than the initial implementation.  These generators make it truly convenient and quick to create a test environment to run your unit tests quickly.  Two key elements of TDD are “fast” and “easy”. In other words, you want to create a testcase quickly and easily, and you want to run a testcase quickly and easily.  The generator scripts enable that fast and easiness.

My current thoughts on improvements to the generator scripts are:

• Look at using the more standard ‘template’ libraries to generate the code.
• Generate the appropriate UVM related infrastructure
• Wrap a simple GUI Test Runner to allow people to execute, report and view test case execution.  Related to this is that I’m currently looking at Jenkins as a continuous integration tool (think continuous regression).  If I could tie the two together that would allow me to launch jobs and then the pass/fail reporting would be done for me.  Jenkins uses a web-based front-end.  Just a “twinkle-in-my-eye” for now.

Eating My Own Dog Food

Now, do I use TDD for creating the UVM-SVUnit?  Kind of recursive, but you bet.  Another task for this framework is to create a set of unit-tests that test the framework.

Neil’s Thoughts

First on the list is eating my own dog food. Personally, I think with TDD we have a technique that can provide immediate benefits to hardware teams in that it’s tangible, understandable and doable. TDD doesn’t require major organizational change or buy-in (really… you can do it by yourself without anyone else even knowing) and SVUnit is easy to use.

On the flip-side, TDD and SVUnit are also unproven in hardware hence me having “eat my own dog food” at the top of the list. I’ve already started using it for all the new code I’ve written and the results have been promising though not yet conclusive. I’ll continue to use it so I can get a better idea of what works, what doesn’t and what makes sense in future.

Next on my list is supporting others that are interested in SVUnit (we’re actively looking for people to… uhh… eat our dog food so if you’re interested in being one of the first few to try SVUnit, you can let us know at neil.johnson@agilesoc.com or bryan.morris@agilesoc.com). I started the SVUnit Demo Series that should help get people going. I’ll continue with more of those in future. We also have some usage examples to update. We need an FAQ page that new folks can reference for the easy stuff. Basically, whatever else people need to make an informed opinion, I’m ready to put it together .

Then there’s “testing my own dog food”. This is something I started a couple months ago. The SVUnit testsuite isn’t comprehensive yet (ie. there are a lot of untested features in the SVUnit code), but it’s started so we can regress some of the core functionality. I’m also verifying new features and updates as they happen.

Honorable mention on my list is to write a project charter. Before we have too many hands in the cookie jar, we’ll need to capture some big picture goals for SVUnit with a list of things that we’re not going to concern ourselves with. Considering it’s an open-source project, we’ll also need some development guidelines.

Q. What are your thoughts on the direction where SVUnit can (or should go)?

Share/Bookmark

Neil and I want to wish everyone following AgileSoC a very Happy New Year!

AgileSoC has grown tremendously over the past year, and we’ve received many appreciative and insightful comments about what Neil and I are trying to do.

We look forward to 2012 — where our conversation with you about AgileSoC will continue to grow and develop.

Bryan & Neil

Share/Bookmark

For this topic espeically, I’d like to hear your experiences with TDD, whether you think TDD is an appropriate methodology for RTL, and if you’ve been following along with our TDD-month(-and-a-half), whether you agree with Neil’s and my position that TDD is a great addition to our ASIC/FPGA development toolbox.

Why RTL Designers Should Use TDD

While a Verification Engineer should see the immediate benefits of using Test Driven Design, the idea of an RTL designer using it is perhaps a little more difficult to grasp.  The difficulty is when you start to think about the amount of infrastructure required to create a unit test — it can be large for a reasonable sized block.  Just providing the required stimulus across the tens, or sometimes hundreds of pins can seem impractical.

However, that presumes that you want to create a unit test from a fully completed RTL block. Instead the TDD methodology encourages the incremental creation of tests and functionality.  In the case of RTL, this would begin with the creation of simple stimulus to test the minimal interface — ideally built as the RTL designer creates the block. Starting from a simple core with clock and reset and minimal functionality the unit test is straightforward.  As the designer continues to add more features, the signals and stimulus required in the unit test build gradually and incrementally.  In other words,  TDD for RTL is very similar to what was described in previous posts regarding adding TDD to the verification process — you build the tests and functionality incrementally.

First, let’s go through some of the benefits of using TDD for RTL development.

Simple Stimulus

Every RTL designer I have ever had the pleasure of working with has always wanted a simple testbench that would allow them to provide some simple stimulus to their block. Using TDD with a standard framework such as SVUnit, the goal is to create every module with very simple stimulus since it is at the module level, and then gradually add the extra functionality and signals required until the module is complete.  The unit test will cover all possible stimulus, many of which would be difficult to provide at a block-level test consisting of several interacting modules.

Documentation

As with Verification Engineers, the unit tests demonstrate the block’s interface expectations. A block that must interface with that block could potentially use the unit test’s functionality as the driver or receiver of data from the block depending on the direction of the data.

Easier SVA Adoption Curve

It’s easy to see from an RTL perspective why adding SystemVerilog Assertions into a module/block is beneficial. One issue to adoption is the complexity of building assertions on a reasonably complex block.  That is, many assertions require complicated system-level test stimulus in order to activate the correct sequence of signals to trigger an assertion.  Creating unit tests is a complimentary technique, and can in some cases provide some of the same value of assertions — the ability to pinpoint where logic is incorrect; identify conditions where the underlying assumptions are thoroughly tested.  It is a complimentary in that the stimulus provided starts out simple and grows incrementally more complex — the checking of the DUTs responses can be done in either the SystemVerilog functionality in the unit test, or with an SV Assertion.  Any assertions created also begin very simply and can grow gradually more complex.  Again, building both confidence in using writing assertions, and confidence that any new assertion builds on previously validated assertions.

TDD for RTL Workflow

How would I create an RTL block using TDD?  Here are my thoughts; I’d welcome your opinions.

Let’s assume that we’re creating a simple round-robin arbiter block.  Briefly, the design has a simple interface:

• Clock and reset,
• Input request signal (one bit per client),
• Output grant signal (one bit per client).  ≈
• Round-robin arbitration across all asserted requests.  One per clock cycle.

As discussed in previous posts, the typical TDD workflow creates a unit test first; by design this test fails because the associated functionality does not exist.  The first unit test can be as simple as instantiating an empty module with pins for clock and reset and then hooking up the clock and reset generation logic to these pings.  The ‘test’ is defined to succeed when the reset signal in the block is deasserted at the same time as the external reset generator is deasserted.   Following the TDD flow, you’d attempt to compile this unit test and it should fail spectacularly as the block does not even exist.

The RTL designer creates the block with the clock and reset signals to make the reset test pass.  The test passes when the internal block comes out of reset.  A simple assertion could be used to test this condition.

The next test to create is the resetting of the internal pointers that maintain the updating of the request and grant logic, and the current client state.  An assertion to check these values after reset should be sufficient.  Again, the test fails as the logic to reset these values does not exist.   Add the logic until the assertion no longer fires and the test passes.

As you can see you gradually build the functionality of an arbiter e.g., adding the independent requests and generation of the grants, keeping state of which client has been served last, meanwhile adding increasingly meaningful assertions and functional checks into the unit test as you continuously add the functional components.

Of course, most modules are more complicated than a round-robin arbiter, but my intent is to show that TDD encourages an incremental approach to design.  Each incremental step building on the strength of knowledge that the previous step is solidly grounded in working code.  Each incremental test and assertion with a corresponding functional addition contributes to increasing confidence that the code is correct by construction. Truly the whole (unit test + module) is greater than the sum of its parts.

You’re also secure in the knowledge that if the interface changes with a dependent block, a simple run of these unit tests confirms whether your block has been affected by the change.  Similarly, any additional functionality that is added later in the design cycle, you’ll have a solid foundation from which to build.

My goal here is to demonstrate how one could create an RTL module using TDD, with the end-result of a working module, and a set of unit tests that ensure the block works as expected.

If you’ve been using TDD-like ideas in your RTL development, please share them.

The Last Post… on TDD… for now

I’ll be pushing out my last post on the TDD topic in a couple of days.  This will define my ideas on some possible future direction for SVUnit.  Neil has pushed the source code into sourceforge and we’re looking for some early adopters to test it out before a general release.  Neil also posted a video demonstrating the svunit in action.  If you’re keen and serious to take svunit for a spin, please let Neil or I know.

Share/Bookmark

In my last posting, I gave a brief overivew of the SVUnit framework, and it’s usage model. This post will look at how to use this framework in your workflow: how it can be integrated into your existing verification methodology to help develop new verification components, and extend existing environments.

At this point, I’ll state that the key determinant for a successful introduction is attitude. The developers must see merit in using this approach, or at least trying the approach with an open mind. Otherwise, it will very likely not be successful.

Adding TDD to your New Project / Verification Component

Neil has discussed in previous posts about the value of adding the TDD methodology, and has shown that as an activity in a new project it is straightforward. The scripts in this framework make adding TDD even easier value by automatically generating code that is immediately usable. The following provides some recommendations on a workflow that we’ve found useful:

• Create the class header files with only the class definition (i.e., no implementation). Don’t worry about getting the class perfect, the intent here is to use the scripts to generate the scaffolding code required for the unit test framework.
• Use the provided scripts to generate your unit test case files and corresponding test suites.
• For each function and task in the system create a unit test case (derived from the svunit_testcase class). The focus while writing these unit tests is to create the usage model for your new verification component, or how each task/function is to be used in your environment.  This should be done iteratively, one task/function at a time building from a logical start e.g., following the UVM phase flow, or from the public tasks/functions and following their call flow.
• Add just enough code into task/function to get the test case to pass, debugging each task/function individually to ensure that they are correct as per the usual TDD “way”.
• Aggregate the testcases into meaningful test suites. Create these test suites using the scripts provided.
• Similarly, aggregate the test suites into the test runner derived from the svunit_testrunner (using the scripts provided).

At this point, you have a couple of very handy things:

• a working, debugged initial version of your class;
• a example usage model for your new users of that class – this usage model is found in the unit tests that your colleagues may look at;
• test code that can be used to ensure that when you add or change functionality (using TDD) in the class that the code continues to function as your users expect; and test code that can be periodically run to ensure that if any updates to classes that your code is dependent on continues to work as expected i.e. you’ll see when someone else affects your code.

Using SVUnit for Existing Classes

When using the SVUnit framework on an existing library of verification components you can either adopt an incremental or full approach. The incremental approach would use the scripts to generate the initial unit tests (all of which would be blank) and then add the unit tests individually as required (e.g., when the corresponding task/function is being updated). The full approach would be to generate and define the unit tests for all tasks/functions immediately.

Which Classes to Pick

Ideally, TDD should be used for every class in your verification component. You will likely come to this conclusion after you’ve used TDD for a while, but adopting TDD strategically at first is still of great value. That is, cherry-picking the most complicated classes, or base classes that are used across multiple environments or projects are clearly the classes you need to pick first.

There are some classes that provide some minimal complication. For example, drivers/BFMs need an SystemVerilog interface or RTL to be able to drive to and respond. When the RTL that you’re driver is talking with is not available, using TDD on a driver is doable by adding some small tasks in your interface, or creating a small piece of RTL to replicate a source/sink for your driver. However, if the RTL is ready to be used, creating a driver using the TDD methodology does allow you to create a very simple mechanism to drive stimulus into your RTL. This is a very low-overhead, test harness that you can give to your RTL designer to use while you build out the rest of your environment.

When to Run Unit Tests

The last key element is to identify in your process when it would be best to run the unit tests. Ideally, the tests should be run by every developer on a regular basis to determine whether they have introduced any issues. The possibilities on where the test runner script can be added into your process are as follows (in preferential order):

• The best approach is to run the entire test suite prior to committing code to the source code management. Any issues must be dealt with prior to check in. Generally, the TDD tests for verification components are usually very quick (usually less than couple of minutes with most of that time taken up by the compilation/elaboration as tests do not use any RTL code).
• Another good approach is to add it to your commit sequence for your source code control system. That is, the check-in for any code is allowed to proceed ONLY if the test runner script passes. This assumes that you have a wrapper script around your source code “commit” or “check-in” command.
• If you use a “sanitizer” script for your source code i.e., where a code snapshot is qualified as being “good” (or in the agile parlance a continuous integration flow). This script usually requires a sanity regression to be run, if the regression passes then all the code revisions that make up the snapshot are labeled as “sane”. Adding the test runner into this script prior to running the sanity regression would be another possibility.
• Alternatively, the test runner script can be run manually prior to starting a regression. The decision to use the pass/fail result from the test runner script as either a “warning” (allowing the regression to proceed but the cause for the failure to be investigated) or as a “fatal” (where the regression is not allowed to proceed until the test failure is (fixed) is up to the project’s regression management team. (I’d vote for the fatal, but that’s just me .
• Lastly, the test runner can be run using a cron job on a regular schedule to take the latest “valid” snapshot of the source code and determine any failures. The key downside of this approach is that then someone has to isolate whose update caused the issue.  However, this should not generally be an issue since the granularity of the tests is so small.

My next post will look at two things: how TDD can be used by RTL designers, and some future directions for the TDD framework. From the comments received to date, there are some folks out there already doing TDD, and are already discovering its value. We as a group of professionals need to evolve TDD to work effectively in a design and verification flow. Give it a try and join the discussion!

Share/Bookmark

A couple of weeks ago, just after we got started with TDD month, Neil added the link to the posts on several industry forums, and got this comment from Alex Gnusin on the verification guild:

“Is it a Designer responsibility to test each line of code? In this case, there is a need to provide designers with working methodology to verify their code…”

To which Neil responded:

Alex: I’m not sure if you guessed we’d be covering the topic of a working methodology – aka: unit test framework – but if you did, I’d like to thank you for that nice bit of foreshadowing!

Alex is right, a proper framework is pretty important for anyone doing TDD; primarily because it gives you the opportunity to get up and running quickly.

In the software world, there are are number of wildly popular unit test frameworks. JUnit might be the best known (for those doing java development) but there are about a million others (as you can see with a quick trip over to Wikipedia). A unit test framework is critical for TDD, that’s why myself and Rob Saxe (both formerly of XtremeEDA) put one together a couple of years ago for people wanting to do TDD with SystemVerilog.

First presented at SNUG San Jose in 2009, SVUnit is a unit test framework that provides:

• structures for creating and running unit tests
• utilities for reporting test status and results
• a test “runner” that executes a set of test suites and report on their overall success/failure
• scripts to automate the mechanical aspects of creating new unit test code

This post briefly covers the usage model for using the framework.  Actually, I’ll split the post into three: this post covers how to use such a framework, the next post will discuss where to use it when  building your verification components, and my last post will discuss how RTL designers can potentially use it. As this post is more a teaser than a full blown article, you should ask the folks at XtremeEDA for a copy of the SNUG paper that Rob and I created.  It gives full details on the framework design and usage model.  Of course, the next question you’ll be asking is: where can we get the code?  We’re working on that… more details very shortly.

Usage Model

At its most basic, there are three hierarchical classes that you need to know about:

• svunit_testcase: an abstract base class that you derive from and add your TDD code into.
• svunit_testsuite: which aggregates and runs the svunit_testcase objects.
• svunit_testrunner: which aggregates and runs the svunit_testsuite objects, and reports the errors.

The following UML class diagram gives an overview of the classes and the hierarchical relationship between the three classes.  For those of you unfamiliar with UML, this diagram shows that a svunit_testrunner holds one or more svunit_testsuite objects which holds one or more svunit_testcase objects.

SVUnit Class UML Diagram

Every testcase that you create derives from the abstract svunit_testcase base class object. It has three methods that you’ll need to know about:

• setup(): where you can optionally setup the pre-conditions for the test.
• run_test(): where you add your test code.
• teardown(): where you can optionally restore the verification component back to reasonable defaults.

There also two macros to help in creating the tests:

• FAIL_IF(expression): similar to an assertion where the testcase fails if the expression is true. e.g., FAIL_IF(my_packet.calc_crc() == TRUE);
• FAIL_UNLESS(expression): corollary to FAIL_IF, where the testcase fails if the expression is false. e.g.,`FAIL_UNLESS(my_packet.get_parity(data) == 0);

Then you can start creating unit tests in a class called class_under_test_unit_test (which is stored in a file called class_under_test_unit_test.sv). This class derives from the svunit_testcase base class.

Add your testcode into the run_test() function. If required, you can add pre-test and post-test code in the setup() and teardown() tasks, respectively.  As discussed in previous posts, the idea is to create a set of tests that focus on one aspect, or an atomic feature of your verification class. These tests will incrementally build over time as you develop your verification class.  I suggest you create a set of tasks inside your classname_unit_test that are called by the run_tests() task. e.g.,

task run_test();
    `INFO("Running Unit Tests for class: packet_bfm:");
    test_pack_bytes();
    test_unpack_bytes();
    test_send_data();
endtask : run_test

There is some “plumbing” code that needs to be created to instantiate and run these testcases. As this is a tedious, repetitive task we created some basic scripts that generate the code for the svunit_testsuite and svunit_testrunner classes. Ideally, all you’ll ever need to write is the unit test code. The script finds all the source files that match the pattern *_unit_test.sv, and then creates a corresponding class (derived from svunit_testsuite class) responsible for instantiating and running your unit test objects. The script then wraps those test suite classes into a test runner class (derived from the svunit_testrunner) class responsible for instantiating and running the test suite objects. At the end of the test, the auto-generated svunit_testrunner object reports on the pass/fail of the unit tests.

As discussed in previous posts, the flow of using TDD is to create the tests first, and then add just enough code into the class being tested until the tests pass. The flow of creating a unit test with the svunit framework is similar:

1. Create your unit test class derived from svunit_testcase. If the ‘Class Under Test’ does not exist, create an empty class that your unit test class can instantiate.
2. Add you test case into the run_test() (or better yet in a separate task that run_test() calls).
3. Create the plumbing code by calling the script to auto-generate the derived svunit_testsuite and svunit_testrunner classes.  The script also creates a Makefile to run the tests). Don’t touch any of these auto-generated since everytime you run the script they will be overrwritten.
4. Run the test (make sure it fails!!).
5. Add just enough code into your class under test until the test passes.
6. Repeat steps 2 to 5 until all aspects of your class under test are completely tested.

The svunit framework provides you with some tools to make TDD easier to use — you focus on the individual tests; the framework runs those tests. Of course, there is more to this framework, but I would just repeating the contents of the paper in a post. Just ask our friends at XtremeEDA for a copy of the paper.

This post provides a brief introduction to the SVUnit framework.  The next posts will provide some thoughts and guidance on where to use this technique for verification and RTL  As well, I’ll discuss some future directions for this pre-UVM framework.

Share/Bookmark

Most teams nowadays chose to split their verification effort up as a combination of block level and top level testing. Block level testing is usually applied to every subsystem in a design and intended to exhaustively cover all the features of each subsystem. Once the block level testing is done, the exhaustively tested pieces are integrated and tested in a top level testbench.

The need for block level testing is justified by the fact that the arms-length control and visibility normally available in the top level testbench is inadequate or inconvenient for testing the first and second order functionality we talked about in the last post. The need for the top level testing is obvious as well because the block level testing does nothing to verify the interdependencies between subsystems and the operation of the system as a whole.

Makes sense… block level testing to verify the details; top-level testing to verify the integration. Here’s a picture of that as an timeline with the design, block level and top-level testing.

The designers start writing RTL. Verification engineers start building a block level testbench at the same time (or shortly thereafter as I’ve shown here). When the RTL and the block level testbench are done, the verification team will start running tests and collecting coverage results. When the block level testing is done, the block level design is signed off and it’s ready for top level testing.

The top level testbench, in the meantime, is usually started after the block level design and verification is well underway. The top level tests get run when most the block level designs near sign off. At the conclusion of top-level testing, the entire design is signed off and the design goes to the fab (or the FPGA goes to a customer, or the IP goes to another team or whatever). You’re done.

Now let’s look at the problems that normally pop-up and the reasons for why things never seem to work out as planned.

1. The RTL is NOT done: I’ve harped on this before so I’m not going to spend much time on it here other than to say that I would challenge anyone that says the RTL Done milestone we all race toward is at all meaningful. You may have a bunch of code, but it’s not tested and it’s not done (see By Example: Done vs. DONE).
2. The testbench is NOT done: same comment as above. Testbenches aren’t DONE when we say they’re done either. That goes for block level and top level (again… see By Example: Done vs. DONE).
3. You can’t plan your way through integration testing: here’s the big one and the real reason you’re here today. I really believe I’ve seen top level testing planned as good as anyone could reasonably plan. Well partitioned design: check. Well defined block level interfaces: check. Standardized model interfaces: check. Uniform, reusable block level environments: check. Customer initiated use cases: check. Skilled team: check. The ducks were all lined up and still there were the same old issues that tend to blow apart a schedule. A plan is one thing; practice is usually something else.

Take those and other issues into account and what you get in practice is more likely to resemble this…

RTL done and testbench done mean you have a bunch of buggy design and testbench code that doesn’t really work. After saying they’re done, experts from both domains transition into a firefighting role as soon as the first few tests uncover major quality issues (the firefighting is shown as red extensions to the original estimates). While that’s going on, the person in charge of the top level testing finds out that everything doesn’t actually just fit together as planned. Because people are firefighting at the block level and because the block level is given priority, the top level people have to make due until help finally arrives; if it arrives. A lack of help means the top level testing drags on, eventually stumbling across the finish line days, weeks or months later than expected. By the time the design is finally released, people are exhausted from weeks of stress and sleep deprivation.

I used to be of the opinion that teams could plan their way through top level verification. Now I’m sure they can’t.

I think the difference between planned and actual release dates is caused by false confidence and ambiguity in our development process. I think we can overcome both with a new paradigm for hardware development that emphasizes early validation and prioritizes top level development. Here’s a simple diagram of this new paradigm…

The first thing to point out is the alternating test-design cycles at the unit level. This is TDD. That’s meant to clean up localized/unit level bugs in the design before they’re committed to the database. Notice also that the unit level design starts with a purple bar. Test first!!

Next major point is that there is no delay to top level testing; it starts immediately. As portions of the design are unit tested, they are also verified at the top level. Top level tests are where we measure progress. When the top level tests pass, then – and only then – is the code considered DONE.

A third note here is reserved for the role of block level testing. In this new paradigm, block level testing is no longer automatic because TDD is used to prevent many of the first and second order bugs and integration testing happens much sooner. While there may still be a need for block level tests to, for example, isolate certain complex features that are hard to isolate at the top level, exhaustive testing of all blocks within a design would no longer be the norm.

We’ll still do block level testing but it’s no longer “on” by default.

What’s required to keep everything together in this new paradigm for functional verification? One word…

TEAMWORK

The top level effectively takes priority in the testing, which means all the development at the unit and block levels must be synchronized such that it feeds the top level effort successfully. Everyone needs to know what is going on and how there work supports the combined effort of top level testing. That takes teamwork, plain and simple.

-neil

Q. What problems do you see during integration testing? Does unit testing and/or early integration testing address those problems?

Share/Bookmark