Through the hardware industry’s continuing infatuation with leading verification technologies – constrained-random verification, functional coverage, numerous fancy methodologies, intelligent testbenches and a host of others – the needs of designers have been thoroughly ignored. That changes with MiniTB.

What is MiniTB?

MiniTB is an open-source Verilog framework designers use to smoke test their code.

MiniTB is easy to use. After instantiating and connecting a module-under-test, a designer can write and run any number of smoke tests to verify their code’s sanity. Smoke tests are compiled into a single executable which means little time lost to compilation and fast turnaround. Test status is determined by simple assertions and MiniTB exits with an overall PASS/FAIL summary status. MiniTB supports Cadence Incisive, Mentor Graphics Questa and Synopsys VCS.

Getting Started with MiniTB

MiniTB comes with a packaged example to help get you started. In the APB slave example, you’ll see tests in apb_slave_minTB.sv that look like this:

When you run the smoke tests in the MiniTB example, you’ll get a log that looks like this:

That’s 4 passing tests that how our example module-under-test is sane and ready for exhaustive testing.

The complete 2-step instructions for running the example are can be found in the ‘examples/module/apb_slave/README’.

Full MiniTB Support for Early Adopters

If you’re a designer interested in using MiniTB, we want to give you a hand. We’ll maintain a short first-come, first-served list of early adopters that will get our full attention as they get going with MiniTB. Being an early adopter also gives you the opportunity to shape future releases of MiniTB.

To become a MiniTB early adopter, email us at: neil.johnson@agilesoc.com.

-neil

Share/Bookmark

My First SVUnit Test Challenge

Get the latest version of SVUnit from Sourceforge

Download SVUnit

Step 1: download SVUnit by pushing the big ‘Download SVUnit’ button to the right.

Step 2: unpack the tarball (‘tar zxvf svunit-v1.*.tar.gz’) and follow the steps in the README to create a UUT and corresponding unit test that demonstrate your commitment.

Step 3: take a snapshot of your code and test result and post it where you and everyone around you can see it.

An example you’re asking? I just happen to have one. My commitment to hardware quality is to verify a development_process won’t produce garbage code.

Here’s the UUT:

Here’s my unit test that confirms my commitment:

When I run my test, this is what I get out:

Test passes. I’m committed. Easy-peasy.

Now it’s your turn. Download SVUnit and follow the instructions in the README to verify your commitment to hardware quality. When you’re done, take snapshots and post them where you can see it and be reminded of your commitment. Bonus marks for people that send me a snapshot of their commitment to post on AgileSoC.com (neil.johnson@agilesoc.com). Double bonus points for creativity .

The world needs more people that are committed to hardware quality… so let’s get on with it.

-neil

Share/Bookmark

I’ve had some good feedback from a couple fellows using the report mock and today I released a new version to start incorporating it. SVUnit v1.4

includes a new UVM report mock.

Two significant changes…

First is that the macros are no longer required. It uses UVM facilities to eventually intercept the global uvm_report_(warning|error|fatal) functions. Logging messages with those globals directly or by using the stock `uvm_* macros both work the same now. Both can be verified using the mock. I’ve expanded the example that’s shipped with SVUnit to demonstrate. Here’s a snapshot of that code that shows our UUT using both the `uvm_error() macro and uvm_report_warning() function:

…and here’s the 2 tests we use to verify the error and warning are triggered properly:

Thanks to my colleague Jean-Marc Tremblay for handling this part of the upgrade.

Second change… with the initial mock, when something went awry and your logging doesn’t match up, the mock told you as much… but only by returning false via the verify_complete(). I didn’t have anything to help people diagnose why, but now I do. In v1.4, when the verify_complete() fails the mock dumps the expected and actual messages it’s logged so far. That dump looks like this:

You get the expected and actual ID, MSG and severity at each index. So this dump from the mock would tell you, for example, that the error and fatal you were expecting were logged in the wrong order (i.e. you expected fatal then error, but you got error then fatal).

Last change… I had the order of the ID and MSG mixed up in the original so if you simply plugin this new version, I expect you’ll see a lot of failures (in UVM, the argument order is ID then MSG but I had MSG then ID). To clarify, prototypes for the expect functions are now…

This won’t match what you have now, so unfortunately you’ll have to go back and swap the arg order. Sorry about that .

If you’re new to the UVM report mock, you catch up with my original post from a few weeks ago.

Enjoy!

-neil

Share/Bookmark

Up until now, I think SVUnit has been pretty decent for building systemverilog class and interface based verification IP because the class based structure of the framework was originally intended to support classes and interfaces. Modules though… not so much… so while class/interface support was a strength, module support – and consequently support for design IP – was a significant weakness. The intention of v1.1 is greater support for modules by changing to a module-based structure.

In v1.1, the unit test template is like a mini-testbench. There’s an instance of the UUT in the mini-testbench. The constructor from past releases that we used to create and connect any peripheral functionality has been replaced by a build() function that’s automatically called by the testsuite prior to anything else happening. There’s the same setup and teardown methods with a minor change then the test macros are the same as what was there. In terms of how tests are run and the sequence of events that are followed, nothing really changes.

I’ll be writing more about it in the next few weeks as I continue to simplify the code base. In the meantime, I’ve embedded a condensed dump of the unit test template created by create_unit_test.pl (condensed in that I’ve removed some of the comments and whitespace so it fits on my screen).

If you’ve got comments on the format relative to prior versions, I’d appreciate you posting them. If this is cleaner and more usable, I’d like to hear it. If not, of course I’d prefer to hear that, too .

-neil

Share/Bookmark

Scaled Agile Leadership – Lessons From the Military

Jim Tremlett’s 9am talk Scaled Agile Leadership got the day rolling. Jim’s talk revolved around the scaled agile framework and the idea that the military, with it’s model of decentralized control, provided a good example for how different teams within an organization can be given the latitude to make their own decisions while still working toward to common good. He talked about how information needs to flow through planning layers in an organization to be effective as well as decentralizing control and knowledge in an organization and the need for supporting technologies.

Jim finished with a specific example that talked about adjustments the American military made to vehicle convoys to protect themselves during the Iraq war. The case study typified the ineffectiveness of imposed solutions while showing that practical solutions can be encouraged by leaders willing to decentralize control of the situation. That allows the people best suited to solve a problem to come together and do so.

Tip of the day from Jim’s talk: more value created with overall alignment than local excellence.

Agile Applied To Hardware Development… We’re Not That Different After All!

Round 1 of 2 for me on the day started at 11am with Agile Applied To Hardware Development. For someone that’s used describing agile hardware to empty seats about agile hardware, today was refreshing. Still lots of empty seats, but only because the auditorium was so large! We had a good crowd .

The talk went well. I started with material I first presented at Agile2011 in Salt Lake City. I gave my take on the challenges we face in hardware development, then proposed how we apply the values of the agile manifesto to clear those challenges. That kind of stuff is pretty general but I think it’s necessary to talk about the needs of hardware developers before you get to how we apply agile. It’s not stuff you can specifically practice, but it’s necessary context to see the possibilities.

The more practical part of the talk was the last half where I talked a bit about the first time I attempted to be agile in 2009 and then shared my view of agile development as it applies to front end ASIC/FPGA development, that came from my Agile2012 talk.

I switched things up a bit this morning by adding a shot of the Commitment to Agile Hardware Development blog I wrote a couple months ago (I actually added it to both my talks today so I didn’t miss anyone). I suggested that the end of a conference was a great time for people to commit to doing something different. I’d love to see people take me up on that challenge!

An Incremental Approach to Functional Verification

Now this talk had potential but I also knew it was a bit of a risk. I wanted to bring some specific examples with details while at the same time, not get so detailed that I lose people. Turned out there was no problem at all. I thought the 2nd session actually went better than the first. Turns out hardware developers like detail. Who knew .

This talk is quite focused on the design and verification aspects of ASIC development. I identify 4 characteristics of late delivery – delays in initial results, subsystems prioritized over products, subjective progress metrics and long debug cycles – with a look at how I’ve addressed them using agile techniques. The big takeaways I’ve emphasized in this talk were TDD of the testbench components and incremental development of the functional model. New for this time around was a look at applying SVUnit to legacy code and using TDD to add features to existing code.

Discussion from both sessions was absolutely excellent. We had questions about how my small case study example would scale to a larger team of people. We had a fellow that worked in validation asking about how he could become part of the design process. There were questions about measuring velocity and how I defined done (code written doesn’t factor into it at all). Also met a guy that maintained a framework similar to SVUnit (it was nice to have discussion about TDD and unit testing spilled into the hallway). Naturally, he won a prize at the end of the session.

It was a great week for me and I’m glad to have had the opportunity to be a part of it. I met lots of great new people, learned some new things and had a great time. It’s weeks like this that keep me motivated to go further with agile development. These guys have their stuff together. I’m expecting big things out of them in the next few years.

Thanks again to the coaches at Intel for inviting me!

-neil

PS: almost forgot the question of the day. I had a coach… who shall remain nameless… ask me how to pronounce A-S-I-C. Love it.

Share/Bookmark

Then you spend a week in Hillsboro and find out that there are hundreds of people working toward the exact same thing you are.

Surprise! The agile hardware community does in fact exist. It’s a relief to actually see it .

This week, I’m at the Intel Agile and Lean Development Conference in Hillsboro. Back in March, the agile coaching group at Intel were nice enough to offer me a few slots on the program and I gladly accepted (who would pass up the opportunity to talk to actual hardware developers about agile hardware??). It’s been a great 3 days so far and as I’ve done for previous agile conferences I wanted to give people a run down of what happened. This’ll be part one (provided I get finished before I fall asleep on my laptop). I’ll probably post part 2 tomorrow.

Monday Coding Dojo

My week started on monday when had the chance to take part in a coding dojo. Having never been to a coding dojo before, I had no idea what to expect. From what I gather, you can have a coding dojo to learn just about anything. People here were focused on TDD. To do it, they took a problem – Conway’s game of life – with the language of their choice and solved it a number of different ways, with a number of different parters and a number of different twists.

When I walked in, the twist was people working together weren’t allowed to talk to each other. The only way to communicate was through code and tests. The next twist was that people couldn’t use primitive data types to interact with their design, only complex data types were allowed. I played passive observer for those 2 rounds and then jumped in for the last. That was a free-for-all where the only limitation was to use TDD. We used C# in visual studio (I think??) which obviously meant my partner was at the keyboard, not me.

Overall, it seemed people were quite happy with the exercise. There was a great mix of people from different sites and teams with different backgrounds and language and unit test framework preferences. It looked like a very good way to learn how to pair and learn TDD so it gives you the chance to kill two birds with one stone.

Attending the coding dojo wasn’t initially part of the plan but I’m glad I did. Here’s a link for more info on coding dojos if you’re interested: http://coderdojo.com/.

TDD in C Tutorial (Tuesday AM)

My big break with TDD came after I saw James Grenning’s TDD in C tutorial at Agile2011 in Salt Lake City. I’ve seen it once since at ESC in San Jose and Tuesday I got to see it again but this time as observer/helper (James is a pro so he didn’t need much help).

The difference this time around that I loved was that James runs his tutorial using Cyberdojo. If you’ve never heard of Cyberdojo, you should check it out. Instead of running on local tools or machines, exercises are done using a simple browser-based editor. You log into your session and everything is set up without any futzing around.

The magical part was that the Cyberdojo keeps a history of your activity where every run was captured as a red, yellow or green dot at the top of the screen; red is a failing test, yellow a compile/link error and green was passing tests. Stringed together, the dots show you how you’re doing. This is TDD so ideally, you’re looking for a repeating pattern of red then green (i.e. your test fails then it passes). You can also, at a glance, catch the pairs that weren’t TDD’ing! No red dots… no failing tests… no TDD.

“What you do tomorrow is your own business, but today we’re doing TDD”.

The best advice James had during the session was for people new to TDD: it’s a discipline to begin with but it ends up being an addiction. I’d agree with that.

An Introduction to Hardware TDD (Tuesday PM)

My turn. A 4 hour session dedicated entirely to hardware TDD using SVUnit. Seriously, what could be more fun!

As I said to everyone attending, the goal of the session wasn’t to learn TDD, it was to practice it. I gave a brief description of the TDD cycle and a few guidelines to start, then we got right into it.

After clearing a few technical hurdles (I’m quite sure no two hardware developers on the planet actually share the same environment setup) we did a kickstart exercise to introduce SVUnit. What’s a passing test look like, what’s a failing test look like, simple stuff like that. On test 3, most of the people in the class went through the TDD cycle for the first time ever. A couple more tests, people were familiar and we got to the real example.

In December I had the idea of putting together a modelling example based on the old HP48G calculator I had at university. It uses reverse polish notation and has about a billion different functions so it’s the perfect way to start simple, get increasingly more complicated and go as far as you want. Everyone used TDD to implement the numbers, stack and almost a few arithmetic operators. We did everything in pairs and one point I had people switch pairs and work on someone else code, exactly the type of thing that happens in the real world.

We had great questions and discussion all the way through. People were engaged and open to the idea. I know at least one person that’s committed to the idea which is great (as I’m typing he just sent an email saying he’s starting Friday morning). It’s pretty rewarding to see the interest.

The thing I didn’t do was hand out the Intel sponsored prizes at the end of the tutorial! I know who my winners are, I just need to find you before end-of-day tomorrow!

All-in-all, very happy with the session. Couldn’t have gone much better.

Wednesday Keynotes

The morning keynotes on wednesday were given by David Hussman from DevJam and James Grenning. I’m lumping these together because both guys had similar messages from different angles.

Both, over the years, have seen agile practiced so many different ways by so many different people and both had examples of how teams have tried to follow agile by the book while forgetting some of the motivations behind why they were using it in the first place.

David emphasized how agile had become a process that teams follow while forgetting about the product they’re delivering. Teams need to forget about the right and wrong ways to be agile and instead find the way that works for them. Today was also the first I’ve seen of dude’s law where Value is directly proportional to Why (why the product does what it does) and inversely proportional to How (how you do what you do); value increases as you focus more on the product and decreases as you focus on the process you use to build it (I think that’s right??).

James pointed out that many teams are so focused on the team and structure that they forget that technical competency is an obvious requirement of every development team; it’s a strategy without the tactics to follow through. He suggested that with Scrum being so popular, teams need to reconnect with the technical practices of XP; it’s the combination of the two that’s powerful.

Keynote highlight: David referring to his fashion sense as business camping. Nice.

Open Space

Wednesday afternoon was dedicated to open space guided by Diana Larson. Open space is a loosely structured way of letting people organize themselves into groups and conversations that matter to them.

Large sticky notes were spaced around the auditorium. People interested in hosting an open space discussion wrote their topic on the sticky note and announced it on the mic so everyone knew what they planned. Boards at the front of the auditorium started as an empty agenda. As people announced their topics, they put their sticky notes on the board to settle on place and time. When the topics stopped rolling in, everyone picked a session and the discussions began.

I teamed up with a few local guys and a couple fellows from Chandler to talk about agile hardware (surprise). We talked about some of the challenges of defining DONE for a hardware team, how to limit work in progress, the interactions between design and validation teams and TDD in RTL design.

Open space was an excellent way to enable the conversations that people were obviously interested in having. There was no looking around when Diana asked for topics, people dived right in. The boards were full and everyone was participating. Quite a good idea.

That’s it for today. I did skip a park bench session of industry experts that I’ll try and cover tomorrow. The two big sessions for me tomorrow are at 11am – Agile Applied to Hardware Development… We’re Not That Different After All – and 2pm – An Incremental Approach to Functional Verification. Hoping to have good crowds at both!

-neil

Share/Bookmark

To quickly rehash what you’re looking at, this survey was conducted by Catherine Louis and I last year. The intention was to get a feel for how successful hardware teams have been with their current approaches to project planning. As you’ll see, we had people respond from all areas of hardware development from all levels. We’ve found the results to be quite interesting.

If you want a little more background, Planning to Fail in Hardware Development is a good place to start. In there you’ll find a link to our initial analysis posted on eetimes. Other articles we’ve posted since:

• Planning to Fail: How Stable are your Project Plans
• Planning to Fail: Is Estimating Really this Difficult
• Planning to Fail: Long Hours, Mounting Stress and Aggressive Goal Setting

But enough talk from me. Since you’re here to see data, here it is!

-neil

Share/Bookmark

That’s what I figured Thursday while listening to Harry Foster’s analysis of the 2012 Wilson Research Group Functional Verification Survey commissioned by Mentor Graphics. The survey is meant to show design and functional verification trends from hardware development. It’s well done and well presented by Harry.

People that follow AgileSoC.com will know that I’ve been interested in one particular data point from the previous Wilson Research survey and I was quite interested to see the updated statistics. It has to do with the amount of time verification engineers spend debugging code. Here’s the numbers from the 2010 study.

Forget about everything except for debug because I don’t think any of the rest of this is significant. Debug is significant though and that’s why I love this slide.  At 32%, a verification engineer in 2010 was wasting 2 hours, 33 minutes and 36 seconds a day, on average, debugging code. That’s fixing defects that we inject.

It’s two years later and what’s happened?

We’re up by 4%… that’s what’s happened. That means we’re now wasting an additional 19 minutes and 15 seconds a day debugging code compared to 2010. That’s almost 3 hours/day and more than we spend on anything else. Horrid.

At first glance, it’s hard to blame that statistic on anything other than garbage code. As much as people would hate to admit it, I think it clearly shows that as an industry we have defect rates that are unacceptably high. Harder to explain, though, is why the number hasn’t improved.

One explanation: adopting advanced verification practices actually leads to higher defect rates, not lower. Why go there? Well… the survey data suggest that’s a possibility. Consider the follow points…

Since the last survey there’s been a significant uptick in teams using UVM. There’s also been a significant uptick in teams using verification techniques like constrained random, functional coverage and assertions (NOTE: that last slide is a comparison with statistics from 2007, not 2010). This has all happened during a period where there’s been a slight increase in debug time. Hence my conclusion that advanced verification practices haven’t quite had the desired effect in the area of code quality.

Do I expect broad agreement with this analysis? Not really. Of course there are other valid arguments that take design and team size into account which could lead to a different conclusion. But whatever you might argue, as much as we want to believe we’re going in the right direction, we need to be able to talk openly about whether or not that’s actually the case. The bottom line is blowing more than a 3rd of a person’s time fixing defects is what it is: unacceptable. We have to wonder if the complexity we’re building into our development process is actually leading to better results? Yes or no? Is standardization leading to portable results? Yes or no? Is reuse leading to faster results? Yes or no? Are our best practices really the best we can do? Yes or no? Should we be looking elsewhere (i.e. agile development) for better best practices? Yes or no?

The answers to those questions will depend on who you are so regardless of who you are, the best thing you can do is to sit down, think about them and answer them for yourself. And don’t overlook the fact that with the possibility of earning back part of the 2 hours, 52 minutes 48 seconds a day we’re wasting now, the potential payoff from a little reflection is huge !

-neil

(fyi… all the graphics in this post have come from Harry’s webinar. That’s the source, I’m just the messenger/arm chair analyst. I’d recommend following along with Harry’s analysis as I assume he’ll be busy posting over the coming months. It was good reading last time around; I’m expecting the same this year).

Share/Bookmark

Hi everyone. It’s good to see you all here.

Before I get started I just want to take a quick poll. By show of hands, how many of you are using constrained random verification? Right.. quite a few people… that’s what I figured.

Ok… keep your hand up if you use functional coverage to measure whether or not you’re finished your verification effort.

Alright… I see a few dropping but most of you have still got your hands up… that’s a good sign since we all know results from constrained random tests aren’t overly useful without functional coverage.

Ok… next… does anyone here test their coverage groups? I mean does anyone take the time to verify that their coverage groups, the actual code, is correct? Anyone? I see a lot of hands dropping… not a good sign.

Last… from the people that just dropped their hand I’ve got a question: should you trust your functional coverage model, probably the most important code you write, if you haven’t gone to the trouble of making sure it’s correct?

Hopefully not.

A fully populated functional coverage model has become a pretty important component of determining whether you’ve sufficiently traveled the design state space. It tells you that you’ve done all that needs to be done by observing all that needs to be observed. The coverage model is the benchmark by which DONE is measured, which is the way it should be… unless you’re coverage model is wrong (i.e. it’s a defect ridden pile of unverified code).

Luckily, writing unit tests with SVUnit is a great way to verify you’re coverage model is correct. Here’s how.

To start, testing your coverage model depends on an “obscure” built-in method for cover points and cover groups (I say “obscure” not because it isn’t important, but because I doubt many people actually use it). The get_inst_coverage() method is what we’re after; what it does is return a coverage percentage from a particular cover group or coverpoint. For example, if you have a coverpoint with 2 bins and you sample 1, the get_inst_coverage() returns 50. Sample both and you get 100. Sample neither and you get 0.

Here’s three unit tests that illustrate what I’m talking about. We have a coverpoint called addr_min_cp and it’s observing two addresses: ’0′ and ’4′. The first test confirms that when we sample addr_min_cp with ’0′ we’ve hit 50%. The second test confirms that ’4′ hits the other 50%. The 3rd test, where we combine ’0′ and ’4′, verifies addr_min_cp is correct.

Easy peasy.

With add_min_cp observing interaction with the low end of the address space, we can also observe interaction with the upper end of the address space. Here’s three other tests we can use to verify that our addr_max_cp is behaving as we expect.

Now how about a few equally spaced bins between min and max, as is often done by verification engineers to see that we’ve plucked an acceptable number of data points from throughout the address space. Here’s how we’d loop through 16 data points, verifying the coverage score along the way.

Important to point out from that snippet of code is that the get_inst_coverage() returns a real equal to ‘N * 1/16′, which means we need to expect a real or we find ourselves with a failing unit test.

Now the coverage points in those examples are pretty basic right? Who hasn’t implemented the pattern of MIN, MAX and a few in between? Anybody? And if we look at the code, there aren’t too many ways to screw that up…

…and yet there are ways to screw up just about anything, isn’t there, even basic coverpoints. Let’s say you define `ADDR_MAX  in some other file as follows:

Nothing wrong with that, until somebody changes the definition for a test that’s being written and mistakenly changes that line to this without understanding the implications:

Now, as I suggest in the title, your coverage model is wrong.

What happens from here? Well, if you have unit tests changing that line wouldn’t be a big deal because two failing unit tests would flag a problem and the person that made the change could go back and fix it. Without unit tests, however, the chances that this defect gets shipped to a customer – via a product with an inadequately exercised address space – is quite high. And what happens when your customer comes to you and says they can’t reach the upper end of the address space?

Oh… sorry. It looks like we had a bug in our coverage model so we didn’t see that. But don’t worry, it was an easy one. It’s fixed now.

Sure it’s easy to fix, though how many times can you get away with “oh… sorry”? That depends on your customer and your track record. A couple times might not be bad; several times will be embarrassing;  one too many can be catastrophic.

For as simple as it is to test your coverage groups with SVUnit, and for as critical as they are to determining progress and design coverage, I’d recommend erring on the side of caution.

-neil

NOTE: As of writing this, the build-in get_inst_coverage() method looks like it’s only be properly supported in Questa. I’m using version 10.1c_1. As of VCS version G-2012.09 and Incisive version 12.10-s008, get_inst_coverage() was not supported by Synopsys or Cadence. If you’re using newer versions, you’ll want to see for yourself if anything has changed. If you see a version of either that works, please let me know and I’ll post an update.

UPDATE: You’ll see in the comments that Cadence also supports the get_inst_coverage() method but you need to use the ‘-coverage <string>’ option on the command line (which I didn’t originally have). That’s confirmed for versions 12.10-s007 and 12.10-s008.

UPDATE: Count Synopsys in as well now as of at least vcs version G-2012.09. I had an AE help me out (he easily saw what I was forgetting). With the “option.per_instance = 1″ setting in your covergroup, the get_inst_coverage() returns a valid result. Without, you’ll see that it returns 0 in all cases. All of Synopsys, Mentor and Cadence support unit testing covergroups!

Share/Bookmark

Here’s how it works…

In a unit test, a user would go through the following steps to verify a uvm_error is being triggered properly:

• set the expectation of a particular error by calling the svunit_uvm_report_mock::expect_error(…) function
• apply the error causing scenario to the UUT
• call svunit_uvm_report_mock::verify_complete() to ensure actual errors are trapped in the UUT as expected (and that there are no expected errors left outstanding).

To help get those points across, we’ll look at the uvm_report_mock example that’s packaged with SVUnit. In this example we have a UUT with a method called verify_arg_is_not_99(). In case it’s not obvious, verify_arg_is_not_99() is designed to trap variables that are set to 99 by asserting a uvm_error. Other values are ignored.

To verify our UUT is working as we expect, we of course need a few tests. Here’s the first from the example that tests the error condition.

That test exactly follows the bullet steps above. We call the expect_error() method once because we expect one error. That pushes an item onto the expected queue of our log message scoreboard inside the svunit_uvm_report_mock. Then we call our UUT function with 99 as the argument. That pushes an item onto the actual queue of our log message scoreboard. Finally, we call the verify_complete. That does an in-order comparison between the actual and expected queues to confirm we’re getting the errors we expect.

To pass this test, this is the code we need in our UUT. An error is triggered for 99; nothing happens otherwise.

At this point, you might be wondering how the svunit_uvm_report_mock knows about UVM errors. Well it’s not as magical as you might think. To do it, we redefine the uvm_error macro to call the svunit_uvm_report_mock::actual_error(…) function instead of the normal UVM reporter. The macros are redefined in the svunit_uvm_mock_defines.sv and they look like this…

So instead of sending errors to your log file where you’d have to visually confirm they’re being asserted, we redirect them by mocking out the UVM reporting and automate the confirmation. For that to work, we have to make sure the macros are redefined as we intend. That’s done with includes at the top of the unit test file. The uvm_macros are loaded first, then the svunit_uvm_mock_defines are loaded to redefine the macros.

To exhaustively verify our UUT, we need one more test to make sure values other than 99 don’t cause a problem. In this test, you’ll see we still call verify_complete() to make sure we get what we expect, but we don’t call expect_error() because we’re… uhh… not expecting an error.

In this case, verify_complete() will return 1. That tells us no errors were expected and none were detected.

A final feature that we’ll talk about is the ability to expect a specific MSG and ID from your error. To do that, you pass in strings to the expect_error() function (the defaults are null strings which means the content of the message isn’t checked).

You can see that test is the same as the first except that the MSG and ID are passed to the expect_error(). Just getting the error in this test case isn’t enough, the content of the message has to match also. If it doesn’t, the test fails.

That’s how to automate checking of uvm_error logging with the SVUnit UVM report mock. FYI… uvm_fatal is also supported so if that’s what you’re looking for, just do a search-and-replace to swap error for fatal. The mechanisms for both are the same.

Lastly, I’ll mention that this is version 0.1 functionality so if you’ve got any better ideas for doing the same kind of automation, I’d love to hear it!

-neil

Q. If you don’t test your checkers, how do you know they’re working?

Share/Bookmark