Yes… I know it’s a good tool. People I respect use it and love it. It’s not that I have any specific complaints about Specman, it’s just that I grew up with Vera and SystemVerilog and with the universal support and attention SystemVerilog gets, I’ve never had the urge to try Specman…

…though it seems tool developers at Cadence are trying to change my mind.

I just watched their Test-the-Testbench webinar delivered by Hannes Froehlich, Cadence solutions architect, that demonstrates the unit testing and mocking features Cadence has recently added to Specman. Those are features that should be very appealing for hardware teams that understand the value of software techniques like unit testing or TDD. I’d recommend people watch the webinar. It’s only 30min. Here’s the highlights I saw.

First, there’s the title of the webinar: Testing the Testbench. I’m assuming someone at Cadence has seen enough of their customers bungling their way through poor quality testbench code that they figured it was time to suggest verification engineers test their own code. I’ve spent a couple years on this same topic and everyone I’ve talked to thinks this is a reasonable idea. Now we can add another (much louder and more influential) voice to that discussion.

Next, there’s a slide that explains the value of adopting a software-like approach to hardware verification with a definition of unit testing. You can see the emphasis given to focused and isolated testing at a fine granularity. This is what we’re missing in hardware development and specifically what Cadence is suggesting we consider. This is software testing in a nutshell that we can’t continue to ignore.

Then we get to the layout of the test framework. For people unfamiliar with unit testing, this is typical of other xUnit frameworks. There’s a top-level entity, which Cadence calls the test launcher. Then there’s a series of testsuites each containing one or more testcases. Also on this slide is the point of fast runtimes.

A little outside of my element is how the testsuite and unittest templates are generated. You can specify a testsuite and UUT fairly easy by feeding a specification like this through specman…

With that, you’re creating a testsuite called calc that focuses on the calc_crc method in the my_checker_u. The tool parses the specification and code then spits out an empty template which I gathered is not something you would modify. I believe you also get a testcase template out at the same time the testsuite is generated.

Here’s the beginnings of the testsuite…

And the testcase…

Now, if Cadence stopped there, you’d have something that looks a lot like SVUnit. Scripts to build testsuites and testcases and the framework that pulls everything together (the underlying mechanics are entirely different b/c of the language differences but for users, the look-and-feel is very similar). The big extra though happens at about 22:15 of the webinar where Hannes quickly (too quickly) shows that Cadence has also built in the means for automatically generating mocks that substitute dependencies with simple hooks that let users target very specific functions.

On this line, you can see a mock in action where ‘eu_return_value_of_valid_address’ is used to override the functionality of the my_monitor_u checker. If you’d have been watching closely, you’d have seen the original functionality back at 18:58 of the video (hint: line 22).

That’s isolation. The mock gives you direct control of the branch conditional on line 22. Hannes sets it to FALSE in the video to target the code on lines 23 and 24. He could set it to TRUE in a different test to target line 26. Mocking in a (teeny tiny) nutshell; replacing dependencies with direct control that allows for exact isolation. I’d like to see Cadence post a follow-up video to explain more about how the mocking works and the types of things you can do. Unless I’ve read too much into this, I think it’s quite a lot!

So, upside of the webinar and new unit testing features:

• Cadence suggesting verification engineers test their testbench code
• A quick description of software testing and the goals of unit testing
• A framework that’s very similar to existing unit test frameworks from a user perspective
• Scripts for building testsuites and testcases
• Automatic mock generation

Minor downside that I pulled out:

• The templates are quite verbose. I’d expect people to grow tired of the comments and want them removed at some point, though they will be good embedded documentation for new users.
• They suggest individual files for each testcase. People will probably find that a tad annoying as well, especially if you’re doing TDD. For me, unit tests have become documentation and visually, it’s nice to have all that documentation in a single file for a single entity. I didn’t see anything preventing more than 1 testcase/file, it just wasn’t the suggestion.
• Hannes notes that the approach is good for new code development but gets tricky for retrofitting existing code. I’m assuming the problems arise from partitioning of the UUT as opposed to the tool itself, but it would be nice to get clarification there. What are the issues for legacy code?

That’s it for my look at the new unit testing features in Specman. My final 2 cents… I think this is a very promising development from one of the big 3 EDA companies. It’s someone else validating that there are pro-active solutions to the debug epidemic. It’s also validation of a lot of the work we’ve done on AgileSoC with respect to unit testing and TDD as well as the work that’s gone into SVUnit and MiniTB.

Finally, if you’re not a specman user but you still want to try unit testing, this SVUnit tutorial is a good place to start. With SVUnit you get the framework but not the mocking.

-neil