Fast Sims Don’t Come From Fast Simulators

Posted on by nosnhojn

How many times have you sat through a conference talk or tutorial delivered by an AE from one of the big three EDA companies, the speaker announces a new version of their simulator, and the first follow-up question from the audience is how much faster is this version? 2x… 10x… 100x… how much faster?

Of course the response is always an increase in speed but it’s also always hard to quantify. It comes with caveats (i.e. “this is what we’ve seen… you may see different”) and no one is ever quite sure what kind of code is being used in the benchmarking. I’ve never had a simulation runtime improve from 10 seconds to 1 second with a new simulator version – maybe other people have seen that, but I haven’t – so I’ve never paid much attention to the speedup numbers. Call me skeptical. Continue reading →

FYI… AgileSoC DVCon Abstract in Progress

Posted on by nosnhojn

So the votes are in. 31 in yes column and 2 in the no column. That means I’ve got 2 days to put together an abstract for DVCon. In case you’ve missed it, the title is How UVM Makes the Case for Unit Testing. It’s based on the work that a colleague and I did unit testing UVM-1.1d with the project UVM-UTest. I’ve got the abstract posted here. If you’re interested in following along as I put it together, you can take a look over here. That’s a googledoc. I have the permissions set such that anyone can leave comments so feel free to tune in and comment as I go.

-neil

Percentage of time verification engineers spend in various tasks

Posted on by nosnhojn

If you saw last week’s post you’ll know that I’m pondering a DVCon proposal. As it stands, my proposal centers around unit testing in hardware development. Trouble is, unit testing on it’s own – nevermind TDD which is where I really want to go – has been a tough sell for hardware developers. Lucky for me, I’ve recently found that if I show off unit testing within a context people actually care about (i.e. UVM), suddenly they take an interest. Odd how that happens :). So that’s where I am, an abstract that describes the value of unit testing as we’ve applied it to UVM. It’s called How UVM-1.1d Makes the Case for Unit Testing. The proposal is not a sure thing yet. If you’re on board with it, you can help make it happen over here.

Assuming people like the abstract, I’m looking for an ice breaker or two that in addition to what we’ve done with UVM help convey the value of unit testing. That’s taken me back again to Mentor’s 2012 Wilson Research Group Functional Verification Study and a twist on the time wasted debugging graphic that I love so much. Here’s a snapshot with the graphic and Harry Foster’s analysis from Mentor’s website. I’ve shown the graphic so many times already. Let’s focus on the analysis this time, shall we? Continue reading →

Poll: Unit Testing Abstract for DVCon 2014… Yay or Nay?

Posted on by nosnhojn

DVCon call for papers hit my inbox last week. This time around, to possibly save myself the effort, I figured it might be more productive to let other people decide whether or not mine is a topic they’d like to see at DVCon. I’ve got a draft title and abstract so far…

How UVM-1.1d Makes the Case For Unit Testing

In six weeks, two verification engineers discovered 10 defects in the UVM-1.1d release. The defects were not lurking in the shadows of complex corner scenarios; it didn’t take hours of constrained-random stimulus to discover them; nor was there any need for functional coverage with intelligent feedback. These were simple, obvious defects discovered with an equally simple and obvious technique. It’s called unit testing and it comes from software development where the value placed on code quality and maintainability is higher than what we aspire to in hardware.

UVM-UTest is an open-source project with the goal of demonstrating how unit testing can be used to bring software level quality and maintainability to design and testbench code. This paper documents the development of UVM-UTest, how it was conceived, statistics and style for unit tests written, its success in terms of defects found in UVM-1.1d, the lessons learned and recommendations for applying unit testing beyond the UVM. The paper concludes by introducing a new UVM component, the uvm_boat_anchor, that further demonstrates the perils of continuing to ignore the value of this basic yet entirely necessary technique.

I’ve submitted proposals twice in the last few years. Neither was accepted. Agile development and test-driven development – the topics I’ve submitted – aren’t your average mainstream, audience grabbing topics so that wasn’t overly surprising. Maybe this proposal is different. Or maybe it’s not. Either way, I’m leaving it for you to decide.

[poll id=”5″]

If there’s enough interest, I’ll submit it and see what happens. If you really like the abstract, share it with your colleagues so they can vote, too. Or if you hate it, may as well share it, anyway. I don’t mind people seeing my incredibly terrible ideas. I’ll just entertain myself with it.

Proposals are due Aug 27th so share the link and get your votes in asap!

-neil

UVM Is Not A Methodology (The TDD Remix)

Posted on by nosnhojn

Forgot about this in last week’s post! Another interesting question from the functional verification seminar I delivered in Mountain View a few weeks ago was: if you could only pick one or the other, would you rather use UVM or test-driven development? Of all the great questions that came up, that was by far the easiest to answer. My response…

I’d take TDD over UVM any day. No contest.

Continue reading →

The First Step is Acceptance (Hardware Verification is Broken)

Posted on by nosnhojn

A couple weeks ago, I had the chance to do a lunch-n-learn seminar for about 20 verification engineers in Mountain View. It was an hour talk about an incremental approach to functional verification; one that I’ve given about a half a dozen times to various teams.

I like giving this talk because I find people on the whole are surprisingly receptive to the ideas. There’s also been some skepticism, though, that I’ve seen more than once. Seeing as how it’s still fresh in my mind, I figured now would be a good time to pass some of this skepticism on as food-for-thought. Continue reading →

How UVM-1.1d Makes The Case for Unit Testing

Posted on by nosnhojn

The point of the open-source UVM-UTest project we’ve been working on is to demonstrate how unit tests can be used to lock down the functionality of legacy code. Being able to run the unit tests means that when you’re changing code you can verify existing features won’t break during maintenance. We think that’s valuable. A nice side benefit of the project is that we can also show unit testing as being effective for increasing the quality of code hardware engineers write. How can we say that? Because over a 5 week period we found 10 defects in the UVM 1.1d release. Continue reading →

Are You Ready for the UVM-UTest Challenge?

Posted on by nosnhojn

We did our first UVM-UTest challenge on Friday. It went surprisingly well so now we’re going out on a limb by inviting others to take the challenge.

The “rules” of the challenge are:

  • A team of 3 has one hour to break our unit tests (we used a lunch hour so it was like a lunch-n-learn… except it wasn’t boring).
  • If a team can make functional changes to the uvm-1.1d library that our tests do not detect, you win. If our unit tests detect the functional changes you make, the unit tests win. Pretty simple.
  • We guide you through the challenge so you know what code/classes you’re able to change (we also keep notes for where we can improve our test suite). Continue reading →

UVM-UTest is Ready for Download

Posted on by nosnhojn

After less than a month of development, the open-source UVM-UTest project is ready for download by the masses. If you care about code quality and maintainability, you’ll be interested in what we have.

For instructions on how to download and run UVM-UTest, take a look at the UVM-UTest Getting Started page. If this is the first you’ve heard of UVM-UTest, you may also want to check out the original UVM-UTest announcement to see what we’re up to. In that announcement you’ll see links to the project charter and progress page. Also be sure to check out The Most Descriptive UVM Tests Ever Written. Those and others like it are part of today’s release.

The next step for us comes this Friday at Noon EST when we have our first UVM-UTest challenge. That’s where we challenge some of our colleagues to find holes in our unit test suite. If they can break the UVM library without our unit tests detecting it, they win. If our unit tests detect all the changes they make, we win. We’ll keep you posted on how that turns out!

..and of course… if you’re interested in taking the challenge, we’d be happy to hear from you :)!

-neil

The Most Descriptive UVM Tests Ever Written

Posted on by nosnhojn

If you want to know how the uvm_printer works, in detail, I don’t think you’ll find a better explanation anywhere than my unit tests. This comes after I went through a refactoring exercise in our open-source UVM-UTest project – that’s where we’re unit testing the code in the UVM framework – specifically for the purposes of making my code more readable.

Here’s one of the tests I’ve written to get the idea of where I’m going:

Screen Shot 2013-05-28 at 4.02.11 PM

Continue reading →