The Time It Takes To Find A Bug

How much time passes between writing a line of RTL or testbench code and knowing that it works. I know it varies, but on average how long would it take? A line of code is written, time passes, then a test is run that either verifies it’s correct or discovers a bug. Does that time pass as minutes? As days? Maybe weeks? Months?

I’m preparing for a talk at DAC next week in the Verification Academy booth and in going through my material I’ve started asking myself that question. It’s the 3rd year in a row the folks at Mentor have invited me for a session in the booth. The last 2 years have been a lot of fun and I’m hoping for the same this year. This year’s session is called Add Unit Testing To Your Verification Toolbelt. As the name suggests, the focus will be on how we can use unit testing and test-driven development to improve the quality of the hardware we build.

I believe quite strongly in the value of unit testing and test-driven development because the quality of what I produce as a verification engineer is noticeably better because of them. I believe so strongly in TDD in particular that I know I get a little preachy at times. It’s the most important engineering skill I’ve ever learned and I don’t shy away from telling people about it.

I always think of rigour first when it comes to benefits. Testing code as you write it a few lines at a time is a rigorous process to say the least. But during the run-up to DAC I’m starting to think it’s timing that’s the real key. With TDD, the amount of time that passes between writing and testing a line of code is minutes, sometimes less. A design and test cycle measured in minutes is almost too short a time to think about anything else. I focus on getting exactly 1 thing right. There’s no thinking back, no back-and-forth with a teammate that found a bug, flipping through my logbook for clues as to what I’ve done or editing bug reports. In fact, with TDD there’s hardly any context switching at all. I write a line or 2, test them, if they’re broken I fix them, then I move to the next line. Most of the bugs I create are killed immediately; which coincidentally is the best time to kill them.

I’ve seen a lot of people talk about how bugs become more costly to fix the longer they live. For me, TDD has been the most effective way to respond.

If you’re at DAC next week in Austin and using TDD to kill bugs fast sounds good – or great! – I hope to see you at Mentor’s Verification Academy Booth at 11am on Monday. It’ll be a half hour presentation with lots of time for discussion. We’ll talk about TDD, unit testing, SVUnit and all the frustration that magically vanishes because of them!

Here’s a link to the session info for mine and others in the booth on VerificationAcademy.com.

See you at DAC!

-neil

Definition of (De)Bug

In 9 years of blogging, I’m guessing at least half of what I’ve written centered around code quality or debug. For all the writing and time I’ve put into these topics, it took me until yesterday, for whatever reason, to realize I’m without a solid definition for debug. Or even a definition for bug for that matter. I guess I was relying on a common understanding of what bugs are and what it means to debug code. Realizing that may not be the case, I came up with a definition that captures what bugs are to me…

Bug: A state where a feature is defined, implemented and/or expected to fulfill a specific purpose but fails to do so.

Continue reading

I Can’t Hide From Unit Tests

There are times when I try and convince myself that unit testing my verification IP may not be necessary. Believe it or not, I can put together a pretty convincing argument. On a good day I shake it off and do the right thing. Other times I’m just convincing enough. I know it won’t end well, but I skip the unit tests and go straight to the code.

Here’s a couple arguments I’ve used on myself to get into trouble. Continue reading

Verilog/VHDL Mixed Language Unit Testing

 

SVUnit supports mixed language Verilog/VHDL unit testing.

Finally.

Download the Latest Version of SVUnit from GitHubIt didn’t take much to do it but for some reason it took me a loooooooong time to get at it. It involves a new switch to the runSVUnit command line. Users can now specify a ‘-m <vhdl file list>’ where vhdl file list is all their VHDL files and extra VHDL-related command line switches. It’s been tested with Modelsim and Incisive. Probably works with VCS though I can’t say for sure because I haven’t seen it with my own eyes. If someone wanted to give VCS a try and let me know what happens, I’d appreciate it!

The updated runSVUnit usage is (emphasis on -m|–mixedsim <vhdlfile>):

screen-shot-2016-09-14-at-9-01-56-pm

As an example, if you’re unit testing a mixed language design with Modelsim where VHDL files in your design are listed in myVHDL.f and your Verilog files are in myVerilog.f, your command line would be…

runSVUnit -s modelsim -m myVHDL.f -f myVerilog.f

As always, if there’s any trouble with it you can log an issue on github, send me an email at neil.johnson@agilesoc.com or leave something in the comments. If you were waiting on mixed language support to get started with SVUnit, now is your chance. Just scroll up and hit the big blue button to get started!

Happy unit testing!

-neil

Hardware Bugs Need to Die

I used to accept bugs as a part of what happens in hardware development. Start a new project, write a bunch of code, deal with the bugs that inevitably arise, stress out about whether I can fix them before development milestones, cross my fingers that regressions pass the day before tape-out, repeat.

After several years of acceptance I’ve started taking bugs personally. I hate bugs. They’re no longer acceptable and I’ve changed the way I write code to avoid the embarrassment of creating them. For me that means using SVUnit to unit test my testbench code as I write it.

Knowing that test-driven development and unit testing with SVUnit has improved my code in every way, that others have found the same and that it’s become an option for everyone, I’ve turned advocating for SVUnit into a bit of a hobby…


svunit-smokes

I’ve grown to attribute bugs not so much to developer ability or quality of specification or language or other commonly held positions but to the habits we rely on to write design and testbench code. Primarily, I’m talking our debug later approach to writing code. Write the RTL then debug the RTL; write the testbench then debug the testbench. We don’t think about it, we just do it. It’s a habit. It kills our productivity, just like cigarettes kill people.

While EDA reacts to long debug cycles by investing in new debug tools, the only way I see to improve quality is by adopting proactive coding techniques (test-driven development being one option that I personally use. There are others). But before we do anything, we need to realize that bugs are formed out of habit. Kick the habit and voila… fewer bugs. Continue reading

Leading the Discussion on Unit Testing

Screen Shot 2016-05-25 at 9.04.54 PMAbout a month has passed since the great unit testing discussion in the Verification Academy booth at DAC in Austin. It was my second year in the booth. Very grateful for having had the opportunity. It was a lot of fun!

The format involved me cycling through a bunch of questions and comments on unit testing as a series of topics during which time discussion would erupt with me and audience.

At least that was the plan! Continue reading

5 Steps to Unit Testing Success

Seeing unit testing catch on and flourish with a new team has made the last few months at work pretty fun for me. Getting to this point, though, has been a ton of work. Considering the journey toward unit testing can have a lot of twist, turns, surprises and disappointments, I figured it would be a good time to recap in hopes my experience helps others grow unit testing within their teams. Continue reading

Simple Clock and Reset Support in SVUnit

Screen Shot 2016-04-28 at 8.54.17 AMIn a post a couple weeks ago called Sorry Design Engineers, I Can Do Better, I told design engineers that I’d do a better job of giving them what they need to unit test their RTL. I felt like I’ve been neglecting design engineers and decided I needed to set a new, more inclusive course. Tonight I took a first step in that direction. I’ve released SVUnit 3.10 with a simple clock and reset utility that’ll make it easier for design engineers to unit test synchronous and asynchronous logic. I also recorded a 25min code demo for design engineers to show how they can use SVUnit to unit test RTL modules. Continue reading

SVUnit for Chip Leads and Managers

Advanced verification methods put chip leads and managers at a real disadvantage, especially when it comes to visibility and predictability. With long testbench development times, early progress in constrained random is essentially based on trust; being random, the inherent uncertainty makes predictable scheduling and release milestones rare. Continue reading