First is a blog posted on eetimes designline by Brian Bailey a couple weeks back called Enough of the sideshows – it’s time for some real advancement in functional verification! In that post, Brian exposes a few techniques – constrained random verification in particular – that have failed to live up to the hype and praise that’s been heaped upon them over the last several years. That’s an industry expert suggesting we re-think the direction we’re going with functional verification tools so if you haven’t read it, I’d suggest doing so. In case you missed it, Brian also inspired the snappy title of this post .

Second is the work I’ve been putting into SVUnit and test-driven development. I’ve been having some decent success doing TDD w/SVUnit, enough that it’s quickly becoming my favorite design technique when it comes to verification IP and testbench development. Others are using it successfully as well which gives me hope that someday soon TDD will go mainstream in hardware development.

Also motivating this was the recently announced UVM-Express from Mentor. UVM-Express is a step-by-step approach that I really like and have already written about here.

The final bit of motivation, I have to say, is my slight distaste for UVM and our collective struggle to cage the mythical beast called reusable verification IP. Realizing it is becoming industry standard and library of choice for those on the cutting edge I will use UVM… but only under protest.

<protest>UVM is HUGE and it’s complicated. It’s extensible to the Nth degree yet at the same time arbitrarily confining (sure you can add as many wacky runtime phases as you want, in any order that you want, but don’t even think of creating a component after the build phase or connecting a driver to anything but a sequencer. Just don’t). It’s biggest problem is that it sure doesn’t appear it was developed with the majority of its users in mind. Bleeding edge users… perhaps. The majority… not a chance.</protest>

Sorry for pointing that out. A little harsh perhaps because I don’t mind it now that I’m using it… though the ramp-up is incredibly steep. If I’m the only one that feels that way, go ahead and disagree in the comments.

Put all those things together and what do you get? I think that over the last decade we’ve “evolved” into forgetting two critical aspects of testing:

• early results are important
• it’s important to use the right tool for the job

Remember the graphic that shows coverage from constrained random verification relative to directed testing? The one that people have used – and unfortunately I have used a time or two – to explain that if we just wait a little longer (aka: spend more time building the testbench), at some point we’ll suddenly get an explosion of positive results; our coverage line will abruptly charge due north and then gently flatten out to 100%.

The flat line at the beginning of the constrained random curve is us convincing ourselves that early results aren’t important.

That same graphic was used to undermine directed testing as a viable technique for making progress. Sure you might use it to verify a few corner cases on your way out of the office the day before RTL freeze, but the big progress is made with constrained random. You’d be crazy not to believe that. Just look at the graphic!

Relegating directed testing to the corner-case-afterthoughts is us forgetting that there’s a right tool for every job.

That’s a pretty bleak assessment of how we’ve faired with cutting edge functional verification tools and techniques over the last decade or so. But I really believe it’s a fair assessment because things have gone off the rails a bit. The good news, though, is that some of the innovation that Brian Bailey is looking for in his article can be done, in my opinion, by rethinking the tools we have as opposed to waiting for what we don’t have. I think that gets done by reasserting the importance of results (i.e. passing tests) as a metric for progress and recognizing there’s a right tool for every job.

Let’s do that by looking at functional verification as a 3 step process.

Step 1: Fix The First Bugs

Basic sanity is a milestone that is so incredibly important for all development teams as it’s that first benchmark for progress. It requires little more than absolute simplicity which is something that teams devoted to constrained random verification have overlooked.

Don’t use a constrained random environment to fix the bugs required to verify basic sanity. Don’t worry about reuse when it comes to basic sanity. And don’t worry about rework. Build the simplest testbench you can with a short list of the simplest (regressionable) directed tests you can think of that’ll verify the simplest function of your DUT. Set a benchmark with basic sanity, then move on to step 2.

If you’re familiar with UVM-Express, think UVM-Express step 1 at this point; a verilog, UVM-less setup that’s perfectly suited to simple, directed tests useful for verification and design engineers.

Step 2: Fix The Bugs You Expect

With only your basic sanity test – and possibly some smoke testing from the designer(s) – it’s pretty safe to assume that the next feature you test will be broken. As will the feature you test after that… and the feature after that… and so on to the end of the feature list. For early code, focus is important. You need to be able to go to a designer and say “this feature isn’t working” and you need directed tests to be able to do that. Directed tests are still the right tool for the job for fixing the bugs you expect; constrained random tests are not.

To fix all the bugs you expect – which in all likelihood will apply to most of the features in your design – you might be thinking isn’t that going to be a lot of directed tests?? You’re right to think that. All those tests may seem like a lot of keyboard bashing but the focus and the short debug cycles are what’ll make this more productive that using constrained random.

Step 3: Fix The Bugs You Don’t Expect

Finally, as you get to testing the unforeseen and the unknown, constrained random tests become the right tool for the job. Here’s where you’re looking for the things you don’t expect; the insidious little bugs that constrained random testing was made for.

There’s three steps to 100% coverage using techniques we already have and applying them at a point where they add the greatest value. Now a few final thoughts related to reuse, reference models and code quality.

I’d recommend not being overly concerned with rework/reuse as you transition from step 1 to steps 2 and 3. I say that because it seems our infatuation with reuse at times pulls us away from the right tool for the job (i.e. why would I do something simple when I can do the same thing with something more complicated that’s poorly suited to what I’m trying to achieve?). And our penchant for extensibility bloats the code with rarely used features. It also adds delay. Reuse is important, but don’t be afraid to add features to the testbench as you need them and rework as necessary (yes… I realize suggesting rework is blasphemous for some but I think it can be extremely productive when properly contained). Incidentally, rework in the software world is called re-factoring and it’s seen as a necessary part of development.

Next is the reference model (and scoreboarding), which is necessary for constrained random tests. I suggest building your reference model incrementally as you build your directed tests in step 2. This kind of bite size progress is more manageable than holing up for a few weeks/months and doing it all in one shot. It’s also a twist on directed testing where the stimulus is directed but the checking is automated. Ideally, when you’re done step 2, your model is done so with your constrained random tests you’re worried about stimulus and coverage but not the checking.

Lastly, constrained random verification is senseless when applied to poor quality code… which unfortunately is what 99% of us write when we code open loop (i.e. without tests). Test-driven development brings the code-and-test feedback loop that’s sorely lacking in hardware development. I’m using TDD and I love it. The quality of the code I used to write compared to what I’m doing now isn’t even comparable. TDD works in software development. It’ll work in hardware development, too. It’s only a matter of time.

You can find more about TDD here.

I’m not sure if Brian had this type of innovation in mind when he wrote Enough of the sideshows…, but I think changing how we use the tools we already have is innovation that’ll take us much further than where we are now.

It’s also innovation that we’re all capable of.

-neil

Share/Bookmark

I don’t think the reusable verification IP we’ve been building is as reusable as we think it is. I don’t think reusable IP is reused as many times as we’d like (if at all). Nor do I think reusable IP is as valuable as we think it is.

There… I said it.

That hasn’t always been my opinion… in fact I used to be a person that preached the exact opposite… but that’s my opinion now and it feels good to get it out. Our entire idea of reuse is something I have a problem with because I think most of us are getting it completely wrong, myself included, for a very simple reason. In our haste to create IP that is reusable we’ve forgotten about the absolutely most fundamental requirement: the IP has to first be usable. Anyone can riddle a piece of IP with all the elaborate features and switches and hooks and callbacks and scripts necessary for infinite reusability (or extensibility). But in a lot of cases reusability ends up being inversely proportional to usability; the more requirements a piece of IP can fill the fewer it fills well.

Jack of all trades… and you know how the rest turns out.

If you’re working in hardware or embedded systems development, I’d appreciated you completing this survey.

If your code isn’t usable there’s no point in thinking it’ll be reusable so I’m going to suggest we step away from ‘reuse’ for the moment and focus all our attention on ‘use’. A simple way to do that is to look at building reusable IP as a two step process.

Step 1: Ensure you code is usable.

I think all IP should fill some pretty basic requirements related to usability:

• it works
• there are options for doing simple things in a simple way
• it takes 5 minutes or less to get the point across
• people only need to ask how to use it once
• to understand the usage model/api it takes nothing more than…
• a README and a short conversation for an expert; OR
• a README and a longer conversation and/or a whiteboard session for someone more junior to understand
• a full day training course (or longer) isn’t required to understand the usage model/api
• it takes a half a day or less to integrate (to the point where it becomes useful)
• it doesn’t unreasonably slow down compilation/simulation
• designers are willing to use it (or at least some part of it)
• designers don’t laugh, cringe or run the other way when you suggest they use it
• it doesn’t frustrate people to point they’re forced to build their own
• it’s easy to access
• it comes with examples
• it comes with a regression suite
• it takes 5min or less to figure out how to run the regression suite
• results of the regression suite are accessible and easy to interpret

Step 2: Ensure you code is reusable.

The tenets of reuse are already trumpeted regularly so I’ll only add a few general rules:

• Don’t make your code reusable if no one intends to reuse it
• Don’t impose a reuse model or framework on code that will never be reused
• Reusability is determined by the person doing the reusing

The upside is that reusable verification IP does actually exist so there is hope. There are teams that put out great IP. For the rest of us though, creating reusable IP isn’t any more likely than prancing through a meadow on a unicorn or funding your retirement with a leprechaun’s gold. And, unfortunately, we’re making it worse. Building reusable IP is hard enough; there’s no need to make it harder by forgetting the most important part.

‘Reuse’ without ‘use’ is just ‘re(diculous)’.

-neil

Q. Have a different opinion on reusable verification IP? Or any use/reuse requirements to add?

Share/Bookmark

The absolute best piece of information in that series comes via my new favorite hardware development graphic. You’ll find it in part 5 of Harry’s analysis… or since I’ve pasted it below, you’ll also see it by scrolling down a touch.

Why do I like that graphic? Because it very clearly illustrates the near ridiculous amount of time we spend debugging code. In fact, if you compare the debug and testbench development slices of that pie, you see that we spend more time debugging code (33%) than we do writing it (28%).

If you’re interested in participating in a study that’s different-but-similar, I’d appreciated you completing this survey.

That should be an alarm bell for hardware developers – and I think it probably was for the people that saw it – but what do we do about it?

Seems we could handle this in a couple different ways. The first way would be to make more productive use of our time spent debugging. If we’re better at debugging, we spend less time debugging. Sounds great but I have a problem with that. The bugs have to come from somewhere. Just a guess, but they probably come from the people writing the code. Which means that what’s not immediately obvious in my new favorite graphic on (wasted) effort spent in verification is that a good portion of the 28% of the time verification engineers spend developing a testbench – and to be fair, the comparable amount of time designers spend creating a design – is dedicated to…

…creating bugs.

So how about this for a different direction… instead of finding faster ways to write then fix buggy code, how about writing more robust code in the first place? Better code… fewer bugs… less time spent debugging.

Serious.

If you want to know how to prevent bugs from getting into your code in the first place, you’ll want to go back to our Test-Driven Development posts from November last year. Those start here.

Say no to debugging. Write better code.

-neil

Share/Bookmark

Project Planning in Hardware And Embedded Systems Development Survey

Are people in the hardware and embedded systems communities confident that their approach to project planning gives them a reasonable chance of success? We don’t know… but we’d like to.

We want you to know, too.

The survey is designed to measure confidence in the project planning approaches we use in hardware and embedded systems development. It is two pages of multiple choice questions covering tendencies in project planning and demographics. The survey takes roughly 5 minutes to complete and data is collected anonymously. Data and analysis will be published and freely/publicly available at the conclusion of the survey. The survey is being conducted by in partnership by me, Neil Johnson, and Catherine Louis of CLL-Group, LLC.

Project Planning in Hardware And Embedded Systems Development Survey

We’d also appreciate people’s efforts in spreading the word. Please help us by posting a link to the survey or to this page on Linkedin, Facebook, Twitter, your company intranet and/or anywhere else you go to share news and information with colleagues.

Thanks very much for your participation and thanks for helping make this survey a success!

For more information, you can contact me at neil.johnson@agilesoc.com.

-neil

Share/Bookmark

At UVM Express step 2, what I’m doing is building a layer of functionality atop the interface BFM that I’ve already built and talked about here. In short, step 1 involved building and testing the APB interface while the goal of step 2 is to build and test a monitor, coverage group and an agent that puts it all together. That means 3 new classes and corresponding unit tests.

I started step 2 by building the monitor. For the monitor, the BFM tasks I created in step 1 are used to passively capture APB bus transactions. I translate the captured bus transactions into apb_xaction objects and write them to an analysis port (the apb_xaction is a new class I derived from the uvm_seq_item). To build the monitor, I ended up writing 10 tests for verifying construction and connectivity, read transactions and write transactions. Having already unit tested the BFM, I could rely on the fact the BFM tasks were working properly. The new tests would focus on the how the pin-level fields were translated to objects and written to the analysis port.

Next was the coverage class. I derived the coverage class from a uvm_subscriber; inside it I declared a covergroup meant to capture a reasonable range of values for address, data and transaction kind (WRITE or READ). Doing TDD of the coverage class is the point where I exceeded what I thought was reasonable with SVUnit. I’ve never unit tested coverage groups – or even thought about unit testing coverage groups for that matter – so successfully building a coverage class using TDD was me surprising myself.

To build the covergroup tests, I alternated between sampling the covergroup and invoking the built-in query functions. Here’s an example:

In that snippet, you can see I’m verifying the sample method sets the sampled object properly as well as verifying 3 coverpoints for:

• the minimum address setting of 0×0
• the minimum data setting of 0×0
• and the kind setting of WRITE

Querying the coverpoint, then sampling, then querying again verifies the coverage number goes from 0 to 100 – in the case of addr_min_cp and data_min_cp – or from 0 to 50 – in the case of kind_cp – as I expect.

I continue this query/sample/query sequence through the rest of the address settings that I’m interested in. Here’s another test that shows how I test the coverpoint with bins between min and max. You can see that it isn’t a simple 0% to 100% transition for the addr_bins_cp coverpoint because there are 16 bins. For the addr_bins_cp I need to calculate the expected percentages that creep up by 1/16 for every bin sampled.

Final test for the address is similar to the first but covers the maximum address.

That wraps up the address related coverpoints. I have tests that similarly go through the data and kind fields. In all I have 6 tests for the coverage class.

That brings us to the last item in the list: the coverage agent. Agents were another type of class where I had initially dismissed the value of creating a unit test. This example changed my mind. The agent testing isn’t complicated but it is valuable in that it verifies connectivity between its internals. Thinking back to the number of connectivity issues I’ve chased down in my day, I reckon unit testing agents and similar container-like components is a low risk-high reward activity.

Here’s a test that verifies the contents of the agent are built…

Here’s the test that that verifies end-to-end connectivity by writing a bus transaction and then verifying the expected coverage results…

With only those 2 tests, I’m able to verify connectivity and functionality of the entire coverage agent.

To summarize, using SVUnit to do TDD of the monitor, coverage class and coverage agent involved me creating a total of 18 tests. I did everything in one evening (roughly 4 hours). With the additions I explain in this post, the UVM Express example I started last week is complete up to step 2. Considering the time I spent on steps 1 and 2 so far, my opinion is that by doing TDD and taking the time to build unit tests, I’ve easily saved myself a few hours of debug time had I instead just written the code and dumped it on to somebody else.

It appears as though all the good things I’ve heard about TDD are coming true. And I have further reinforcement that UVM Express is a decent approach to building verification IP.

This entire UVM Express example is available to SVUnit early adopters. If you’re interested in becoming a SVUnit early adopter, let me know at neil.johnson@agilesoc.com!

Stay tuned for step 3 and the addition of random stimulus!

-neil

Share/Bookmark

What is the definition of the DoR?  Similar to the DoD, DoR is the criteria applied to every Product Story (aka feature or requirement) before the story is accepted into your workflow (or iteration).   It is the hurdle that every story must get over before the engineering team should add it to their todo list (e.g., sprint backlog if you’re using scrum, or your Kanban board). The DoR can be applied as the input criteria before your implementation.

Simply put, every product story should follow a simple three state process:

1. a feature is considered from the Backlog, but must pass all (or most) of the items in your DoR,
2. the feature is implemented and verified, and finally
3. must pass all of the items in your DoD.

The entire cross-functional team is responsible for creating (and presumably agreeing to adherence to) the DoR.  As with all things ‘agile’, the definition is permitted to be refined and adjusted throughout the project e.g., discussed during a Sprint Retrospective.  As well, the formality of a DoR can range from a checklist that must be strictly adhered to, to the “back of the napkin” / whiteboard session to indicate that the story is ready to proceed to the next phase.   It’s up to the team to decide what is appropriate.

While it’s true that this is another step in your process, it should be a step that is considered to help accelerate the implementation rather than slowing it down.  Having the entire team accept a DoR ensures that you are increasing your chances for success because everything you’ll need to complete the work is lined up (or is being lined up) to assist the completion.

While the specific elements of a DoR will likely be different for every team and project, there are a couple of common elements that I think a DoR should contain:

• What:  Is the goal or end-result of the story clearly defined?  i.e., is the expected behaviour or coverage goal as clear an unambiguous as possible at this stage of development?
• How: Have you considered how to build the feature?  It doesn’t have to be much, but some thought about how to build and integrate the product story into the existing working product.
• Who / Resources:  Are all the necessary resources (i.e., people, tools, IT infrastructure) available and committed to the product story to have a reasonable chance of completing within a reasonable amount of time?

What would I include in a DoR?  I like to see the following:

• Is the expected behaviour as clearly defined and understood as possible?  While this clearly leads into defining some acceptance criteria and is a prerequisite for the Definition of Done,  at this point in our development we simply need know that the behaviour expected is as clear as possible.
• Have I given some thought on how it could be implemented?  I prefer creating a quick UML class and sequence diagram of how the pieces work together.  My key goal here is to define what impacts will this new feature have on the existing work.
• Should I create a short document with the design?  Since I use UML I like to create a short document that glues the UML diagrams together.
• Do I have a realistic handle on how long it is going to take to build?  It won’t be right, but it needs to be a reasonable estimate given all that I know.
• Do I have all the resources I need to complete this task?
• Can I see any dependencies that would block me from starting or completing? e.g., will I need to complete some other task? or wait on someone else? or some script is required before I start this product story.
• Does everyone agree?  Getting agreement is an important part of this checklist.

My last point is your DoR should be viewed as an implicit agreement that should not be followed dogmatically.  Commitments made during the discussion may change even while your implementing the product story — that’s the nature of the work we do… priorities change frequently.  To summarize: use the DoR to get all your ‘ducks in a row’, but to stretch the metaphor, don’t expect the ducks to continue to walk in a straight line.

What would you include in your DoR?

Share/Bookmark

Here’s a few excerpts from the UVM Express page on Mentor’s verification academy with some additional commentary:

The UVM Express is a collection of techniques, coding styles and UVM usages that are designed to increase the productivity of functional verification. The techniques include raising the abstraction level of tests, writing tests using BFM function and task calls, adding functional coverage, and adding constrained-random stimulus generation.

Seasoned (aka: skeptical) verification engineers that have seen their share of new product announcements promising “increased productivity in functional verification” and suggesting “raising the level of abstraction” might be tapping the back button by this point but I’d encourage those skeptics to read on. I think the “revelations” appear in the next few sentences.

If you don’t have a full-time verification expert on staff, or if you are not a full-time verification engineer, UVM Express might be for you. Most verification teams do not have a full-time verification expert on staff, have time and budget restrictions and cannot adopt the UVM in whole or adopt it as quickly as they might like. These teams are usually under-staffed, under-funded and over-worked. They are exactly the kind of people that the UVM is meant to help, but the first step towards adoption is too high.

The UVM Express can be thought of as a series of steps or train stops on a journey that may ultimately lead to Full UVM.

That, to me, is an all important recognition that while UVM is great new technology for functional verification, adopting UVM and actually realizing the benefits is extremely challenging.

In reading on you’ll see that UVM Express is a 3 step approach for building BFMs, coverage agents and stimulus agents with UVM. It’s not an all or nothing thing which means there’s value in going 1 step, 2 steps, 3 steps or further to “Full UVM” adoption. It also doesn’t touch all aspects of building a testbench (there’s still modeling and scoreboarding which UVM Express doesn’t touch); it focuses on a few important aspects of testbench development and leaves the rest, presumably, for later. A 3 step approach means you can use UVM Express to encourage people to master one particular expertise before moving on. I like that because UVM tends to be a giant wave of classes, macros, global variables, under-the-hood control and hidden optimizations that easily overwhelms newbies. Seems UVM Express could reduce that giant wave into a more manageable series of ripples. There’s more to it obviously, so I’d encourage people read through the material and watch the videos Mentor has posted to get the idea.

Unfortunately, the one thing that Mentor doesn’t explicitely say in their UVM Express collateral (maybe I missed it) is that UVM Express takes steps towards restoring a sense of inclusiveness within front-end development that has been steadily crumbling over the last 10 or so years.

It used to be that designers and verifiers all wrote code in the same language and used simple directed testing techniques. Then, in around 2000, came the invention of HVLs and release of verification libraries like eRM, RVM, VMM, AVM, OVM and now UVM. The upside of that transition was increased productivity for verification engineers, provided they could successfully climb the learning curve. The downside, however, is that HVLs and new verification libraries – not mention constrained random w/functional coverage, assertion libraries, formal and others – quickly became a wedge that’s now been driving design and verification engineers apart for more than a decade. In my opinion, UVM has the potential to become the biggest wedge of all. Left unchecked, not only will it drive designers and verifiers further apart, it will also further the gap between novice and expert verifiers.

UVM Express seems to be a baby step toward restoring the inclusiveness we had when everyone wrote simple Verilog or VHDL. That’s huge and I don’t think the importance of a restored sense of inclusiveness can be overstated, especially when you consider the newest gap in our development process between hardware and software engineers (we’ll leave that for another time ). That’s really why I like UVM Express and hope that similar ideas are on the horizon.

For teams that are interested in agile development, UVM Express should be of particular interest. The method of incremental UVM adoption plays very nicely with the idea of incremental development. For example, a first product increment might require implementing a small subset of design features, completion of UVM Express step 1 (aka: the BFM step), creation of a simple test harness and a short list of directed tests. With that, a team can sanitize the design without UVM and designers would have a testbench that they’d be comfortable with, just like the old days. Subsequent increments could mean either expanding the functionality of this basic arrangement or, if necessary, moving on to UVM Express steps 2, 3 and beyond.

Very reasonable idea, no?

Another thing to mention is that UVM Express still does not address the issue of quality and robustness of verification IP (not a slight against UVM Express… it’s not meant to do that). Verifiers don’t normally do much in terms of testing-the-testbench so initial testbench quality tends to be pretty bad. I think test-driven development (TDD) is the answer there and I think TDD plays very nicely with the focus and incremental approach that UVM Express suggests. If you read my last post or follow me on Twitter (@nosnhojn), you’ll know that I’ve already started building examples for how one would pair TDD, SVUnit and UVM Express. I’ve described how I went through the BFM step in my last post. You can expect more examples that describe the addition of coverage and random stimulus as I get through them.

To finish up, I’d recommend people take a look at UVM Express and form their own opinion while thinking about how it can be used in agile development as you go. I’d also recommend Mentor and others carry on from UVM Express. Accessibility to UVM and a battered sense of inclusiveness in hardware development have become a big problem that’s only getting worse. Seems to be a sliver of daylight on the horizon though so I get the feeling it’s not too late to turn things around.

Hopefully lots of exciting stuff to come!

-neil

Q. What ideas out there am I missing that improve accessibility to UVM and foster inclusiveness in hardware development?

Share/Bookmark

This particular example involves development of an APB master BFM. APB isn’t the most complicated of bus protocols but it’s a very good subject for an example like this because the code and tests are easy to understand (there’s an APB example in the latest UVM release also, presumably for the same reason).

UPDATE: for people looking at UVM Express that was announced by Mentor Graphics on Feb 22, this is my interpretation of how TDD can be applied at the lowest layer development of the BFM. Examples showing the addition of functional coverage and completion of an agent that includes sequence generation are still to come. You can see UVM Express on the Mentor website by going here.

In terms of requirements and documentation for our BFM, I’ll describe what’s going on with a couple of user stories:

There’s 2 stories that aptly describe this APB master verification IP. The ability to write/read memory of a peripheral and that’s pretty much it.

Here’s the acceptance criteria for both stories:

Because there’s 101 ways to build an APB BFM, the acceptance criteria here is important for narrowing things down. We’re looking for something equally useful for designers and verifiers so we’re not going to use anything fancy like UVM and we’re not going to use a transaction interface or anything constrained-random-like (which always seems to be the impulse of verification engineers). We’re just going to have IP with a simple method user interface on 1 side and pins on the other that could be connected to a peripheral. Arguably, a systemverilog interface is best for this so that’s the direction I went.

I was missing some necessary features for SVUnit so before I got started, I had to add support for generating unit test templates for systemverilog interfaces. I based the template off what I had already done for the module unit test template so it didn’t take long to get that done. With 1 test to verify the template output and 2 others to verify it compiled and simulated in both Questa and VCS, I could move on to the actual example.

The first tests I added to the unit test template were synchronous and asynchronous reset tests. No, they weren’t present on a story card but I thought this would be a good place to start anyway because they’re useful and would require all the pins of the interface but next to no functionality. Here’s one of those tests (fyi… `MST_EXPECT is a macro that checks each of the output ports on the interface. I expect all to be 0 after the reset):

The first time I ran this test, it failed – as I expected – in compilation because none of the pins nor the reset method existed in the interface yet. I added the pins and an empty method and ran it again. That ended in a runtime failure – again as I expected – because the reset method was empty and my FAIL_IF assertions were firing (here, FAIL_IF is hidden in the MST_EXPECT but you’ll see it in the next test). Add the reset functionality in the interface method, run the test a 3rd time and voila… my first passing test and my first complete loop through the TDD cycle:

Add a test, confirm the test fails, add just enough design code, confirm the test passes.

The next test I wrote had to do with testing the functionality of the select pin for a write, which takes me into implementation of our first user story. Here’s that test:

You see here that I’m calling the write method on the interface and in parallel I’m checking the functionality of the select pin. Running this the first time ends of course with a compilation error because the write method doesn’t exist yet. I add an empty write method, run it again and see the failures from my FAIL_IF assertions again… which is exactly where I want to be. Now I add the code required to control the select. That means adding a state machine that takes the interface from IDLE to SETUP to ENABLE and back to IDLE while the select pin is active in SETUP and ENABLE but inactive in IDLE. Here’s the code I added to do that:

You’ll notice that my definition of just enough includes the write method, the FSM, the logic to control the select for a single write and some synchronization. I have nothing more than that because that’s all I need to pass my test. Run SVUnit again and now I have 2 passing tests: I know I can reset my interface and I know the select works properly for a single write.

One last example to drive it home… the next test I wrote was still focused on the select pin but was verifying it would remain active for 4 cycles in the case of back-to-back write transactions. Here’s the test:

That failed because the BFM couldn’t yet handle the transition from ENABLE back to SELECT. Here’s the update to the BFM that I had to make to pass the test:

Several more baby steps through the TDD cycle eventually led to a completed ABP BFM. Instead of boring you with the rest of the code, I’ll leave you with some statistics of what I ended up with after I was finished.

• My APB BFM ended up being ~200 lines long
• It includes methods for writing and reading that designers and verifiers would both find easy to use
• I ended up with 26 tests in a unit test template that was ~550 lines long
• I had 2 tests for each port on the interface to verify it behaved correctly for single read/write transactions and back-to-back read/write transactions
• SVUnit tells me in about 4 seconds (of which about 3.5 sec is spent in compilation) whether or not my APB BFM is working properly
• Including the addition of interface support to SVUnit, this whole thing took me about 5 hours… from 8pm to 1:30am with a half hour break for tea at midnight

I’ll summarize by saying that as a beginner, I still find it a little tricky to get into TDD mode where I think in terms of tests first. Once I’m there though, I find it’s a very practical way to write code. And the best part is that you know the code is working as you write it as opposed to debugging it after the fact.

If you’re interested in seeing and running the results, this APB BFM example is currently available to SVUnit early adopters in the examples directory. If you’re not an SVUnit early adopter but you’re interested in becoming one, you can get a hold of me at neil.johnson@agilesoc.com.

-neil

Share/Bookmark

In the 3rd installment of the SVUnit Demo Series, I take people through a simple – yet complete – example of what’s required to test a UVM component within the SVUnit framework (if you haven’t seen the video yet, I’d recommend watching it here before reading on). The example I put together, and more importantly the plumbing under the hood required to make it work took me quite a while to put together for reasons that I don’t really get into in the video… but I will talk a bit about them here.

The usage model for SVUnit involves sequentially running a series of classes or modules through a corresponding list of unit test methods. In UVM however, due to the tight coupling between the phase methods in all uvm components and the instance of the uvm_root that ultimately drives the invocation of each phase method, uvm components are run in parallel.

One usage model wanting to be sequential and the other coded for parallelism gives us two models that are fundamentally at odds… so for unit testing UVM components, I had to get creative .

I wouldn’t consider myself a UVM expert so the best way forward wasn’t immediately obvious. First I attempted to disable the components I wasn’t interested in by temporarily removing them from the component hierarchy. That was a dead end however because of the local access restrictions. My second idea was to replace components with a dud that effectively had no implementation… which means if the dud ran in place of an actual component, nothing would end up happening. That too was a dead end because of the same local access restrictions of the component hierarchy (in hindsight I should have known right away this was a no-go but I was still learning!).

The last implementation that ended up working involved creating and adding a new uvm_domain that ran in parallel with the existing UVM domain (where the run phases are stored). In the video, I talk about idle components that do nothing and a single unit under test that is driven through the run phases, iteratively if necessary. The idle components end up being idle because the newly created domain they’re assigned to – after of course going through the common phases of build, connect, end of elaboration and start of simulation – ends up raising an objection in the pre_reset_phase. That objection effectively means that components assigned to the domain never advance further. That’s what idle means… never advancing further than pre_reset.

I was pleasantly surprised to see that components could be assigned to different domains at any time which means I could activate and deactivate components whenever I wanted simply by changing which domain they’re assigned to.

When a component is deactivated, it is assigned to the new idle domain by…

function void svunit_deactivate_uvm_component(uvm_component c);
  c.set_domain(svunit_idle_uvm_domain::get_svunit_domain(), 1);
endfunction

Similarly, where I show a component being activated, it is being assigned to the normal uvm domain by…

function void svunit_activate_uvm_component(uvm_component c);
  c.set_domain(uvm_domain::get_uvm_domain(), 1);
endfunction

That seems to be the trick to being able to run uvm components sequentially in SVUnit without adding any new control or requirements to the components themselves, something I was looking to avoid from the outset.

If you want to take a closer look at the code, you can become an SVUnit project member and early adopter by getting a hold of me at neil.johnson@agilesoc.com.

Bonus marks go to the person that looks at the code and can suggest a better way of doing the same thing!

-neil

Share/Bookmark

We need to find a way to work smarter (not harder).

The ultimate bit of useless advice… work smarter, not harder – or some variation thereof.

Thanks. Will do.

From managers (sorry for picking on managers but I think they’ll probably agree with me on this), the “work smarter not harder” imperative never seems to have the intended effect. It comes across more like “maybe if you thought about things a little bit we’d be done by now, we’d be getting rich off our stock options and instead of wasting your evenings in the office, you could be out waterskiing behind your new boat or golfing in Hawaii”.

What’s wrong with “work smarter”? It’s vague, there’s no method or actual goal, it devalues the hard work you are putting in, it always seems to come at a time of intense stress, and it injects panic instead of hope. Not exactly helpful. It is logical advice though and for some reason, I spent my weekend thinking about options for work smarter, not harder that teams can use to challenge themselves before the steady 12 hour work days kick in.

Here’s what I came up with…

• What would we have to do to cut our tools budget by 50% while maintaining our productivity and quality?
• How could we cut our regression times in half without increasing the size of our server farm?
• How could we get 5 days worth of work done in 4 days?
• How would we organize ourselves if we had no one to manage us?
• How could we limit the number of known defects to 1 or less (where less == 0)?
• How could we cut the size of our product by 25% without losing customers?
• How could we condense our design documentation by 75%?
• How could we lower our defect rate by 90%?

You’ll notice that these are pretty aggressive ideas and that’s on purpose. Aggressive goals like these would force teams to think outside the box (there’s another useful cliche ) to find new/interesting/revolutionary changes to the way they work instead of looking for the evolutionary optimizations that never seem to get us much further ahead. These ideas are probably a good way to recognize the wasted time and effort in your development process that you’ve been accepting without realizing it.

So the challenge is… get your team together, pick a seemingly outrageous goal from the list I have or think of one on your own, brainstorm a list of ways to make it happen (no idea is too stupid) and then go do it.

Small optimizations are not an option. Go big or go home .

-neil

Q. What “work smarter” goals would you add to the list? How would you (or have you) reached those goals?

Share/Bookmark