By Example: Done vs. DONE

In a previous post, When Done Actually Means DONE, I shared a slide that I’ll present at Agile2011. I use it to illustrate the differences between waterfall and agile development models in the context of hardware development. After posting that, the first response I got was examples could maybe make it even clearer from AgileSoC guest blogger @rolfostergaard.

Thanks Rolf. Good idea.

In case you haven’t read When Done Actually Means DONE, I’ve included the slide I used to get things started again in this post. I use it to show that there are different ways to describe how done you are based on the development model you’re using. If you’re basing progress on tasks you’ve completed, you’re using done to measure progress. If you’re using features, you’re using DONE.

What’s the difference? Being done means you’ve hit a milestone that won’t hold water mainly because there’s no way to objectively measure its quality. You may think you’re DONE, but without tests or some other reasonable way to measure quality, you’ll very likely need to come back to it. For that reason, done is misleading and it gets people into trouble.

DONE means you’ve hit a milestone that you can unambiguously quantify (or at least quantify with far less ambiguity). Here, you’re confident that what you’ve just finished will require very little or no follow-up because you can see and measure results.

In short, done isn’t done at all… but DONE is. Confused? Here’s where a few examples might help.

My RTL Is Done

Classic. Your design team is under pressure to meet a scheduled RTL complete project milestone. As always, it’s a high visible milestone – to the development team, management and possibly even customer – because it comes with the connotation that the product is nearly finished… save for the minor details that it hasn’t been verified nor has it been pushed through the back-end. The RTL is done though, so that’s great. Cross it off the list!

My Test Environment Is Done

This is a close second to my RTL is done. Your verification team has finished its test environment and supposedly all there is left is writing and running tests. Though of course there’s been very little to confirm that the test environment does what it is supposed to do. That’s immediately obvious when running the first test: the configurations are invalid, the stimulus transactions are poorly formed, the BFMs don’t obey protocols and the model is outdated; all unfortunate because now people are anxiously expecting results that the environment can’t quite deliver yet! Sure the test environment is done… except for everything that doesn’t work.

Feature <blah> is DONE

Now we’re getting somewhere. No, your RTL isn’t done. No, your verification environment isn’t done. But who cares! You have something better: a small portion of both are DONE and that’s enough to run tests and collect coverage results that verify feature <blah> is ready to go. No ambiguity there. The feature works and you have the passing tests to prove it. You’re delivering something that’s DONE.

(Ideally, you would have passed the design to the physical team as well. But by the fact that you’ve made a big step forward in terms of credibility and confidence relative to the first two examples, we’ll forget about the physical design for now.)

The Software API Is Done

A hardware team normally implements an API according to a spec received from the software team. After the hardware team is done, it’s assumed that sometime later the software team will build drivers and applications on top of the API and release the finished product. Problem is, the initial API was a best guess from the software team and early in their development, the team finds that the API that’s been sitting done for a couple months doesn’t give them the access to the hardware that they need. Sure the API was done, but until it’s updated, system performance is seriously restricted and some functionality is completely absent.

The Software Demo Is DONE

An SoC, by definition, is a part hardware, part software solution. So why settle with an API that’s done when the software is required for delivery? As the hardware team completes API functionality, give it to the software team so they can actually use it. Deliver it as a C model, an emulation platform or some other form that makes sense. Use this demo version of the entire solution (hardware + software) to judge whether or not you’re DONE.

I’m Still 90% Done

I’ll end with a personal favorite that kind of fits into this discussion. Like everyone else, I’ve played this card many times. What does 90% done mean? It means you think you’re almost DONE but you really have no idea because there’s no way of knowing for sure. Before you say it again, do yourself a favor:

  1. admit that you don’t know how DONE you are
  2. find a way to measure what you think you are DONE
  3. isolate what you aren’t DONE

I’m going to try and follow my own advice on that one :).

Done is a false milestone. It’s ambiguous. It’s one dimensional; you may have written some code but that’s it. There’s no reliable way to measure done and teams that measure progress in terms of done eventually find they’re not as DONE as they thought.

DONE on the other hand, comes with results. It’s multi-dimensional; you’ve written some chunk of code and it’s been tested so you know it works. DONE is measured in passing simulations, software demos and any other means that objectively confirm the code you’ve written is high quality. Teams that measure progress in terms of DONE know how far they’ve come and how far they have to go.

Done is a feeling. DONE is progress.

-neil

Q. What examples of done do you see in SoC development?

When Done Actually Means DONE

In presentations I’ve given on agile hardware development, there’s one slide I have that seems to get the point across better than any other as far as how agile development differs from the waterfall-like process a lot of hardware teams follow. I’ve used it a few times before and I find myself counting on it again for my talk at Agile2011 in Salt Lake City.

In prior talks, I’ve built up to this slide with a verbal contrast between waterfall and agile. I talk about waterfall development as a sequential process that culminates with one big bang deliverable. Progress is tracked against a master schedule and based on what pre-defined tasks have been completed. The lessons learned at the end of a project are applied to the next project (i.e. you have a chance to improve once/project). I don’t claim that waterfall is exactly what hardware teams do, but it’s close’ish and the task based planning certainly does widely apply.

The agile model, on the other hand, is an iterative process where each iteration culminates in production ready code equivalent to some subset of the end product, meaning you can quit and release whenever you and your customer want. Progress is measured based on which features are complete at a point in time and lessons from each iteration are applied to the next (i.e. you have a chance to improve several times/project).

These are obvious differences, but the underlying message is never quite apparent until we get to the visual. That’s where people can see that the critical difference is in the definition of DONE.

Let’s say a team following a waterfall model is half done building an XYZ, which to the team means their design documentation is done, the design has been coded and they’ve built some of the verification infrastructure. Lot’s of good stuff there, but because the team has based its progress on predefined tasks they’ve completed as opposed to what they know actually works, half done can be pretty misleading. The documentation they have could be severely out-of-date, the design is probably somewhere between buggy and dead in the water and the verification environment might not even compile yet. Needless to say, the second half of the project is going to take much longer than the first half did!

Contrast that with the the definition of done in the agile model. Here, progress is based on features. When a team says “we’re half done”, they mean it. With little warning, they could release a product with half of the intended functionality. They know it’s done because it’s been tested, it’s been through the back-end, the software has been written and it’s ready to go.

Two different ways to measure progress; two very different meanings for the word DONE. To me, its this visual contrast with the waterfall and a redefinition of what it means to be DONE that helps the value of agile development really stand out.

neil

Q. How do you measure ‘done’? Code that’s written or code that’s production ready?

Regular Delivery… It’s Part Of Being Agile

In a post a few weeks ago I talked about how regular delivery wasn’t actually the point of being agile in hardware development. While the software guys can deliver on a weekly or monthly basis, taping out an ASIC every week doesn’t make much sense. Obviously. But even though regular delivery makes no sense for an ASIC doesn’t mean agile isn’t applicable. The value of agile in ASIC development is the ability to demonstrate progress. I believed that when I wrote it and I believe it now.

So why the contradiction in this post? Why am I now saying regularly delivery should be part of agile hardware development? While it might not make sense for ASICs, regular deliver is entirely possible with an FPGA or an IP block.

Before you shake your head, think about why agile software teams deliver product regularly. The goal for any software team is to satisfy customer need. Sounds simple enough except that sometimes customers don’t know what they need… so they ask for what they think they need or worse, what they want. If you (the development team) disappear for, say 9 months and then hit them with finished product, there’s a decent chance someone ends up being disappointed.

Development team: “All done! This is exactly what you asked for in this giant ambiguous requirements spec! I hope you love it!”

Customer: “I don’t love it because that’s not what I asked for.”… or… “Hmmm, I guess that’s what I asked for but what am I supposed to do with it.”… or… “Super, but I don’t need that anymore”.

Experience has shown agile software teams that keeping your customer involved in development helps avoid delivering a disappointment. Involved doesn’t mean telling, it means showing. Showing customers product as it’s being built helps them identify what they need and prioritize what they might not need. This works in software. There’s no reason to think it couldn’t work for an FPGA or an IP block.

“You mean we should be able to ship an FPGA load or an IP block every few weeks/months?”

Right.

“Even though it isn’t done?”

Yup.

“But we’d have to change our whole delivery process or we’d end up wasting all our time on packaging the thing instead of building it.”

Correct.

“…and we’d need automated regressions…”

Uh huh.

“…and we’d have to help them with integration so they know how to use it…”

Right again.

“…and we’d need user guides and documentation in place…”

Correct.

“…we’d probably need a reference design, maybe even a board ready way before we’d usually have one…”

Definitely.

[pause]

“…but what if there’s a problem with what we’ve built?”

That’s kind of the point.

With an ASIC, it’s not possible to incrementally deliver the real thing but there is still value in hardware simulation/co-simulation demos, emulation demos, intermediate size and power calculations, etc. It’s not perfect but it’s better than going the distance, sharing progress through status reports and hand waiving only to find disappointment when you’re done.

With FPGAs and IP it’s different. Regularly delivering product to your customer even though it isn’t done could end up being an invaluable learning experience. And unlike an ASIC, it’s entirely reasonable. Yes it could require major changes to how you develop and package your product but the customer feedback could be worth your while. Even if you only do it once, think of it as working out the kinks so there’s a better chance of bug free delivery when you are done.

Better to discover that what you’re building isn’t quite what your customers need in a month or 2 than funding a massive development effort and finding out months later.

Possible? Not possible? Ridiculous? Let me know what you think!

neil

Regular Delivery Isn’t Really The Point

Living the role of agile hardware development advocate has been a learning process for me. Obviously, I’ve learned a lot in terms of agile development process but that’s not all. I’ve also had a chance to speak with people seeing new ideas for the first time. That’s been very interesting for me.

Through talking with people, I learned quickly that people sit in 1 of 2 camps when you present the basics of agile hardware development. The first camp responds with “what a great idea… this makes a lot of sense”. The second camp responds with “this doesn’t make any sense at all… how the heck am I supposed to tape-out an ASIC every week/month/quarter?”.

To the first: I enjoy the long discussions we have. I’d say great ideas, opinions and concerns flow both ways. There’s always some skepticism (which there absolutely should be) but there’s a level of acceptance and that’s been encouraging.

To the second: this may seem a little odd… but discussions with you guys have become more valuable to me. I’ve found that a great way to learn is to have my opinions methodically dismantled by an all out pessimist. I get to hear all the reasons why I’m wrong in one short burst. Awesome! The argument against usually starts with how product and delivery are too different between software and hardware so agile can’t work. They might be able to deploy product weekly, but us taping out an ASIC several times is entirely ridiculous.

That discussion can go further but it’s the “product and delivery are too different” argument that allows people to dismiss agile out-of-hand.

Conveniently, I recently found that I’m not the only one bumping into that argument. In this article just published on eetimes (which is a good intro level article and not just for embedded software developers), James Grenning makes a very good point related to development and delivery of embedded software. It seems some in the embedded software world are using the same argument to dismiss the potential of agile:

Because value cannot be delivered each iteration some say that agile cannot be used on embedded software development. My opinion is different. Instead of delivering value, we provide visible progress. I don’t mean doing show and tell on what might be build, but rather a demonstration of real working running software.

Substitute “hardware” for “software” in that quote and I think it’s a decent response to the “product and delivery are too different” argument. Of course it’s absurd to think an ASIC should be delivered every week or every month. In fact that argument is so valid that it’s silly to get caught up discussing it. Instead, and just as James notes for embedded software, the potential for agile in hardware comes from regularly demonstrating progress instead of just describing it in documentation and discussing it in meetings.

Regularly demonstrating progress is where the discussion of agile in hardware development should be starting, regardless of what camp you’re in.

Neil