Looking for a job? Perhaps you are fresh out of a college degree in digital design and looking for a digital design job. Perhaps you are mid-way through college with a little bit of RTL experience and just looking for an internship. Perhaps you are a single individual fresh off of your last contract, looking for a new one. Either way, you are looking for something new.

If so, read on. I’d like to share a bit of my perspective from the other side, from the side of trying to find someone with your skills.

You see, my time is currently oversubscribed. This is one of the problems with success–there’s never enough of you to go around. The problem is, I’m just a one man shop. I’ve avoided needing to learn labor and tax laws by not hiring. While I’m not opposed to expanding Gisselquist Technology, I’m just not in a position to do so today.

Recognizing this, one of my clients has been trying to find and hire talent. He’d like to find others that can do the work that I do. Indeed, good talent can be hard to come by. The problem this client has is that he doesn’t have the ability to do digital logic design, nor does he know how to recognize it when he sees it. Therefore, he has asked me more than once to review the people he finds to know if they are worth hiring.

Recently, he found an Indian man, fresh from college, who was offering his services as a “digital design engineer”. So, my client asked me, can I offer him a project in “digital design” that we could give to him whereby he might demonstrate his ability? It should be something simple, but something that will help us know whether to hire him for more significant work or not.

I should point out, this isn’t the first time I’ve been asked that question. In the past, I’ve used my formal verification courseware to interview students. I’d give them one of the problems from the course, and then see how well they would solve the problem. If they weren’t prepared to do that, then I’d teach a lesson or two to see how well they pick up the material. Those that pick up the material quickly are good material for hiring, those that don’t will get a gentle, “No, thank you”.

That takes about 2-4 hrs of my time, however.

This time we tried another approach. This time I offered this budding digital designer a project to demonstrate his ability with. I suggested that he might try building the projects outlined in the (draft) first four lessons of my intermediate design tutorial. My thought was this: the tutorial outlines a project, tells you how to build it, simulate it, and verify it. To spare you the time of typing, the tutorial also comes with broken examples that need to be fixed. I reason that, if you can fix the bugs within those designs, then you must know something about digital design.

Yes, perhaps this was a selfish proposition. I haven’t had the time of day to work on the tutorial recently, and these first four lessons are just about done save that the logic within them needs to be tested on actual hardware. They make for a nice, well defined problem set to give to a new person to whom I’m not sure if I want to continue working with or not (yet).

Consider it from my perspective: managing takes time. Defining a project for someone to work on takes time. Real life problems don’t come ready built, ready defined, ready packaged for someone to just pick up and solve. Further, would I really trust someone new with a real life problem before they had proved themselves? This project, on the other hand, contained problems that were already well defined, and so it looked like an ideal task. On top of that, my client was offering to pay this individual to do the work.

It sounds like a good proposition for any newbie: You can get paid for doing some work immediately. If you do well, more work will follow. If you don’t, then you can still walk away with both what you’ve learned as well as the cash you were given to do the problem. That sounds to me like a good deal, no?

The response from this individual rather shocked me. He said, sorry, but I’m looking for a design job, not a verification job.

I bit my tongue on my first response.

My next response would’ve been a simple, “Thanks, but no thanks.” It’s not that wouldn’t be interested in a well qualified digital design engineer. It’s just that I’m not going to recommend anyone who demonstrates an attitude problem before his first day on the job.

My problem, however, was that this applicant wasn’t really mine to reject, and so decided I should soften my words and encourage him to rethink his position.

Perhaps he didn’t realize that he was being offered to prove his design ability by verifying and then fixing a broken design?

The following is copied from my response to him, with a few minor edits along the way.

Digital design is cost driven

Every industry is driven by costs. In the world of digital logic design, the most expensive thing you can do is to debug something in hardware. When working with FPGAs, it can take 5-15 minutes to generate a build, and another 5 minutes to run a test (on a good day–some designs require 2hrs to build …). Once you run that test, you might only get to see between 15 and 100 signals and those for only about a thousand clock ticks or so. That is, you might find the bug if you were lucky and just happened to be both looking for it and looking in the right place. If those signals don’t reveal your bug, you’ll have to iterate and repeat–assuming the problem is repeatable. The process is very expensive and painful. You can spend months on a project stuck not knowing what’s wrong with it. I call this problem FPGA Hell. Sadly, people get stuck in FPGA Hell quite often.

The problem is worse in ASIC design. (Yes, I’m now involved in that too …) If you have to debug an ASIC, you’ll have already wasted several millions of dollars to get there. It’s a bad place to be. Sometimes I’ve called this ASIC Hell. Unlike FPGA Hell, ASIC Hell is much hotter by at least two orders of magnitude.

To make this task easier, the digital design industry uses simulations. Lots of dollars and hours are spent on simulating designs. On a good day, a simulation will reveal problems. In many of the cases I’ve been dealing with recently, a simulation will take 20 minutes to run and generate 200GB of data which then need to be sorted through. Unlike running in hardware, simulations are able to provide you with every signal taking place within the design. However, that leads to a large data handling requirement that can make debugging simulation results a real challenge.

Simulations are great — when they find bugs.

One of the realities ingrained in the history and culture of Gisselquist Technology is that simulations rarely find the critical bugs. Hence, for all the work you put into your simulation, the simulation isn’t complete enough to find all your bugs. Many of the studies we’ve done have revealed bugs that pass the simulation tests that still end up needing to be tested in hardware. Worse, I’ve had several designs pass hardware testing only to be found to have latent bugs remaining within them at a much later time.

It’s not just me either. Xilinx is one of a small number of industry leaders in FPGA design techniques, and yet I’ve been tracking bugs in Xilinx’s IP for years. Some of these bugs have existed since 2016 and remain even now in Vivado 2020.2. Others have passed Xilinx’s “best practices” quality assurance testing. They were checked via the best simulation tests, and yet the bugs still made it through. I found them in short order via the use of formal methods while testing out SymbiYosys. (Why Xilinx? Because I use their hardware a lot. A quick look at Intel reveals some of the same types of bugs …) Indeed, AXI interface bugs are unfortunately quite common in the FPGA industry.

About myself

But let me now back up and tell you some about myself, because that might help you understand my own perspective when it comes to looking at job applicants. I learned engineering as an officer serving in the US Air Force. Towards the end of my time in the service, I picked up Verilog. That was roughly in 2008-2009, when I built two Verilog projects. Like any Air Force officer, I was also an engineering manager responsible for managing and reviewing many projects.

I then left the service and formed Gisselquist Technology, LLC, in 2013. My initial business efforts weren’t well focused, so it wasn’t until 2015 when I finally decided that I wanted to focus on digital design and then started building several projects to use as a portfolio. As a microbusiness of one trying to find work, I had no choice but to verify my own designs. There was no separation between being a design engineer and being a verification engineer within Gisselquist Technology. The two tasks were just different aspects of the same job.

This continued for about two years, bringing me to 2017. That year, I was still trying to find business and so I had started a blog. During this early time of my blog, I was asked to try out formal methods. Seriously? I wasn’t interested. I wanted to do digital design, not play with new fangled toys. However, I needed business and testing out a formal verification tool might make a good blog article. So … I tested out a new formal verification tool known as SymbiYosys. Much to my dismay, it quickly found bugs in a design I had used for years..

I was shocked. No one wants to hire a digital designer who produces buggy code. No one. The costs of finding and fixing bugs are just too expensive to deal with. So, I reasoned, why would anyone be interested in a digital designer, if his design portfolio was filled with bugs? What if someone randomly sampled my work and quickly found bugs within it? I would quickly lose that potential customer–they’d look for someone else who could actually do the job. So I started using formal methods on all of the designs in my portfolio–from the smallest to the greatest. Sadly, using the formal tools, I found bugs in every single design. I even found bugs in my flagship design, the ZipCPU.

That’s when I started getting seriously involved in formal verification. The more bugs I found, the more I liked formal methods, and the more I got used to using formal methods and the better I became–both in using formal methods and in digital design.

The ZipCPU Tutorial

There’s another part to this story. When I got started in 2015, I used various online forums as a means of trying to become known. I spent a lot of time on the OpenCores forum as well as Digilent’s forums. This placed me in a position to listen to a lot of students who were trying to accomplish final design projects. I quickly learned that these students were being taught the language to express a design and a scripted simulation, but not the tools they would need to actually debug their designs. As a rule, those students I came across were routinely struggling with bugs in hardware that they didn’t know how to debug.

Fig 1. Learning approach: start with all three in parallel

So, once I got started with formal methods and discovered how easy it was to debug a design using them, I wrote a tutorial that new students could use to learn Verilog. This tutorial started from the ground up teaching both Verilog, Verilator based simulation, and formal methods. This tutorial has now been well received by many.

I’ve since tried putting an intermediate tutorial together.

I haven’t gotten nearly as far with it, as you may have noticed. There’s clearly a demand for it it’s just that …

I’m now over booked with too many projects on my plate.

The key to finding work was, among other things, formally verifying all of my designs. No, I don’t use a separate verification team. I don’t have the cash to afford one. In this world of small business, digital design engineers need to do their own verification. It’s not just small businesses either–I know of big businesses as well that have been forced to cut their verification teams in order to cut costs. It’s in this environment that I’ve been doing digital design for hire. My customers depend upon my work being formally verified and they trust me to do it. I also try to maintain a reputation for quality. That quality comes at the cost of formally verifying all of my work.

That should provide you with a sufficient technical background to know why the ability to verify your own logic is valuable. I wouldn’t be where I am today without that ability.

Prima Donnas

There’s one more thing you need to know. Back in my time working as a Lt. Col. for the US Air Force, I quickly learned that some of the hardest people to manage are technically trained individuals. They like to believe they know what’s best. They know it so well, they tend to do what they think is right over and above what they are told. They’ll be the one’s telling the boss how the boss needs to do his business. If you don’t like it, they’ll threaten to walk out on you. We had a name for people like this. We called them “Prima Donnas”.

Prima Donnas are really hard to work with. They are very hard to manage as well.

I’ve struggled with my share of Prima Donnas over the years–both with those who have worked for me and those who have worked with me on my team. If you give these individuals a task, they’ll often work on something else, and then they’ll expect you to value their other work better than the paid work that’s actually bringing the money in. (But … who then does the work that’s being paid for?) Handling them is a difficult management task, and one that’s hard for the manager to win. This makes choosing people that much more challenging. It’s also why you put that much more effort into finding the best people–not just those who are technically skilled, but also those willing to work on whatever project they are given.

The Bottom line

To put it simply, here are my bottom lines:

  • If you aren’t interested in learning and using formal methods, then my recommendation to anyone hiring will be that they should find someone else. You would be missing out, however, on some of the opportunities that have come up to work on next generation digital designs: new CPUs, new memory controllers, imaging systems, next generation SONAR and Radar systems and more.

    No, you don’t have to take my course in formal methods, nor am I trying to convince you to sign up. Indeed, I might turn you away if you contact me today. You can, however, check out the course material. The course slides are posted, as are the exercises. I’ve also been known to answer questions individuals have over e-mail–regardless of whether or not they’ve taken my course.

  • If you are a design engineer who can’t humble himself enough to verify a broken design that’s been given to you, then I foresee that you will also be difficult to manage. It would be easier to work with someone less talented than someone with more talent who isn’t willing to do what he’s asked. Again, my recommendation to any hiring manager would be that they look for someone else.

    Right now, it’s a buyers market. There are lots of new individuals graduating each year with a degree in Electrical Engineering who have design experience.

    Finding a new graduate is easy.

    Finding someone who has both talent and character, that’s a much harder task.

Take some time. Think about it.

Yours,

Dan