Today, let’s put some things together. We’ve already built a debugging bus approach that we can use to interact with an FPGA and draw information out of it. [1] [2] [3] [4] [5] [6] [7] [8] [9] We’ve also talked about Verilator and what Verilator allows us to do in simulation.

Suppose we spend the time today to build a Verilator based simulation of our FPGA design that can be controlled via this debugging bus interface?

While most of the difficult parts have already been accomplished, there are still there are three basic components that we are going to need to work on to do this. The first is the Makefile for our top level Verilog design. The second is the C++ driver for our test bench. We already discussed how to generate a generic driver here and our debugging philosophy in general here, so this discussion will only describe the differences between a generic test bench driver and the one we’re building for this project. The last item we’ll need to figure out is how to build all the pieces together–i.e. the final simulation Makefile.

When we finish, let’s run some tests on our “hardware” and see how it works!

The RTL Makefile

The RTL makefile needs to handle two things. First, it needs to convert our our Verilog test bench into a C++ class using Verilator. Then, the second step directed by the RTL makefile will be to build that C++ class into a library that can be linked with our C++ testbench to create a fully functional simulation.

We’ve already discussed the basics of how to run Verilator. In quick sum here, you can run Verilator on a top level Verilog file as in:

verilate -Wall -trace -cc testbus.v

Today, let’s just highlight some of the neat things that can be placed into our rtl/Makefile in order to do it well.

For example, I’d like to be able to use the test bench in the dbgbus repository to test several different debugging busses. For this reason, I’m going to add,

BUS := hexbus

to the Makefile, and -y “../../$(BUS)/rtl” to the Verilator command line. That’ll cause Verilator to look for our sources in the hexbus project subdirectory for any bus sources.

Our top level Verilog file testbench.v includes references to multiple files. We’d like to only re-run Verilator any time these files change. Is there a way to get dependency information for Verilog code in a fashion similar to C++ code? Yes, there is. For this, we’ll add “–MMD” to the Verilator command line, and include the dependency file it generates at the end of our Makefile:

DEPS := $(wildcard $(VDIRFB)/\*.d)

ifndeq ($(DEPS),)
include $(DEPS)
endif

We’ll also set two more Makefile variables before we call Verilator.

VERILATOR ?= verilator
VFLAGS := -Wall --MMD -trace -y ../../$(BUS)/rtl --Mdir $(VDIRFB) -cc

This will do two things. First, it will allow you to run make with your VERILATOR version defined in an environment variable. Second, it will simplify our Verilator call later.

Now that we have these variables defined, we can create the “verilate” target. This target will build the C++ code. Since the actual name of the C++ code is somewhat obtuse, we’ll rename what we are building to “verilate” and then forward that to what needs to be built. Finally, we’ll actually run our Verilator call–simplified by the VFLAGS we defined above.

.PHONY: verilate
verilate: $(VDIRFB)/Vtestbus.h
$(VDIRFB)/Vtestbus.h:
	$(VERILATOR) $(VFLAGS) testbus.v

The last step is to build the library. This step is somewhat anticlimactic, since Verilator did most of the work for us. Building the library depends upon the source files Verilator just built for us. Making things even easier, Verilator just generated its own make file. Hence, we’ll just recursively call make on that make file to build all that’s left.

.PHONY: library
library: $(VDIRFB)/Vtestbus.h
	$(SUBMAKE) $(VDIRFB)/ -f Vtestbus.mk

This recursively calls make in the $(VDIRFB) subdirectory, and it does so with the “-f” argument specifying that it should use as its make file $(VDIRFB)/Vtestbus.mk.

That’s about it. The RTL Makefile also contains a clean target, but in general that’s all that it takes to create a Makefile that will run Verilator to build a simulatable library.

The CPP Testbench

As we discussed when presenting Verilator, Verilator needs a small C++ driver program to run the simulation.

The driver we are going to use for this simulation, both the generic one and the specific simulation one, should look very similar to the ones we built during our prior discussion. Here, we’ll look at two differences.

The first is that we have made the termination criteria a touch more general than before. For this test bench, we’ll terminate our simulation either whenever the $final Verilog instruction is reached (which we didn’t use in our Verilog code), or whenever the o_halt GPIO is asserted.

bool	TESTBUS_TB::done(void) {
	if (m_done)
		return true;
	else {
		if (Verilated::gotFinish())
			m_done = true;
		else if (m_core->o_halt)
			m_done = true;
		return m_done;
	}
}

That’ll allow us to control things from within the simulation (such as by CPU software instruction) if we’d like.

The other changes are the necessary parts of putting a UART interface onto this test bench. We’ll do that using the uartsim simulator found in the wbuart repository. There just a couple of steps there.

First, we initialize the UART component module with a zero, indicating that we want the UART to interact with us using stdin/stdout. We’ll adjust that later so that we can interact over TCP/IP.

Second, the UART simulator needs to know the baud rate we are using to communicate with. This needs to be the same as the UART configuration parameter found within our testbus.v. If you wish to run this on your hardware, you may need to adjust the parameter in the testbus.v top level file, as well as in this file–to make sure you are communicating at a common baud rate. Otherwise, we’ll just assume that one baud is 25 clock ticks. You can check the UART component specification document for more information on how to understand this setup value.

That’s it for setting up the C++ portion of the simulation! All that’s left is to build it and test it.

The CPP Makefile

For building our simulation executable, we return to the Makefile necessary to do it. What we want to do is really quite simple. We’d like to issue a command such as,

testbus_tb:
	$(CXX) $(CFLAGS) $(OBJECTS) -o $@

where $(OBJECTS) are the object files we need, and $(CFLAGS) are whatever flags we wish to build with. Our final solution will be just about that simple.

The questions remaining are, what flags, what objects, and how do we build those objects. The Makefile itself includes dependency generation, but we can leave that as a topic for another day or perhaps even for another blog.

For the first question, which flags shall we use, its in many ways up to you. We’ll use the “-Og” flag to turn on the optimizer, although any optimizations should really also be a flag to Verilator as well. Here, we’ll also use the “-Wall” flag as well to see if we can encourage g++ to find any bugs in our code. We’ll also need to tell the C++ compiler to look for the include files found both in our RTL/obj_dir directory, as well as those found in the Verilator root include directory.

# Show make how to find our compiled verilated output
RTLD  := ../rtl
VOBJDR:= $(RTLD)/obj_dir
# Show make how to find Verilators support files
VROOT := $(VERILATOR_ROOT)
VINCD := $(VROOT)/include
# Generate a full include list
INCS  := -I$(VINCD) -I$(VOBJDR)
# Build our C-Flags
CFLAGS := -Og -g -Wall $(INCS)

You might also wish to set the VERILATOR_ROOT directory. On my system, this is found in “/usr/share/verilator”, but that’s not the same among all systems. At the encouragement of others who’ve tried my code, I’ve switched to:

VERILATOR_ROOT ?= $(shell bash -c 'verilator -V|grep VERILATOR_ROOT | head -1 | sed -e " s/^.*=\s*//"')

This will call a shell script, which will run “verilator -V” (try it!). The output is then searched for VERILATOR_ROOT. Since Verilator produces two lines with this value within it, we grab the first one and use it. Further, we’ll only do so if VERILATOR_ROOT isn’t already set–perhaps from a parent make file (that’s what the ?= does).

The next question is, what objects do we need to include? First, there’s the obvious one, we’ll need to include testbus_tb.o. We’ll also want to include the Verilator class definition library from Vtestbus__ALL.a. Going one step further, we’ll want to include our UART simulation object file, uartsim.o. Finally, we’ll want to include the tw Verilator files, verilated.cpp and verilated_vcd_c.cpp once compiled to object files. This is as easy as:

VSRCRAW := verilated.cpp verilated_vcd_c.cpp
VSRC    := $(addprefix $(VINCD)/,$(VSRCRAW))
VOBJ    := $(addprefix $(OBJDIR)/,$(subst .cpp,.o,$(VSRCRAW)))
SOURCES := testbus_tb.cpp uartsim.cpp
OBJECTS := $(addprefix $(OBJDIR)/,$(subst .cpp,.o,$(SOURCES))) $(VOBJ)

To build the sources into object files, we’ll add two generic build rules. The first will apply for the objects built from this directory.

$(OBJDIR)/%.o: %.cpp
	$(mk-objdir)
	$(CXX) $(CFLAGS) -c $< -o $@

If you are not as familiar with make files, the first line specifies a generic pattern: files within the current directory and ending with .cpp can be turned into object files in the $(OBJDIR)/ directory having the same name but the .o suffix instead.

The second line, $(mk-objdir), calls a function that we’ll define elsewhere in our makefile that builds our object file directory, $(OBJDIR). While it’s not necessary to place your build products in a separate object file directory, I find that it both cleans up the main source directory, and it also provides options later in case you need to build on multiple systems.

The third line is used to call your C++ compiler, to give it the CFLAGS variable, and to ask it to produce an object file (-c) rather than a completed program. Other make file variables include the name of the source file that tripped the rule, $<, and tell it to place the output (-o) into $@–the result the rule is supposed to produce.

We repeat this for the Verilator source files in the Verilator include directory,

$(OBJDIR)/%.o: $(VIND)/%.cpp
	$(mk-objdir)
	$(CXX) $(CFLAGS) -c $< -o $@

Since we’ve already presented the final rule, that’s about all there is to the makefile necessary to generate this simulation.

Should you need to build a make file of your own, you might wish to start from this one and just change the names of the files necessary to run your simulation.

Simple Tests

We now have a Makefile in bench/rtl that will compile our debug bus and a test wishbone interconnect into a C++ class. We also have a Makefile in bench/cpp that can then be used to build the project into an executable. Shall we run our executable?

Let’s try it out. The only caution before starting is that this program as written will generate megabytes of trace information very quickly. That trace file, if you aren’t careful, may very well fill up your hard drive. So, don’t leave the simulator running at the end of each test.

With all that said, let’s run the program:

./testbus_tb

Now that it’s running, it’s waiting for an input. So, let’s try reading our constant address. If you type,

A2040R

your design will return “A00002040R20170622”. The first number, A00002040, indicates the address that we just changed the interface to. The second number (starting with the R) indicates the value we just read: 0x20170622.

Let’s try another. Let’s try reading that counter, and let’s do it a couple of times. The counter itself is at address 0x204c. However, if you remember from when we built our simple bus master we chose to use the LSB as indication of whether or not the address should increment between reads. By setting it to zero, we’ll keep the address constant between subsequent read commands. For this reason, we can issue an address command, followed by several read commands and they’ll all read from the same address:

A204dR
R
R
R

Make sure you type the newlines between the R’s! This bus isn’t yet robust enough to handle more than one command given to it at once. (It needs FIFO support.) Still, I get the response:

A0000204dR000485b4
0005234b
0005cbf1
00068df3

Not bad. Looks like we’ve got something working.

Can we trigger an interrupt? You may remember that we had an interrupt tied to the LSB of address 0x2050. So, let’s write to 0x2051 to trigger, clear, and then trigger this interrupt again I’ll prefix the commands I typed with a “< “, so you can see the responses in line.

< A2051RW1
A00002051R00000000K00000000I
< W0
K00000000
< W1
I
K00000000
< W0
K00000000

Did you see the “I” response? That tells us we triggered our interrupt as desired.

How about the scope we put on board? Can we trigger the scope? Let’s first read the scope’s control register, then write to a block RAM memory address (triggering the scope), and then read the scope’s control register again:

< A02080R
A00002080R12a001fc
< A4000Wdeadbeef
A00004000K00000000I
< A2080R
Z
A00002080R72a001fc
< A4000R
A00004000Rdeadbeef

The first command (A02080R) reads from the scopes control register. If you look up 0x12a001fc in the wishbone scope specification document, you’ll learn several things about the scope’s state as given by this number. First, the high order nibble of ‘1’ indicates that the scope is primed but hasn’t triggered yet. Second, the ‘2’ indicates that the DATA pointer within the scope is set to the beginning of the scope’s memory (where we’d expect it). Third, the ‘a’ indicates that the scope has been configured for 2^10 samples of memory. Last, the ending 0x01fc tells us that the scope will wait 0x01fc samples after the trigger, placing our event roughly within the center of its collection window.

The second command, A4000Wdeadbeef, writes 0xdeadbeef to the first location in block RAM (address 0x4000). Since we are also using wishbone accesses to that location to trigger our scope, an “I” is also returned together with the acknowledgement (‘K’). Incidentally, we’ll need to reset the scope in order for this interrupt to clear, so don’t expect any more interrupt indications for a while.

The third command, “A2080R” reads from the scope control register again. It now has a value of 0x72a001fc. Looking at the top nibble, the ‘7’ indicates that the scope has not only been triggered, but that it has also stopped collecting.

Shall we read from the scope as well? Let’s read from address 0x2084, with the increment bit set to prohibit address increments between reads (i.e., addres 0x2085):

< A2080R
A00002080R12a001fc
< A4000Wdeadbeef
A00004000K00000000I
< A2080R
A00002080R72a001fc
< A2085R
A00002085R001b6c00
< R
R001b6c00
< R
R001b6c00
< R
R001b6c00
< R
R001b6c00

We’ve now read our first several values out of the scope! All three values were 0x1b6c00. Sure, there’s more interesting information within the scope, but we’ll have to dig further into the scope to get it. The big problem with getting there, though, is that typing 1024 “R”s to get through all the values in this scope is going to get old real fast.

We’ve also found some annoying features in the protocol as well. For example, there’s no reason why a write acknowledgement should produce 8-zeros, or why an address response to address 0x2080 should produce the extra four leading zeros. Another really annoying feature of not having a FIFO is that we can’t cut/copy/paste into our interface–that would be a nice feature. Still, having valid answers is a good first start!

You can also look in your directory and find a trace.vcd file containing the trace information we made. Feel free to run “gtkwave trace.vcd” and explore what took place after each interaction.

Next in the Series

As with many FPGA capabilities, building it just the way you want it can take several iterations. We’ve got some more iterations yet left in this design.

We’re still missing two final capabilities before we put this series to rest and move on to other topics.

  • We’d like to have a software interface to debug both our FPGA and our simulation.

    If you haven’t noticed, typing hexadecimal commands and values, and interpreting hexadecimal results isn’t very intuitive. Indeed, it’s out right cryptic! We’ll fix this by switching from this by-hand interface to a software interface.

    That software interface has been written, though not yet posted at this time. (Any Patreon supporters want to try an early version?) You’ll find it a whole lot easier to interact with large amounts of data over a software driven interface.

  • We want to be able to debug our simulation just like we would debug the hardware, by using a network enabled interface. The UART simulator has this capability within it already. Our hardware does not. Hence, we’d like to take a program like the netuart from the basic ZipCPU repository and incorporate it into this debugging bus.

It may now be time to go back and consider the difference between the student’s design process and the expert’s design process. Consider how much work it has taken to build this design, and then consider the typical student project. The difference helps to illustrate why what we are building is a tool that a professional might use. The reason is simply that building it takes time. In the case of the professional, the time is spent once, and the tool used many times over. In the case of the student who is trying to build a design from start to finish within a semester, the student often doesn’t realize the value of the tool he doesn’t have.