Debugging Go (golang) programs with gdb

/* ---[ Update: July 2015 ]--- */

At this point, I think the Go community has given up trying to make gdb work with Go programs. It is pretty painful, so I don't recommend it. Recently, I've tried both delve and godebug, two contributions from the Go community.. I had better luck with godebug. In fact, it performed perfectly for a recent issue I was having and was a joy to work with.

---

/* ---[ Debugging Go ]--- */

At the time of this writing, the only debugger I know of for the Go language is the FSF's gdb debugger. gdb can be used to debug programs written in Go and compiled with gccgo or 6g compilers.

At present, I'm using Go version 1.1.2 (on Xubuntu Linux). Do not upgrade to version 1.2 if you want to be able to use gdb. The 1.2 release introduced changes that breaks single stepping through code in gdb: http://code.google.com/p/go/issues/detail?id=6776.

As a side note, I find this situation pretty disappointing. It says that the Go developers are not including gdb compatibility tests in their testing of Go. That really isn't acceptable in my opinion if gdb is the only debugger tool available. Happily, the last entry in the comments from one of the Go maintainers/developers is "If possible, fix this for 1.2.1."

/* ---[ GDB ]--- */

I won't give a gdb tutorial. There are lots of good ones on the web, such as these:

• http://betterexplained.com/articles/debugging-with-gdb/
• http://www.unknownroad.com/rtfm/gdbtut/gdbtoc.html
• http://beej.us/guide/bggdb

/* ---[ Using GDB with Go ]--- */

I've written a unit test, using Go's testing library, for the code I'm debugging. Here's the code under test:

//
// Read in the username and password properties from the CONFIG_FILE file
// Returns error if CONFIG_FILE cannot be found or opened
//
func readDatabaseProperties() (uname, passw string, err error) {
    propsStr, err := ioutil.ReadFile(CONFIG_FILE)
    if err != nil {
        return
    }

    for _, line := range strings.Split(string(propsStr), "\n") {
        prop := strings.Split(line, "=")
        if len(prop) == 2 {
            switch prop[0] {
            case "username":
                uname = prop[1]
            case "password":
                passw = prop[1]
            }
        }
    }
    return
}

And here is the unit test for it:

func TestReadDatabaseProps(t *testing.T) {
    uname, passw, err := readDatabaseProperties()
    if err != nil {
        t.Errorf(fmt.Sprintf("%v", err))
    }
    if len(uname) == 0 {
        t.Errorf("uname is empty")
    }
    if len(passw) == 0 {
        t.Errorf("passw is empty")
    }
}

The CONFIG_FILE on my system has:

username = midpeter444
password = jiffylube

The unit test is currently failing:

$ go test -test.run="TestReadDatabaseProps"
--- FAIL: TestReadDatabaseProps (0.00 seconds)
    indexer_test.go:150: uname is empty
    indexer_test.go:153: passw is empty
FAIL
exit status 1
FAIL    fslocate    0.023s

So let's check it out in gdb. First, compile the go code with the following flags:

$ go test -c -gcflags '-N -l'

The -c flag causes the go compiler to generate an executable in the current directory called xxx.test, where xxx is the name of the directory your code is in. In my case, it generated fslocate.test.

Next ensure you have the GOROOT environment variable properly set and start gdb.

$ gdb fslocate.test -d $GOROOT

After doing this, if you see an error about /usr/local/go/src/pkg/runtime/runtime-gdb.py, see my side note #1 at the bottom of this posting.

The way we just fired up gdb will give you the command line prompt only. If you want to use it that way, I recommend using the frame command in order to keep track of where you are in the code as you single-step through it.

However, I like using the -tui option to gdb which will split the screen and give you a visual display of the code as you step through it.

$ gdb -tui fslocate.test -d $GOROOT

You get a screen like this. Don't worry about the assembly code - that will switch to your Go code once you get underway.

To proceed, I'm going to set a breakpoint at the start of the test. To do this you'll need to specify the path to the function you want to break on. The code is actually in package "main", but it is in the "fslocate" directory, so I set the breakpoint like this:

(gdb) b "fslocate.TestReadDatabaseProps"
Breakpoint 1 at 0x43c730: file /home/midpeter444/lang/go/projects/src/
fslocate/indexer_test.go, line 144.

If you get a message like:

Function "fslocate.TestReadDatabaseProps" not defined.
Make breakpoint pending on future shared library load? (y or n) 

You didn't get the path right. See side note #2 for some help.

Now that we have the breakpoint set, run the program to the breakpoint by typing run or r:

(gdb) r

Now you'll see that the TUI window jumps to the current line of code and highlights it:

Next advance line by line with n (or next) to step over to the next line where the function under test is called. Once there I step into it (s or step) and step over the next few lines until I'm here:

Now let's inspect some of our variables with print or p:

(gdb) p propsStr
$2 = {array = 0xc2000b3240 "username = midpeter444\npassword = jiffylube\n",
      len = 44, cap = 556}
(gdb) p err
$3 = {tab = 0x0, data = 0x0}
(gdb) p line
$4 = 0xc20009b7b0 "username = midpeter444"

The function ioutil.ReadFile returns a byte slice and error. Inspecting them shows the string value of the byte slice, its length and capactiy and that the error is nil (as indicated by its data value being 0x0).

So the file read worked. Then we read in the first line as a string and it looks good. Then I called:

prop := strings.Split(line, "=") 

to split the line into a string slice. Inspecting this slice shows us the contents of the slice "struct", but not the contents of the underlying slice array:

(gdb) p prop
$5 = {array = 0xc2000b1260, len = 2, cap = 2}

To peek into the array in gdb, we can use standard array indexing:

(gdb) p prop.array[0]
$20 = 0xc20009b7b0 "username "
(gdb) p prop.array[1]
$21 = 0xc20009b7ba " midpeter444"

or we can use the dereferencing operator * and @N to look at N contiguous portions of the array. The inspection so far told us that there are two entries in the array, so this is how to see all of the array entries in one command:

(gdb) p *prop.array@2
$22 = {0xc20009b7b0 "username ", 0xc20009b7ba " midpeter444"}

And with that I can see the defect in my code: I need to trim the string before comparing to the cases in my switch statement.

/* ---[ Looking into Go structs ]--- */

You can also use dot notation to look into Go structs, using dereferencing where you have pointers rather than values.

For example here's a snippet of code from my fslocate program:

fse := fsentry.E{"/var/log/hive/foo", "f", false}

// query that new entry
dbChan <- dbTask{QUERY, fse, replyChan}
reply = <-replyChan

fse is a struct of type fsentry.E:

type E struct {
    Path       string // full path for file or dir
    Typ        string // DIR_TYPE or FILE_TYPE
    IsTopLevel bool   // true = specified in the user's config/index file
}

reply is a struct of type dbReply

type dbReply struct {
    fsentries []fsentry.E
    err       error
}

dbChan is a go channel (that takes dbTasks, another struct).

For that snippet of code I can inspect the contents in gdb:

(gdb) p fse
$1 = {Path = 0x637690 "/var/log/hive/foo", Typ = 0x61fd20 "f",
      IsTopLevel = false}

(gdb) p *dbChan
$3 = {qcount = 1, dataqsiz = 10, elemsize = 56, closed = 0 '\000',
      elemalign = 8 '\b', elemalg = 0x5646e0, sendx = 1, recvx = 0,
      recvq = {first = 0x0, last = 0x0}, sendq = {first = 0x0,
      last = 0x0}, runtime.lock = {key = 0}}

I had to use * on dbChan since channels in go are pointers (references).

The reply struct is a little more tricky:

(gdb) p reply          
$9 = {fsentries = {array = 0xc20007e640, len = 1, cap = 4},
      err = {tab = 0x0, data = 0x0}}          

Here we see that it has two entries: fsentries and err. The error is null. fsentries is an array of length one, which using the techniques we walked through earlier can be inspected:

(gdb) p reply.fsentries.array[0]
$11 = {Path = 0xc2000cbfe0 "/var/log/hive/foo", Typ = 0xc2000005d8 "f",
       IsTopLevel = false}

/* ---[ Inspecting "slices" of Go arrays ]--- */

Suppose you have a large Go slice and you want to see elements 20 through 24 only. How would you do that? You can use the indexing operator and the @N operator together. Here's a slice of length three, where I look at the last two elements in the last command:

(gdb) p configDirs
$6 = {array = 0xc2000f8210, len = 3, cap = 3}

(gdb) p *configDirs.array@3
$7 = {0x618dc0 "/d/e", 0x627170 "/new/not/in/db", 0x626fd0 "/a/b/c/foo/bar"}

(gdb) p configDirs.array[1]@2
$8 = {0x627170 "/new/not/in/db", 0x626fd0 "/a/b/c/foo/bar"}

Hopefully, reading through a gdb tutorial and this post are enough to get you through your Go debugging sessions.

Side note 0: Key bindings with -tui

One tricky/annoying aspect of running with the TUI is that the arrow keys navigate around the TUI screen rather than on your command line. So to go back and forward in history (usually the up and down arrows) use the bash bindings Ctrl-p and Ctrl-n respectively. To go left and and right on the command line use the bash/emacs bindings Ctrl-f and Ctrl-b respectively.

Side note 1: python error when starting gdb

On Ubuntu, I get an error after gdb starts up stating:

File "/usr/local/go/src/pkg/runtime/runtime-gdb.py", line 358
print s, ptr['goid'], "%8s" % sts[long((ptr['status']))], blk.function

I've been able to ignore this and still use gdb with Go code just fine.

According to this thread, this is a python version issue used to build gdb and isn't an issue with the Go distribution. It may be specific to Ubuntu-flavored Linux distros. This posting in that thread says you can fix it with:

sudo 2to3 -w /usr/local/go/src/pkg/runtime/runtime-gdb.py

But when I did that I now get errors when I try to look inside a string, struct or array:

(gdb) p propsStr
$1 =  []uint8Python Exception <class 'TypeError'> 'gdb.Value' object cannot
      be interpreted as an integer:

So I recommend that you back up the /usr/local/go/src/pkg/runtime/runtime-gdb.py file before trying it in case you get the same error I did.

Side note 2: Function "fslocate.TestReadDatabaseProps" not defined.

If you get this type of error message when you set a breakpoint on a function:

Function "fslocate.TestReadDatabaseProps" not defined.    
Make breakpoint pending on future shared library load? (y or n) 

Then you either got the path wrong or the notation to the path wrong, so type "n" and try again. First check your spelling.

If that doesn't fix it, you might have a nested path. For example in the fslocate project, I have a stringset subdirectory:

midpeter444 ~/lang/go/projects/src/fslocate
$ tree stringset/
stringset/
├── stringset.go
└── stringset_test.go

In my case the fslocate directory is in my $GOPATH/src directory and the test file is in the fslocate/stringset directory. If I were to go into that directory and compile the test for gdb,
the breakpoint path would be:

(gdb) b "fslocate/stringset.TestReadDatabaseProps"

Notice that you uses slashes for package separators and dots to indicate a function in a package.