HekaFS

In previous jobs, especially at Revivio, I’ve spent a pretty fair amount of time creating gdb macros to make the inevitable debugging sessions a bit more productive. I’ve generally tried to stay away from that on GlusterFS, partly because there are usually better ways to debug the sorts of problems I have to deal with and partly because gdb macros are one of those things that will make you ill if you know anything about real scripting languages. For example, you can define recursive macros, but convenience variables are always global so you basically can’t use those. Instead, you have to take advantage of the fact that macro arguments are local and rely exclusively on those instead. What you end up with is this grossly inefficient and unreadable tail-recursive mess, just to work around the macro language’s deficiencies. You’ll see what I mean in a minute, but let’s start with something simple – printing out the contents of a dictionary.

define pdict
	set $curr = $arg0->members_list
	while $curr
		printf "%s = %p %s\n", $curr->key, $curr->value, $curr->value->data
		set $curr = $curr->next
	end
end

That’s not too bad. Now let’s look at one to print out some essential information about a translator.

define pxlator
	printf "--- xlator %s type %s\n", $arg0->name, $arg0->type
	set $d = $arg0->options->members_list
	while $d
		printf "    option %s = %s\n", $d->key, $d->value->data
		set $d = $d->next
	end
	set $x = $arg0->children
	while $x
		printf "    subvolume %s\n", $x->xlator->name
		set $x = $x->next
	end
end

Now things get a bit hairier. What if we wanted to print out a translator and all of its descendants? This is where that global vs. local issue comes back to bite us, because any convenience variable we use to traverse our own descendant list will also be used in each of them to traverse their own descentant lists, and finding our parent’s next sibling when we’ve finished traversing such a list is really ugly. Instead, we end up with this.

define ptrav
	pxlator $arg0->xlator
	if $arg0->xlator->children
		ptrav $arg0->xlator->children
	end
	if $arg0->next
		ptrav $arg0->next
	end
end
 
define pgraph
	pxlator $arg0
	if $arg0->children
		ptrav $arg0->children
	end
end

As you can see, ptrav has that ugly tail-recursive structure we talked about. The same thing happens when we try to print out a DHT layout structure.

define playout_ent
	if $arg1 < $arg2
		set $ent = $arg0[$arg1]
		printf "  err=%d, start=0x%x, stop=0x%x, xlator=%s\n", \
			$ent.err, $ent.start, $ent.stop, $ent.xlator->name
		playout_ent $arg0 $arg1+1 $arg2
	end
end
 
define playout
	printf "spread_cnt=%d\n", $arg0->spread_cnt
	printf "cnt=%d\n", $arg0->cnt
	printf "preset=%d\n", $arg0->preset
	printf "gen=%d\n", $arg0->gen
	printf "type=%d\n", $arg0->type
	printf "search_unhashed=%d\n", $arg0->search_unhashed
	playout_ent $arg0->list 0 $arg0->cnt
end

I’ve really just started defining these, so if you have some suggestions please let me know. Otherwise, you can use them by just copying and pasting into your .gdbinit or (better yet) into a separate file that you can “source” only when you’re debugging GlusterFS. Share and enjoy. ;)