I guess (and I'm not really an expert) there's nothing really wrong with
saying mi klama fi vo'e ledo zdani (="I went to your house and back")
without specifying my start and endpoints. My concern was merely with
the referent of vo'e, which, now that I read the jbovlaste definition, I
seem to have been wrong about anyhow. It does seem to refer to the
outermost bridi.
That bridi doesn't do what you want it to do. What you're after is:
{mi klama ledo zdani [zo'e] soi vo'e vo'i}
{soi} is a reciprocity marker, so that bridi means "I went to your house from some-unspecified-place and vice versa".