If I understand correctly, you are proposing that "operand" would no longer be "quantifier" or "lerfu-string" (among other things) but some new mixed entity. Whether this is a problem or not depends on your idea of simplicity, and whether or not it's worth striving for.
The thing is in this particular context, it's causing so many problems. Simplicity is worth it up to a point, but not when it causes so many problems.
Don't get me wrong; I actually support your proposal. I'm just trying to come to a compromise for what I and others see is the part of it that causes the most problems, for the least benefit.
We now have to attach "bu" to every vowel, is that so different? Lerfu-strings are relatively rare in usage. I just don't see much need to have special rules for them.
For one thing there is an experimental letteral series that uses a'y, e'y, i'y, o'y, and u'y (however, it's not been decided what to do with y'y under this system). For another thing, it introduces opacity to the Lojban numeral system. They are embarrassingly simple compared to most other languages in that they are read out nacle'u by nacle'u. Having to figure out when {bu} needs to be added to them before reading any character string aloud ruins that simplicity.
It's like this. "abu, ebu" etc are treated as one "word" in people's minds, and represent the letters {a} and {e}. However, deciding when to use {pa} and when to use {pabu}, when both are rendered as the simple numeral {1}, introduces practical usage difficulties.