Muppets tend to be lefties. Look here, here, and here.
Not all Muppets are lefties, mind you. Fozzie is often referred to as Kermit's right hand, and, it turns out, for good reason. Observe:
First of all, you may be wondering why Muppets tend to be lefties. The reason is this: most of the smaller Muppets (meaning most of the Muppets) are operated with the Muppeteer's writing hand (usually the right hand, as you might guess) inside the head. This leaves the left hand to strum whatever stringed instrument the Muppet might need to strum. In the case of larger, live-hand (as they are known) Muppets such as Fozzie, it is more natural to strum with the hand with which the performer would usually strum, were that performer actually playing an instrument.
Oz was usually operating Fozzie's head with his right hand. He manipulated Muppet heads with his right hand most of the time when he was the lead operator, meaning that his left hand was free to perform both hands in those cases where he was operating a hand-and-rod Muppet, such as Miss Piggy. His left hand was free to perform only the Muppet's left hand, however, in those cases where he was operating a live-hand muppet.
(In cases where Oz was not the lead performer--such as Rowlf, whose main performer was Henson--Oz was the right hand. In even rarer cases--such as Swedish Chef, whose head was operated by Henson--Oz was both hands.)
I say "most of the time" because there were multiples of a number of the more commonly featured Muppets (see: Bike-Peddling Kermit). It might not always be the case that Fozzie would strum or write with his right hand, as another version of him (one that wasn't built for sitting down, as the one in the embedded video seems to have been) might have required different hand placement.
My guess is that Oz was squatting on a platform some feet below the visible stage floor, with his right hand in Fozzie's head, his head in Fozzie's belly, and his left hand on the guitar neck. Another performer (I don't know that there was a standard assistant for Fozzie) was probably crouched down to Oz's right, right arm rising up from behind the guitar, hand draped over the strings--at least for the close up.
It's unclear to me who would have been performing Fozzie when both he and Piggy share the screen--or if the Fozzie in that shot is the same as the one used for the closeup. We don't see Fozzie's feet in the closeup, so it's likely that it is an entirely different Fozzie.
Whatever the case, a lot went into selling the illusion. It is a testament to the Muppet performers' ability to suspend the audience's disbelief that a scene such as the above can continue to be as affecting as it is thirty years on, and it is a testament to Oz's mastery of the craft that he could provide Dave Goelz's Gonzo with such gentle backing harmonies--one as a pig, one as a bear, and neither in his natural voice--without it seeming ridiculous.