How do your neighbors know you collect from the gov't? I'd also tell them to F-Off. They don't know your entire circumstances.
My hubby works 2 - 3 days a week and still receives SSDI, which has different rules than SSI or Retirement. We've been fighting SSA for 3 years now to get them to stop trying to stop payments and collect over $9,000 they say he was overpaid. Back when he started working part-time, there was hardly any info on the SSA website to let you know what you can or can't do while collecting SSDI.
On top of that, he was told the same wrong information by reps over the phone. What we figured out after we were finally told SSA can't count income paid that wasn't earned by actually working such as vacation, sick, travel reimbursements, etc. and also what expenses can be written off, he wasn't overpaid at all. Unfortunately, as many times as they've stopped a month here and there, they've already gotten half the money and continue to take out $100 per month until we get a hearing, which was suppose to take place by now. The case worker assigned to hubby's case was constantly out sick and never returned one phone call in 2 1/2 years and the hearing was denied by a judge b/c the caseworker didn't do her part in the process. So now we're back to filing for a hearing again and we have a different caseworker.
My husband really can't work enough hours to replace his SSDI, so the local office told him to cut his hours to be sure he doesn't go over. You should do whatever you feel you can do. Some people can't find jobs that pay them enough or they don't have the stamina it takes to work and do without gov't help. Good luck!
Stella ~ You kill me

But what do you do about the people who like doing the work you give them as long as they have somewhere to go and are always drinking your beer? In STL, I dealt with guys coming over on weeknights b/c they couldn't drink at their house, so they'd come over to do a project and stay until after midnight, drinking beer and blasting the music when I had to work the next day

Quite a few times, I had to go down in the basement and yank a few fuses out of the fuse box to let them know I was pissed and it's time to go home!