I understand not wanting to talk to the manager. But I don't think it needs to be a fight. An awesome manager wants to know what is going on in their establishment.
I'm feeling particularly sympathetic to the servers today because I have what we call support staff at work. There is one who is not the least bit supportive. Management won't do anything about it. It could be that way in some restaurants as well. But if you've encountered this problem more than once at a particular restaurant then that is a big problem. Too many great places to visit.