I'd say have a sit down and bring it up to his face. Man to man talk it out. Don't let him think he's in your head and your sulking around worrying about it. Confront him with how ya feel. He'll respect you more and probably give you more freedom to do what you want.