Am I right to be upset?

I can see why you have concerns and fears around that - try to talk to your boyfriend about it and how it makes you feel when he says two different things. Don't blame him for anything, besides, you weren't 'with' him at the time it happened. Make sure you tell him how you feel before it turns into a big wall of misunderstanding between you both. Don't get paranoid about it, it will just leave you insecure and have doubts and that will end your relationship sooner or later. Your being a bit irrational if it's eating you up, you need to have trust that he will always tell you the truth, but if you find he continues to distort the truth then start to think about whether you should be with him at all.