What size mlb jersey should i buy?

I am a female and I'm 5"4' and 150lbs..I wear a l-XL in t-shirts. I want to buy a brewer jersey. XL is to big!! I was thinking a medium??