I agree with ninjanurse also. Some people just aren't meant to drink milk. Lactose intolerance has actually been linked to genetics and ethnicity. And unpasteurized milk can carry harmful bacteria. But if you have been drinking it for two years without experiencing any negative reaction to it, I don't see how continuing to do so would cause any harm. I would recommend sticking to low-fat or non-fat milk but even drinking two cups of whole milk shouldn't cause noticable weight gain. Especially since you are practicing martial artist and, I assume, quite active.