Nipple change

Hi, first time poster here. 
I have had breast pain (one sided) for a number of years now, GP advised that it was likely hormonal so I haven't been worried. 
lately the pain has been different....more burny than achy and i noticed my nipple on that side slightly pointing downwards. Asked the husband to check and he agreed it looked different. I made an appt with the Gp and was throughly checked, they couldn't feel anything and suggested we take a watch and wait approach. 
this is all fine but I'm struggling to find any literature on what exact nipple changes could be linked to something sinister....everything I read suggests to look out for inverted nipples. Am wondering if I should push for a referral or be confident in watching and waiting? 
 

as an aside I'm 42. 
 

thanks in advance