Should I cancel my breast clinic referal

About a week ago I found a solid lump that didn’t move just under my right breast. I waited 5 days and asked for an emergency doctors appointment to which the doctor examined me and gave me a referral to the breast clinic. There is family history of breast cancer with my grandmother passing away at 45. 
But after my appointment about a day or two later I can’t feel the lump anymore. I’m not sure if it was a cyst or fatty deposit and it has just gone. 
I don’t want to go to the clinic and waste time or an appointment. 
but I also want to make sure everything is ok as I’ve never found a lump before. 
Should I just go to be safe ?