Hi Kylek.  You won't be too surprised to learn that this question pops up quite regularly here. You aren't the first to be worried about accidental asbestos exposure and you won't be the last.

We all get exposed to asbestos from time to time so the occasional exposure is unlikely to do any harm. For instance, when I was a student in the early 1970s I worked my vacations in a factory which later on it transpired had played fast-and-loose with asbestos. From my own recollection I believe I was exposed a few times. I told my GP and then had a chest X ray, which was clear. Phew!

Obviously, you don't want to come into deliberate contact with the stuff, but the occasional accidental exposure can't be helped and it happens to us all.  Most people who get asbestos diseases are exposed to it over long periods, and often they're smokers. Tobacco and asbestos are a lethal combination. 

Unfortunately, I don't have a clear understanding of the regulations around asbestos, but maybe this is something where you should get yourself more informed.  Your employer has a duty to protect you from danger at work, and asbestos is (unfortunately) always a possibility in an old building.  Could you maybe ask for some training courses on asbestos awareness?