As long as it is white asbestos and in good condition I'd leave it be, you can create more of a hazard removing it then leaving it in place.I'm not sure if blue or brown were ever used in fuseboards but they need to be removed by a specialist company if they are found.
To be honest the small amout of asbestos modern day sparks will encounter, especially in domestic situations, is not going to kill you. Most sufferers worked in industries that had it everywhere! Ships boiler rooms, roof manufacturers etc.
Personally, I carry Asbestos warning stickers now and slap them on the front of the board. I would advise removal and upgrade of the CU but that's upto the owner.
You are, however, well within your rights to refuse to work on it.