Should I be concerned about possible trace elements that are potentially hazardous in polished gemstones used for jewelry since I handle them a lot and have frequent skinpossibly residual (which could end up internally) contact with them whether making jewelry or wearing it.

For example, I read that Fluorite can have traces of Yttrium or other heavy metals that can be toxic. Yet I also read Fluorite is only slightly if ever water soluble. So it this a cause for concern?

I should also mention that sometimes my gemstones are cleaned with distilled water if they get dirty and dusty, or sometimes are in the way when hand washing when it comes to bracelets.

If this is a potential long term hazard, what stones should I try to minimize contact with?