Just picking up is harmless to anything but your time. Saying anything (like “yes” or “no”) can be used by crafty fraudsters to answer voice prompts on your behalf in some cases.
It is definitely a good idea in general not to answer if you don’t know who is calling.
Comments