A frequent caller, indeed. My rule is to not pick up the phone if the number is not displayed or I do not recognise it. In the latter case, I google for the number. If it and the caller are worth remembering, their coordinates are added to a list hanging above my phone. This way, I have a quick way to pick-up or not to pick-up -- so that's not a question anymore.
Canada