Why might the nature of the crimes Macbeth committed have led Macduff to believe his family would be safer at his own castle?

Macbeth's first crime was murdering Duncan under Macbeth's own roof, not in Duncan's own home.  Obviously, Macduff knows better than to send his wife and children anywhere near Macbeth at this point.  Macbeth's next crime was arranging for the murders of Banquo and Fleance (although Fleance escaped); this happened while Banquo and his son were on the road, away from home.

Macbeth has never gone to an enemy's home and committed a crime there.  He's only committed crimes while the victim has been away from the victim's own home, and this might lead Macduff to believe that his wife and family would be safest at home.  On the road, like Banquo and Fleance, they are vulnerable.  Therefore, if they just remain safely tucked away in their home in Fife, it doesn't seem like Macbeth's style to reach out so far and harm them there.