Whitelisting is Important

In order to reduce the impact of greylisting, the spamlogd(8) daemon should also be run, with a pf ruleset configuration which logs the smtp sessions to and from the mail servers(s). 

pass in log on $EXT_IF inet proto tcp to $MAILHOSTS port smtp keep state
pass out log on $EXT_IF inet proto tcp from $MAILHOSTS to any port smtp keep state

spamlogd will whitelist (or update the whitelist entries for) each host it sees successfully connecting to the mailservers, or, every host it sees the mailservers speaking smtp with. It does this by updating the expiry times in /var/db/spamd

This prevents active entries from expiring, and automatically whitelists the mail servers your server talks to, making a best effort at ensuring replies are not subjected to greylisting delays.

Static whitelists for big sites are also available at http://www.greylisting.org/, and can be added in as a pf table.