We own 2 mail-servers running the same configuration postfix + courier + mysql_auth etc.
The servers have the same configuration but one of them had 3-4 seconds delays when sending e-mail.

After a bunch of hours of testing and retesting i found out that the DNS was faulty.
The first server had it’s own dns server running on localhost, the second one (the one with the delays) connected to our provider’s dns.
I’ve installed a bind nameserver on the second server and all is ok now.

Hope it helps.