Mail handling in YAF is run as a scheduled process -- if mails are not going out - there could be issue with application being 'shutdown' by IIS.
Do subscriptions work (watched topics) - or PM notifications?
Test email in YAF install is not handled by task scheduler of YAF as far as I know (mainly because in installer, database may or may not exist - so mail queue table might be inaccessible - so most likely test message is sent direct).
Investigate to see if PMs are sending notifications (and that notifications are turned on in host settings) and/or watched topics (subscriptions) -- if you're not getting PM notifications either, then your system is either not populating the notification table or the scheduled task is not running (ie: YAF application shutting down before queue is processed)
If you can't find it using the forum search, try my signature link -- searches this site using Google:
Google is my Friend