Setup form based authentication for FIM self service password registration portal and avoid using default windows based login popup.

I would like to authenticate the user with the help of a custom login page , and not using the default windows login.

I tried doing it , changing settings for Sharepoint-80 to use Forms Authentication in IIS Server Manager but no redirection to the custom login page was recorded.