Planning on writing an emal server software, but need some help how it all works

Ok I understand a VERY BASIC system with SMTP for sending and POP3 for receiving, when the situation is that everyone is using the same service provider (said provider provides the SMTP and POP3 for everyone involved). It would be like this diagram.
http://i.imgur.com/He65h.png

However in real life it is VERY LIKELY that 2 different people are using 2 different email service providers (such as the sender using GMail and the receiver using HotMail). So how does "Provider 1" get the email to "Provider 2"? If I am to write an email server software and be my own provider so that when people send me email they'll be sending it to myname@myserver.com I will want to be able to recieve it from ANYBODY using whatever service they are using (HotMail, GMail, etc). So I'm going to need to know how to get my POP3 server I'm writing to be able to receive email from ANY OTHER PROVIDER. The diagram below shows my situation.
http://i.imgur.com/ZZC2L.png

I'm going to need to know the port number I've marked on my diagram as "Port ?????" and I'm going to need to know the type of software that's shown in my diagram as "Special software on service provider 2's computer that lets service provider 2 get emails from other service providers". Once I can establish these to "unknown" pieces of info, I can successfully finish writing my email server software.