Live Scenario 1:
One of our customers complained that their remote outgoing mail queue is rising rapidly. We found that the Internet link’s capacity to relay mail has suddenly dropped. So while mails were going, but very slowly and hence the queues were rising. Apparently no change and even as confirmed by the ISP. We were quite convinced that there was possibly an introduction of an SMTP proxy in the path which had some rate control or tar pit policies. To prove our hypothesis, we routed the mail from our hosted servers over a different port (not port 25). The mail flow became normal while sending through the same Internet link. As of the time of this writing, the ISP is still to acknowledge that there is an impediment in the path for port 25.
Live Scenario 2:
One of our customers recently complained of mail queues between their site servers getting stuck. Mail were flushing but very slowly and most mail were deferred in the queue with a timeout error while connecting to the recipient site server. It was noted that there was no change in the environment and the servers. After we audited and cleared the application configuration, we asked the customer’s network team to scan the network for any problems. We even got the servers rebooted, and the routers and switches rebooted. But it didn't help. The network team's report was all clear. The connections between the servers were over a VPN. We suspected the VPN and to prove this hypothesis, we routed the mail between the servers over the Internet. This helped temporarily and then again, within an hour or so, the same phenomena was observed – mail stuck in a queue, flushing very slowly and deferring with a timeout error. Somehow during all this, none of the teams could think of asking the ISP to do some diagnosis on the Internet links or the VPN. They were invisible. As a last resort, and after recalling some previous experiences, we decided to route mails over another port (not port 25, we had to map another port on the MTA). Viola, the mail flow resumed normalcy and is working fine now. This only means that the ISPs have proxied port 25 to intercept messages and/or put some rate control on it. Its strange that they didn’t think of at least informing the customers of this change
Live Scenario 3:
BSNL has blocked port 25 entirely. This means that on the last mile links supplied by BSNL, the devices cannot connect over port 25. When this happened, a lot of our users who were on BSNL links, lost the ability to send mail from their desktop clients. As a result, we had a lot of support calls to map another port on our mail servers, affected by this change. The users then had to use this other port in their desktop client configuration, outgoing SMTP server settings to resume normal functioning.
Live Scenario 4:
A while back, when our users would travel to foreign destinations like the USA, Europe, or Dubai, they complained that they were unable to send mail from their mail servers, which were back in India. This problem was because most international ISPs have proxied port 25 to intercept email traffic and redirect the traffic to the MX servers of the sender’s mail domain. Normally MX servers don't expect mail “from” a domain that is hosted on that server itself. Hence our users were stuck. This was rectified with a solution similar to the one explained in Live Scenario 3. We had to map another port to the MTA and get users to change their desktop client configurations to use this new port for outgoing SMTP connections.
Why block port 25?
It's now clear that across the world (India included), ISPs are intercepting/choking or blocking port 25 access. The main reason for this is because when a computer gets infected by a virus it can be hijacked by the virus writers to send out thousands, if not millions of spam emails – the ISPs have to prevent this from happening or else it would clog up their precious bandwidth, in turn providing a poor experience to their users. Blocking the port is the only real solution.
What's the impact of this on my setup?
All users of email clients like Thunderbird, Outlook, etc, who connect to their mail servers over Internet links which they may use like broadband or leased lines, may experience problems sending mail if the provider of their Internet link is blocking port 25. (Refer Live Scenario 3 and 4)
All mail servers which are hosted at Datacenters (professional IDCs or in premise) typically send emails to other servers over SMTP port 25 ONLY. This is done since it is a standard and there is no real way of knowing the port of a recipient server if it's not standard. In this situation, if your ISP has blocked port 25, then your mail server can never deliver mail to any other server. This will render an In premise setup useless. (Refer Live Scenario 1 and 2)
So what next?
We have no option but to accept it and work around it. Possible working solutions are given below:
A. For clients connecting to the mail server (who could be using broadband from their office, or a wireless broadband modem, or a leased line from home, etc), change the port of access for SMTP to any other port other than port 25. This is a very simple configuration in Connect Xf and most other mail servers.
B. Since this problem is mostly in last-mile connections, landing at an office or a home, its quite possible that you may encounter this issue even with mail flow between servers and mail which is relayed directly from the servers in your location (since those mail will be sent to the recipient servers over port 25)…refer Live Scenario 1 and Live Scenario 2. The solution to this problem is to route all mail between local servers over another port, other than port 25 and for outgoing mail, use the relaying services of an email hosting provider (who would be hosted at an IDC and their links don’t have such restrictions since they serve large racks of servers and are providing controlled bandwidth to customers). Relay all outgoing mail to the Mail service provider’s relay servers over another port (other than port 25) and let them relay to the recipient servers over port 25. In other words, we recommend a setup that is a hybrid between a hosted email setup and a local mail gateway server setup.
I recently wrote a note on how network problems are difficult to diagnose or prove.
As for the services community, IT teams, I recommend that they must look at the ISPs as a very important part of the stack during their troubleshooting.