Please take a moment to read http://bit.ly/demandglobalchange, to help share the message and support the initiative to tell our leaders to focus on addressing the global world problems, instead of complaining about the effects of their lack of leadership. Be a leader yourself, and share this with as many people as possible. #demandglobalchange // https://www.facebook.com/demandglobalchange



Please consider donating: https://www.corelan.be/index.php/donate/


2,632 views | This page as PDF

Free Tool – Exchange 2007 Outbound SMTP gateway redundancy

(a.k.a. PVE Exch2007 Rerouter manual)

Keywords : Exchange 2007 outbound SMTP relay redundancy SendConnector outgoing mail high availability

Microsoft Exchange 2007 allows you to consolidate your messaging infrastructure and host more users on one server. Combined with the introduction of new server roles, Exchange 2007 allows for high levels of redundancy and resilience, ensuring uptime and availability.

However, there might be a specific yet not uncommon scenario where availability may not be guaranteed without administrator intervention.† This scenario is related to outgoing SMTP emails, using a third party relay server, and 2 sites that are connected over a (very expensive) WAN connection.

I’ll try to explain the problem and what we can do to automate failover

Note : I have intentionally left out some redundancy components such as clustering, because they are irrelevant in this post.

Let’s imagine this scenario :

Site A has a mailbox server, a HUB/CAS server, a local internet connection, and a third party SMTP relay server that is hosted in the DMZ

Site B has a standby mailbox server, a HUB/CAS server, a local internet connection with a different ISP, and a third party SMTP relay sitting in its own DMZ.

Building redundancy for incoming smtp emails from the internet is not very hard.† Each domain name has 2 MX records. One pointing to the relay server in site A, the other one pointing to the relay server in site B.

Each SMTP relay is configured to transfer incoming emails to the local HUB server(s).

But there is a problem with outgoing emails.† Each HUB server has a SendConnector that points to the local SMTP 3rd party relay server in the local DMZ. But what will happen if the local internet connection in site A goes down, and the SMTP relay server is still up and running ?† The HUB server in Site A is not aware of the internet connection state, and will still continue to route outgoing emails to the local SMTP relay.† Emails will be queued until the connection becomes available again.† But sometimes you don’t know when the connection will be up and running, and you may not even be able to change routing configuration because the internet connection went down at night and you cannot VPN into the office to solve the problem.

There are some ways to overcome this problem, but most of them require manual intervention, or will make your mail routing topology somewhat cluttered and obscure.

So I came up with the idea to build an application that allows you to monitor the internet connection, disable SendConnectors and restart the Transport Service.† That way, you can temporarily disable the SendConnector to the local SMTP relay, so emails are routed to the HUB in SiteB and then transferred to the second SMTP relay, automatically.
And as soon as the internet connection in Site A is back, the connectors can be changed back to their original state.

This application is currently in beta phase, however I have decided to make it available for download already, so you can start testing and see if my concept works for you.†

The application can be downloaded at http://users.telenet.be/internet.activities/freetools/exch2007rerouter

You should install the application on the HUB server that has the SendConnector to your SMTP relay. The HUB server will need to be able to either ping hosts on the internet or connect to websites on the internet (depending on how you set up the internet connection check parameters in the Configuration of the tool). Additionally, the application uses PowerShell and Exchange specific cmdlets, so make sure the Management Shell is installed on the HUB server.

After installing the application, you will find these 2 components :

1. A service, called "PVE Exch2007 Rerouter". The startup state is set to manual, so don’t forget to change the state to "Automatic" yourself.† Don’t try to start the service yet, you will need to run some Configuration settings first.

image_2

First of all, change the credentials for running this service from "Local System" to an account that has administrative access to the Exchange environment. (The account should have permissions to disable/enable the SendConnectors)

image_10

2. A configuration tool. If all goes well, you should see an icon the the Configuration tool on your Desktop.

image_4

Send Connector actions

While logged on with a user account that has access to the Exchange environment, launch the Configuration Tool. While the application starts, the list of SendConnectors will be obtained using PowerShell, so it might take a couple of seconds before the application is displayed.

image_8

First, you need to think about the purpose of the Send Connectors. Pick the SendConnector that allows the transfer of emails to the internet towards your local SMTP Relay server.†

When the internet connection is down, you would want to disable that connector. Select the connector in the upper part, right click and choose "Disable Connector".

When the internet connection becomes available again, you’d want to re-enable this connector. Select the connector in the lower part of the window, right click and choose "Enable Connector".

Internet connection detection

Next, you need to specify how the application can determine whether the internet connection is up or down.

Open the "Internet Connection Detection" tabsheet

Right-click in the listview and choose "Add" to create a new connection check. You can add 3 types of tests : either a ping to an IP address or hostname, perform a HTTP connect to a webpage (http or https) or do a tcp connect to an open port. (Make sure to enter the correct syntax for this last type of test. You need to specify the hostname, followed by :, then followed† by the port number)

image_12

Make sure to pick target hosts that are thrustworthy and make sure that you have received permission to use these hosts as a target, if applicable.

The set of connection detection checks will determine whether the internet connection is down.

If you get a warning about the fact that the config file cannot be saved, then the service probably has locked the config file. Either wait a couple of seconds and try again, or stop the service, save the changes, and start the service again (which is in fact the recommended way)

General settings

In the third tabsheet, you can change the behaviour of when and how the tool should decide whether a connection check has succeeded or not, and whether the internet connection should be considered up or down.

image_13

Ping Settings :

– Set number of pings that should be sent to each target host.
– Define the minimal number of pings that should work, otherwise the test will conclude that the ping has failed
– Set the ping timeout

Internet Connection Detection Behaviour :

– Consider internet connection down when …. or more tests failed.† This setting is related to the entire set of connection checks that are defined in the second tabsheet. Suppose you have defined 3 checks. If you want to have the tool decide that the internet connection is down when 2 or more out of these 3 checks have failed, then set this value to 2
– Number of internet connection check runs before state can be changed : If you want to make sure the internet connection is really down, you can combine the results of multiple ‘runs’/iterations to really make sure the line is down.† The results of each set of connection checks will then be gathered.†
– The next setting "Within those runs, number of failed results…"† allows you to set a threshold within the iterations.
– The 4th setting allows you the specify the frequency (or interval) that is used by the tool to run the connection checks.

Ok, all of this may sound a bit confusing, so let’s create an example.

Suppose you have set the parameters to the same values as in the screenshot above, and suppose you have created 3 connection checks.† So when will the connection state change from "Internet is up and running" to "Internet connection down" ?

Hour 0:00 : Run 1
3 tests are run. If 2 or more of these tests fail, then remember that run 1 out of 2 has failed.
Hour 0:02 : Run 2
3 tests are run. If 2 or more of these tests fail, then remember that run 2 out of 2 has failed.

So 2 runs have been made, which is the minimum to trigger a state change.†† 2 out of these 2 runs have returned "failed connection", so the state is changed to down.

Maintenance window:

You can prevent the tool from checking connection states during a given interval. Specify the start and end time, and choose the days of the week on which the windows should be active.

Version updates :

It is strongly advised to turn on "Automatically check for updates". If you have enabled admin alerts and specified SMTP settings (see below), then you will be notified when a new version is available.† Version update checks will happen every 10 hours.

Logging and Alerting

If this is not clear yet, you can use the "Logging and Alerting" section, and enable "Log internet connection changes in event log", and enable "Verbose logging in event log" so you can see how the application works.

Open the Application Event Log and you’ll see every step that is taken by the application to determine line state, and will show you the actions it has taken based upon these results.† Make sure to disable the verbose logging again, as this will fill up your event log very fast.

If you want email reports, then enable "Send admin alerts to (email address):" and enter a valid email address. You can specify multiple addresses by separating the addresses with semi-colon.

Next, at the lower part of the tabsheet, you’ll need to specify the SMTP settings. Use the "Test SMTP connection" button to verify that the settings are correct.

image_18

Don’t forget to SAVE the changes.† After saving the changes, you should restart the service† (or just start the service if this is the first time configuration). Have a look at the event log and make sure the application has not generated any errors.

Final note : make sure to start the service when the internet connection is up.† When the service is started, the tool assumes that the connection is down. Shortly after starting the service, the status should go from DOWN to UP.† If you have enabled admin alerts, you will get an email stating that the state has changed.† This proves that the tools works and that admin email alerts are configured correctly.

© 2008 – 2009, Corelan Team (corelanc0d3r). All rights reserved.

Related Posts:

22 Responses to Free Tool – Exchange 2007 Outbound SMTP gateway redundancy

  • frederik_larsen says:

    Hi,
    Will this also work with Exchange 2003?
    Or do you know of any other tool that can do this on Exchange 2003?

    Regards

    Frederik Larsen

  • frederik_larsen says:

    Hello,
    okay, thats a shame.
    My setup is that i have an Exchange 2003 Cluster, that is setup with one smtp iis server as smart host. The smart host sends mails out over the internet.
    But i have 2 ISP’s and i would like to make it redundant, so we can always mail out, nomather if one of the ISP’s is down.
    Any good ideas?

  • Hello Frederik,

    This tool only works with 2007 (it uses powershell commands in the background)… I have not seen any tools that can do this on 2003… sorry

  • are these 2 ISP’s connected to the same firewall ? or are the gateways distributed/separated from each other ?
    There may be a couple of ways to do this, but I need more info about your network setup to find a solution.
    Can you please start a thread in the discussion forum about this ?

    thanks

  • WVFleck says:

    Hi Peter,

    Does this tool use IE Proxy Settings for your Internet Connection Detection?

  • it’s possible…
    I’m using the .Net Webclient class to perform the http connect checks… I don’t know if it uses IE in the background or not – I guess you’ll have to test it

    If it is using IE proxy settings, and if you don’t want that, you can still do a tcp connect to the webserver on port 80 and you’ll have the same results
    (the tcp connect is a tcp ping, it does not use IE)

  • WVFleck says:

    Thanks for getting back to me! I know this progam should be installed on the Hub serrver but was wondering if this program can be installed on the Edge server role?

  • I don’t have an edge server at hand right now, but I guess the Edge server wouldn’t see the send connectors, would it ? I guess the tool needs to be able to see & modify the send connectors on the hub servers, so I’m afraid it won’t work on edge servers. (It should install fine on an edge server though – you can still use it as some sort of notification tool, because you can launch an application when the connection changes from up to down and vice versa)

  • zia_khan2k says:

    Hi, I don’t have any Edge server in any of the site, I am using Barracuda Spam firewall. Site A has MailBox (CCR), Hub and CAS, site B has Standby Mailbox, Hub and CAS.
    My question is, if I need any third party SMTP relay to make this scenario work?

    Thanks,
    Zia

    • Technically no, but for security reasons, I would use a dedicated SMTP relay in the DMZ
      What server is accepting the emails ? the HUB&CAS ? Or the Barracuda ?
      Do you have a Barracuda in both sites ?

  • zia_khan2k says:

    Hi,

    Thanks for very quick response!!
    What SMTP relay server do you recommend?

    It is like this:

    Internet->Barracuda->HUB&CAS

    Thanks again,
    Zia

    • well, there are many smtp relay solutions… I guess it all depends on the budget you have
      If you don’t have any budget, perhaps you can
      – use your ISP mail server for outgoing emails, and let your mails arrive on the mailserver of your ISP (and then only allow those ISP servers to deliver email to your mailserver)
      – use a hardened sendmail/postfix solution
      With more budget, there are a number of commercial solutions that will work well :
      – Cisco Ironport
      – MailMarshal
      – …

      • zia_khan2k says:

        Hi again,

        Thank you very much for all your help.
        Using this solution, what happens if all server goes down or internet goes down at site A? All mails will be delivered in standby mailbox in site B?

        Thanks,
        Zia

        • it depends on MX records

          If both sides have a smtp relay, and the email domain has 2 MX records (one pointing to the server in site A and the other pointing to the server in site B), then mails will be delivered to the server in site B.
          If the mailbox server in site B is configured to take over when the mailbox server in site A goes down (using some sort of clustering, then mails will be delivered to the mailboxes on the server in site B
          If internet goes down in site A (and site B has internet + smtp server + proper MX records), then mails will be delivered to site B, and then transferred to the mailbox server in site A.

  • zia_khan2k says:

    Hi thanks again.

    Is there any way to automate the standby mailbox in site B to takeover when site A goes down?

    Thanks,

    Zia

  • zia_khan2k says:

    Hi Peter,

    At site A, we have CCR locally and at site B, I have SCR target.

    Thanks,

    Zia

    • As far as I know, there is no way to automate the failover or failback between site A and site B (SCR)
      It’s a manual process (which, imho, you should practice on a regular basis, as it can be quite complex)

  • zia_khan2k says:

    Hi Peter,

    So it means we must use CCR, one CCR node at site A and another CCR node at site B?

    Thanks,
    Zia

    • for the mailbox servers that would be an option if they are running 2008 server (version that supports clustering across different subnets), and if bandwidth allows for the CCR sync
      If you already have set up the servers, you’ll have to break them down again, to install them in a CCR cluster I think
      If mailbox server in site A goes down, mailbox server in site B takes over

      If HUB/CAS in site A goes down, you need to redirect clients to the HUB/CAS in site B. So you may want to use a virtual fqdn for the CAS servers… so if cas in site A goes down, you change the IP address behind the virtual fqdn, point it to CAS in site B, and everybody will be able to connect again

  • zia_khan2k says:

    Hi Peter,

    Thanks a lot for your responses. Let me explain you what I want to accomplish, I am sure you can help me to achieve that.

    1. I don’t want to miss any messages in the period when everything was down in Site A.
    2. I want to give access to the users for the messages when the servers were down, maybe using web pop or imap.
    3. I am OK if I have to have a server at some other location.

    This is what our management want!

    Please advise if you have some ideas.

    Thanks,
    Zia

    • Hi,
      I don’t consider myself an Exchange expert, so I would only be able to give you my personal view
      Perhaps you should post this question in an Exchange newsgroup – you’ll get more accurate results & alternatives so you can decide what to do

      I guess a lot depends on the amount of downtime you can afford. I don’t see an issue where you could loose a lot of emails when everything was down in Site A. Messages would be queued on the various hosts (remote smtp servers would queue messages, outlook clients would queue messages, etc)
      If you want to give access to the users when the servers are down in site A, you need a hub/cas & Mailbox server in site B, which is replicated with server A. Then when site A goes down, clients would need to be reconnected to site B. You could set up OWA using a different hostname on site B, so they can use the second OWA URL to access their mailbox in site B. Should work without intervention
      Again, this is my personal view..

Corelan Live training

Since 2011, Corelan GCV has been teaching live win32 exploit dev classes at various security cons and private companies & organizations.

You can read more about the training and schedules here

Demand Global Change

The world needs your help !

Please take a few moments to read the "Demand Global Change Call For Action" document at
http://bit.ly/demandglobalchange
Read the full document at
http://bit.ly/demandglobalchange_full and share the message with as many people as possible.

Like the Facebook page, and SHARE it with everyone you know.



Donate

Want to support the Corelan Team community ? Click here to go to our donations page.

Want to donate BTC to Corelan Team?



Your donation will help funding server hosting.

Protected by Copyscape Web Plagiarism Tool

Corelan Team Merchandise

You can support Corelan Team by donating or purchasing items from the official Corelan Team merchandising store.

Corelan on IRC

You can chat with us and our friends on #corelan (freenode IRC)

Categories