Exchange cannot send mail to some domains

Have you encountered that Exchange cannot deliver mail to some destinations on Internet? This is becoming more and more common. You may ask why this happen in the first place and why it’s becoming more common.

The answer lies in how Exchange do DNS queries. Exchange was designed to run on a corporate network where you have full control on how DNS is setup and configured. Basically Exchange believe that DNS will always respond with a correct answer. But when Exchange send mail to Internet, DNS queries and answers might not always be what you expect, this is especially true when more and more organizations start using IPv6.

Using Network Monitor or any other network sniffer when Exchange tries to deliver a mail over Internet you will see that there is query for MX. One problem here is if the destination domain have IPv6 information in their Internet DNS but do not have AAAA records for hosts specified in their MX records, Exchange will simply do not another query for A records for the MX hosts and mail will queue on Exchange.
There are variations on what information is in the destination DNS zone and how your DNS is configured, if you have IPv6 yourself etc, but the behavior is the same, DNS will sometimes fail.

Solution is very simple. configure the sendconnector used for sending mail to Internet to use an external DNS, that is not to reconfigure your windows box to query another name server but simply use the Exchange configuration
Set-SendConnector <SendConnectorNameToInternet> -UseExternalDNSServersEnabled $True

You don’t even have to specify a specific name server on your HUB/Edge server, but you can if you like.

This will change the behavior of Exchange DNS queries to not to stop when there is no AAAA records if other IPv6 information is found, but to continue to do IPv4 DNS queries. remember that Windows prefer IPv6 over IPv4. This can be verified by using a network sniffer.

Have done testing both with HUB and Edge servers and with Exchange 2010 and 2013, and the behavior is the same.

The only reference on Technet on this matter is this article http://technet.microsoft.com/en-us/library/bb676467(v=exchg.150).aspx that talks about normal and lenient mode, but it doesn’t explain the changed behavior if using external DNS or not.

So in short, configure your sendconnectors sending to Internet to use an external DNS to make your live easier.

Looking for Exchange 2013 CU3 ?

look no further other than KB2892464. As usual it contains several bugfixes, support for IE11 in OWA, less memory consumption for the search infrastructure, bug around backup and restore which you can read more about in KB2888315.
To install CU3, you must deploy schema updates so talk with your Active Directory team to have them deployed before you run the CU3 setup.
Download is found here

Microsoft also released Exchange 2010 SP3 UR3 which can be found in KB2891587 and download from here

As always, read the KB and notes carefully before deploy.

happy patching.

Microsoft Masters program canceled

I June I did my rotation of the Exchange Masters training and also passed the test to become Exchange Master, MCSM (Microsoft Certified Solution Master). Training and test are intense and I am both glad and proud to have gained a lot of knowledge and new friends together with the MCSM title. so life is good with med and the Masters community until an email from Microsoft Learning sits in my inbox on Friday saying that the Masters and Architect programs are to be canceled. Read Neil Johnson (who is one of the teachers at the training) blog for the full email.

It has been a very intense day of people expressing their feelings and thoughts about this and the overall saying is that this is cannot be true it must be a very bad joke. Personally I couldn’t agree more. Cancel the highest certification you can achieve on Microsoft technology is something you simple don’t do, no matter what. Think about what signals this send out to people out there.

See what others are writing on the subject on Internet:
http://www.stevieg.org/2013/08/are-microsoft-losing-friends-and-alienating-it-pros/

http://paulrobichaux.wordpress.com/2013/08/31/microsoft-certified-systems-master-certification-now-dead/

http://up2v.nl/2013/08/31/microsoft-retires-its-top-level-certifications-mcm-mca-and-mcsm/

http://www.devinonearth.com/2013/08/defending-a-bad-decision/

http://michaelvh.wordpress.com/2013/08/31/microsoft-is-retiring-the-mcsmmca-program/

A SQL master guy even posted suggestion to vote on Connect site, sadly the connect site has been up and down the past 24 hours.

Windows 2012 shortcuts

As much as I like Windows Server 2012 I also dislike the logoff/reboot/restart functionality because the are very hard to do when you use remote desktop to your server, fiddling with your mouse in the corners trying to get something to click on.

Thinking about this I searched Internet and discovered this: http://gallery.technet.microsoft.com/scriptcenter/Create-a-ShutdownRestartLog-37c8111d which is fine. I took this script and did some small adjustment  (adding Windows Update and shortcuts to desktop)

Get the script DesktopLinks.ps1

Another version 2 update again

Exchange has a long history of being a solid product, of course it has been bugs in it but they have been addressed with patches and service packs. But the last year or two the set has changed, who doesn’t remember the series of version 2 UR for Exchange 2010 SP about two years ago. My thinking of this is that Microsoft has changed focus from delivering Exchange as regular product with a roadmap of looking forward to a service pack that has a set number of bug-fixes and features. With the service in mind this is not the case to the same extent but rather that the service has become a living thing that constantly are patched and upgraded. With this approach, developers are constantly tossing in new fixes and functionality that sometimes break something else. One reason for this could be that Microsoft is trying to introduce a new feature or fix a problem really fast which is good but it looks like Microsoft is more focused on speed instead of quality it might also be that Microsoft trust and rely on the Managed Availability not to fix the problem but restore functionality for the end-user.
Both myself and customer that for some years now has been comfortable with applying patches as they comes out of Redmond compared to what was the case 15 years ago when patching was something that wasn’t that common. This fear of patching is now back in the Exchange world because of the rollups and CU’s that correct some stuff but also introduce others. Of course the message is to try patches in your lab, but to be honest who has time and can spend money on a lab that mimics your production system? Labs are almost every time smaller and you cannot try everything users do in production. With this several of my customers has now taken a step back and not applying Exchange patches as fast as they would like because of fear breaking things.
Nevertheless Microsoft has now release a version 2 of Exchange 2013 RTM CU2. It mainly addresses the problem with mail enabled Public Folder permissions introduced in CU2 http://blogs.technet.com/b/exchange/archive/2013/07/12/e2013-rtm-cu2-issue-public-folder-permissions-loss-after-pf-mailbox-move.aspx.
The new version 2 of CU has build 712.24 compared to the CU2 712.22. If you already have deployed CU2 you can simply run “setup.exe /m:upgrade /IAcceptExchangeServerLicenseTerms” to install CU2v2.
Download link Exchange 2010 RTM CU2 ver 2 and look on the details to see that you download the correct CU2.
Read Microsoft Exchange team announcement http://blogs.technet.com/b/exchange/archive/2013/07/29/now-available-updated-release-of-exchange-2013-rtm-cu2.aspx

RBL and Exchange 2013

When you install the Antispam agents on Exchange 2013 servers you get all of them installed like you did for previous versions of Exchange server. most of them will get installed on the mailbox role but not the Connection filtering agent aka. RBL, DNS Block List etc.

The powershell script: install-AntispamAgents.ps1 will look for which server role is installed and will not install Connection filtering if the server hold the mailbox role. This is understandable since SMTP connection should come in from the CAS server and then the original sending IP will not be show since CAS do Source-NAT. So the logic would be to install the connection filtering agent on CAS but the install script will not let you do that either. Connection Filtering will only install on Edge role.

I can only speculate why this is, but either Microsoft want it to be like this or they have found some trouble with the Connection Filtering Agent running on CAS.

I figured I will give this a try anyway, and here is how you get it to work.

Start Exchange Management Shell as administrator.

Change Directory to scripts folder.
cd $exscripts

Install the agent.
Install-TransportAgent -Name "Connection Filtering Agent" -TransportService FrontEnd -TransportAgentFactory "Microsoft.Exchange.Transport.Agent.ConnectionFiltering.ConnectionFilteringAgentFactory" -AssemblyPath "C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\agents\Hygiene\Microsoft.Exchange.Transport.Agent.Hygiene.dll"

If you have multiple agents running on the frontend transport you must set them in the correct order with the priority parameter

Add a IPBlocklistprovider of your choice
Add-IPBlockListProvider -Name zen.spamhaus.org -LookupDomain zen.spamhaus.org -AnyMatch $true -Enabled $true

You can add more than one provider if you like. If you Don’t provide a custom response it will be “Recipient not authorized, your IP has been found on a block list”

Enable the agent
Enable-TransportAgent -TransportService FrontEnd -Identity "Connection Filtering Agent"

Restart FrontEnd transport service
Restart-Service MSExchangeFrontEndTransport

Now the agent should be live and kicking. Logging for the frontend agent is here “C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\FrontEnd\AgentLog” instead of the directory for the backend transport “C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\Hub\AgentLog”

Since the script don’t install the Connection filtering agent on CAS it is probably unsupported to install the agent manually, but I had it running for months without any problem so make your own judgment.

Exchange 2013 Admin Center URL’s

You have just installed Exchange 2013 in your Exchange 2007 or 2010 organization. You start a browser to access the Exchange Admin Center, the Exchange GUI admin tool, point the browser to https://servername/ecp and login with an account you know have permission in Exchange but are prompted with an error message.
Reason for this is that the mailbox you try to logon to is not located on Exchange 2013. solution is to move the mailbox to Exchange 2013 mailbox server or change the URL to “https://casservername/ecp?ExchClientVer=15”. This will tell EAC to show the Exchange 15 version of EAC instead of the Exchange 14 version which should be shown because the mailbox is not located on Exchange 15 server.
With the new URL you get into EAC and can start configure your Exchange 2013 server and life is good until you realize that EAC will be reachable from Internet. It will still require authentication, but nevertheless, reachable.
Since you have TMG to publish Exchange with you figure that you can deny access to /ecp URL from Internet but this will unfortunately stop users from accessing the OWA options web.

You find out that there is a parameter “AdminEnabled” on the ECP website to disable EAC. By setting AdminEnabled to False. Sadly this option disable EAC completely and you now only have the Exchange Management Shell to use and most people want a GUI to manage Exchange.

Solution is to create a new website that is not reachable from Internet but only from Internal network. Easiest is to change the listening port on the new website.

here is what you need to do.

# port used for the EAC website
$port = 9443

# create a new folder to host the new website
mkdir C:\EAC

# create a new webiste
New-Website -Name EAC -PhysicalPath C:\EAC -Verbose -Ssl -Port $port -Id $Port –ApplicationPool MSExchangeOWAAppPool

Then you must assign a certificate to the website. This can be done on the bindings option on the newly created EAC website options.

#create FW rule to allow traffic to website
New-NetFirewallRule -Name "EAC website" -Description "Exchange Admin Center website" -DisplayName "EAC website" -Protocol TCP -Profile Any  -Action Allow -LocalPort $port 

and then create the ECP applications in the EAC website.

#hostname
$hostname = ([System.Net.Dns]::GetHostByName(($env:computerName))).hostname

#InternalUrl
$IntUrl = (Get-EcpVirtualDirectory -Server $hostname).InternalUrl.tostring()

# Get path from the original ECP website
$DirPath = (Get-EcpVirtualDirectory -Server $hostname).path

# Create new EAC web
New-EcpVirtualDirectory -WebSiteName EAC -Server $hostname -InternalUrl $IntUrl -Path $DirPath -Role ClientAccess -AppPoolId MSExchangeECPAppPool

# Finally , diable EAC on the default ECP app
Set-EcpVirtualDirectory $hostname\'ecp (default web site)' -AdminEnabled $false

New-ECPVirtualDirectory states that you must create a OWA application also, but I have not encountered any problem by not doing this. The only problem I have when doing this is that browsing in the OU structure in AD when creating new mailboxes don’t work. have tried both with the above settings and also by creating the OWA appl. as suggested but it simplyu don’t work either way.
Simply put the commands in a powershell script and run it from EMS. when done you can access EAC with the new URL “https://casname:9443/ecp”.
Tips: when later changing URL and certificate on your CAS, you should also change them in the EAC website to make everything work correctly.

This is most likely unsupported but I have it running for several months without any problem except for the browsing thing in EAC.

Irrational behavior of Exchange 2013 receive connectors

Scenario: Exchange 2013 server with both mailbox and client access role combined.
You create a receive connector and scope it down to some specific IP addresses for applications running on servers with the specified IP. You configure to allow anonymous relay on the connector.

Applications on server are happy since they can now anonymously submit and relay messages off your Exchange installation to Internet.

Suddenly after a few hours, applications cannot send and relay mail.
Luckily you have protocol logging enabled and discover that your newly created connector is not used anymore. some more troubleshooting later and you desperately restart Exchange transport service. After the restart mail flow is restored the way you want.

But again after some hours, mail flow stops. further investigation shows that exactly the same thing has happened again, another restart of Exchange transport get things going.

Solution: Restart the transport service with a few hours interval, nahh don’t think so. Exchange 2013 is brand new and should work, and before the migration you had the exact configuration with Exchange 2010 server.

Real solution: I very simple. remove the old receive connector and create a new one, but connect it against the frontend transport instead of the default Hub transport (backend).
Transport system has been running fine as it should for a couple of weeks now. I believe this is a bug in the transport service and the symptom is selecting the wrong receive connector when there is an incoming connection to Exchange.

It makes sense to connect receive connectors and possibly send connectors against frontend (CAS server) but I think it should work equally fine if you select the backend (mailbox server) especially when the backend is the default option when using Exchange Admin Center.

Exchange 2007 and 2010 anti-spam automatic installation

Some organizations use the built-in anti-spam feature in Exchange 2007 and 2010. http://technet.microsoft.com/en-us/library/aa997658(v=exchg.141).aspx.

Content Filter engine is using definitions created by Microsoft and in an ideal world they are downloaded, approved and installed with help of windows update, but for many reason this is not always possible.

I created a small vbscript that uses windows update to search for only Exchange standard antispam updates and automatically install them without doing anything with other updates.

WU_Exchange_AntiSpam.vbs script can easily be scheduled to run with task scheduler.

' search and automatically install Exchange Server standard antispam definitions
' Lasse Pettersson, http://anewmessagehasarrived.blogspot.com
'
 
Set updateSession = CreateObject("Microsoft.Update.Session")
'Updatetitle to search for
updateTitle = "Microsoft Exchange Server Standard Anti-spam Filter Updates"
 
WScript.Echo vbCRLF & "Searching for: " & updateTitle & "..."
Set updateSearcher = updateSession.CreateupdateSearcher()
 
'Search for all software updates, already installed and not installed
Set searchResult = updateSearcher.Search("Type='Software'")
Set updateToInstall = CreateObject("Microsoft.Update.UpdateColl")
updateIsApplicable = False
 
'Cycle through search results to look for the update title
For i = 0 To searchResult.Updates.Count-1
    Set update = searchResult.Updates.Item(i)
    If Left(UCase(update.Title),Len(updateTitle)) = UCase(updateTitle) Then
        'Update in list of applicable updates 
        'Determine If it Is already installed Or Not
        If update.IsInstalled = False Then
            WScript.Echo vbCrlf & "Result: Update applicable, not installed."
            updateIsApplicable = True
            updateToInstall.Add(update)
        Else 
            'Update Is installed so notify user And quit:
            WScript.Echo vbCrlf & "Result: Update applicable, already installed."
            updateIsApplicable = True
            WScript.Quit
        End If
    End If 
Next
 
If updateIsApplicable = False Then
    WScript.Echo "Result: Update is not applicable to this machine."
    WScript.Quit
End If
 
'Download update
Set downloader = updateSession.CreateUpdateDownloader() 
downloader.Updates = updateToInstall
WScript.Echo vbCrlf & "Downloading..."
Set downloadResult = downloader.Download()
WScript.Echo "Download Result: " & downloadResult.ResultCode
        
'Install Update
Set installer = updateSession.CreateUpdateInstaller()
WScript.Echo vbCrlf & "Installing..."
installer.Updates = updateToInstall
Set installationResult = installer.Install()
        
'Output the result of the installation
WScript.Echo "Installation Result: " & installationResult.ResultCode
WScript.Echo "Reboot Required: " & installationResult.RebootRequired

If you run this script manually, use the cscript engine because some text is written and looks better if you don’t use the wscript engine.

Exchange 2010 Service Pack 3 is out

Exchange 2010 SP3 is finally out and people have been waiting for it to be able to upgrade to Exchange 2013, but hurry slowly, Microsoft at the same time announce that to get coexistence with Exchange 2010, you not only need Exchange 2010 SP3 but also Exchange 2013 CU1 

As usual you need to update Active Directory schema before applying SP3. See the Exchange 2010 SP3 release notes.
If the Exchange 2013 CU1 will need to extend the schema once more, only time can tell.

Some useful links.
Exchange 2010 SP3 download
What’s new
Exchange 2010 SP3 UM language pack

The same time Exchange 2010 SP2 Update Rollup 6 and also Exchange 2007 SP3 Update Rollup 10

Hopefully we don’t need to see a version 2 of these patches or any new bugs introduced like we sort of have been used to the last year, sigh.