Home » Blog » Issues Reported with Exchange Server 2013 CU14 and Exchange Server 2016 CU3

Issues Reported with Exchange Server 2013 CU14 and Exchange Server 2016 CU3

Fellow Exchange MVP Jaap Wesselius has written about reports of content index failures for Exchange servers that have been installed or updated with Exchange Server 2013 CU14 or Exchange Server 2016 CU3. Those cumulative updates were released by Microsoft in September of this year.

Although I have not personally observed the issue in the field, a number of customers are reporting the same issue in a Microsoft TechNet forums thread.

One customer who opened a support ticket with Microsoft has reported back to the forum thread to say that a bug has been acknowledged, and that the solution is to deploy a new Exchange server (either Exchange Server 2013 CU13 or Exchange Server 2016 CU2) and migrate mailboxes to the new server.

If you are experiencing the issue in your own production environment then I recommend you raise a support case with Microsoft to determine whether you are impacted by the same bug (don’t rely on blog posts or forum threads to confirm a production impacting issue).

If you are considering deploying a new server as the resolution to the issue then you can refer to my like for like Exchange migration article for some high level guidance.

Update: the indexing issues are reportedly fixed in the December 2016 releases for Exchange 2013 and 2016

Paul is a Microsoft MVP for Office Servers and Services. He works as a consultant, writer, and trainer specializing in Office 365 and Exchange Server. Paul is a co-author of Office 365 for IT Pros and several other books, and is also a Pluralsight author.
Category: Blog

29 comments

  1. Phuong, Nguyen Van says:

    I have just migration from exchange 2010 to exchange 2106 cu3 full completed. But not yet migration public folder. I have a questions. Do i need move public folder from 2010 to 2016. Because the public folder do not use in exchange my organization
    Thanks

  2. Ashish Kumar says:

    It will not Impact and will not give any error message in Outlook after decommission.
    Here are registry values which can be helpful to tell outlook that there is no public folder now.

    Key:

    HKCUSoftwareMicrosoftExchangeSetup

    DWORD Value:

    HasPublicFolders

    HasPublicFolders = 0

    Paul wrote a very helpful article on decommissioning Public folder Database. That can be very helpful in your scenario.

    http://practical365.com/exchange-2010-remove-public-folder-database/

  3. Conrad says:

    I had a long contentindex problem on a fresh 2016 CU3 DAG install a few days ago..

    but I was able to resolve it with some usual/old Trick “stop services (stop-service MSExchangeFastSearch stop-service HostControllerService) and delete index”, many Update-MailboxDatabaseCopy xxxx -CatalogOnly -BeginSeed, – CatalogOnly, and some reboots..

    It seems to me that the problem was also somewhat related to circular logging…

    Anyway, after many this and that, CI was ok and still is today.

    Not sure if it is really related.. (Don’t have time to read many forum etc..) but perhaps it can help..

  4. Patrick says:

    I just migrated from exch2010 to exch2016(cu2) without issue. I then created a DAG on (2) 2016 servers running exch2016(cu3). I moved some mailboxes to new DAG DBs and cannot connect to mailboxes (password prompts, OWA hangs after login). I rebooted the DAG hosts hoping that it was just some old virtual directory configurations that needed to be refreshed. After the reboot connection issues persist and now I see a few other issues on hosts:

    1. Cannot connect to either 2016(exch2016 cu3) host via the Exch Mgmt Shell get following error:
    “[NullReferenceException]: Object reference not set to an instance of an object.”

    2. System logs is flooded with WAS EventID 5011 Warnings for ALL Exchange related app pools:
    “A process serving application pool ‘MSExchangeAutodiscoverAppPool’ suffered a fatal communication error with the Windows Process Activation Service. The process id was ‘31780’. The data field contains the error number.”

    3. HTTPERR logs constantly report errors for all Exchange App Pools:
    “Connection_Abandoned_By_ReqQueue MSExchangeServicesAppPool”

    4. The Microsoft Connectivity Analyzer reports:
    “A Web exception occurred because an HTTP 503 – ServiceUnavailable response was received from Unknown.”

    I have confirmed that the certificate looks good. If I move the mailboxes back to the 2012R2(Exch2016 CU2) host all is good.

    Any suggestions?

    • Patrick says:

      This directory also filling up with AppCrash_w3wp.exe files:

      C:ProgramDataMicrosoftWindowsWERReportQueue

      With the following contents:

      Version=1
      EventType=BEX64
      EventTime=131221402635417971
      ReportType=2
      Consent=1
      ReportIdentifier=e090d9f2-9d1e-11e6-836e-00155d005f00
      IntegratorReportIdentifier=e73c91da-1e2d-456c-bc22-260e64ff8e9d
      NsAppName=w3wp.exe
      AppSessionGuid=0000616c-0000-000d-a661-669e2b31d201
      TargetAppId=W:0000f519feec486de87ed73cb92d3cac802400000000!000054443c275eddd9bbe75c6d47cf9c24076b5d785c!w3wp.exe
      TargetAppVer=2016//07//16:02:27:22!15ee0!w3wp.exe
      BootId=4294967295
      Response.type=4
      Sig[0].Name=Application Name
      Sig[0].Value=w3wp.exe
      Sig[1].Name=Application Version
      Sig[1].Value=10.0.14393.0
      Sig[2].Name=Application Timestamp
      Sig[2].Value=57899b8a
      Sig[3].Name=Fault Module Name
      Sig[3].Value=ntdll.dll
      Sig[4].Name=Fault Module Version
      Sig[4].Value=10.0.14393.351
      Sig[5].Name=Fault Module Timestamp
      Sig[5].Value=5801a332
      Sig[6].Name=Exception Offset
      Sig[6].Value=000000000009520f
      Sig[7].Name=Exception Code
      Sig[7].Value=c0000409
      Sig[8].Name=Exception Data
      Sig[8].Value=000000000000000a
      DynamicSig[1].Name=OS Version
      DynamicSig[1].Value=10.0.14393.2.0.0.400.8
      DynamicSig[2].Name=Locale ID
      DynamicSig[2].Value=1033
      DynamicSig[22].Name=Additional Information 1
      DynamicSig[22].Value=318f
      DynamicSig[23].Name=Additional Information 2
      DynamicSig[23].Value=318f8eb61e176e217a9ae2192c7b640f
      DynamicSig[24].Name=Additional Information 3
      DynamicSig[24].Value=23de
      DynamicSig[25].Name=Additional Information 4
      DynamicSig[25].Value=23ded2732a75ed3d0d1da6bc85112415
      UI[2]=c:windowssystem32inetsrvw3wp.exe
      UI[5]=Check online for a solution (recommended)
      UI[6]=Check for a solution later (recommended)
      UI[7]=Close
      UI[8]=IIS Worker Process stopped working and was closed
      UI[9]=A problem caused the application to stop working correctly. Windows will notify you if a solution is available.
      UI[10]=&Close

      FriendlyEventName=Stopped working
      ConsentKey=BEX64
      AppName=IIS Worker Process
      AppPath=c:windowssystem32inetsrvw3wp.exe
      NsPartner=windows
      NsGroup=windows8
      ApplicationIdentity=FCBB9E96A3E01E440E0E4D9368EBD4AC
      MetadataHash=2046253764

      I excluded LoadedModule sections as it was quite lengthy.

    • There are reports floating around of trouble with Exchange 2016 CU3 on WS2016. You could log a Microsoft support case to see if it’s a known issue, or just wait to see if something is publicly confirmed.

      • DavidPM says:

        I can confirm you (and Microsoft support will do it if you open a case) that if you create a DAG 2xExchange 2016 over Windows 2016 will cause IIS failure and EMS, ECP, ect will fail. It’s still under investigation.

        Workaround: do not create DAG with Exchange 2016 over Windows 2016 , use Windows 2012 / R2 until fix release or next CU

    • Benoit Boudeville says:

      I have the same issue, hopefully it’s only a lab environment:

      Try disabling stopping and disabling the following services:
      MSExchangeHMRecovery
      MSExchangeHM

      Hope this helps,
      Benoit.

      • Patrick says:

        Yes, only testing. Unfortunately stopping/disabling those services did not solve all of the issues. It resolved the application and system log errors and warnings, but the exchange management shell will still not connect to the local exchange host. I have a ticket opened with MS. Will update if I get an answer.

  5. Onye says:

    I am also having issues with the latest Exchange 2016 CU3 update. Outlook clients are connecting fine, the problem we’re having is with our mobile clients. iOS and Android devices are able to connect and receive email, but they can’t send.

    Oddly enough the mobile MS Outlook app on both iOS and Android works

      • Onye says:

        Hey Paul, after hours of research, we went back through our initial Exchange 2016 notes and went through all the settings one by one, what we found is that any custom IIS changes that are made, were overwritten. So the changes we made to allow AD Certificate Authentication were wiped out. I went ahead and re-did the settings and rest IIS and voila my test devices started working.

        Wasted a whole Saturday on this problem…LOL, but glad it got resolved.

        P.S. ,I love this site, alot of great tools and write ups

  6. Matthew Pollock says:

    Hey Paul,

    We’re running Exchange 2013 CU14, and have had clients complaining of 0 results returned when using Outlook to perform mailbox and public folder searches.
    Would this be symptomatic of the content indexing errors reported here, or just coincidence?

    As I understand it, Exchange doesn’t index PF content as it does mailbox content – unless this behaviour has changed with modern public folders?

  7. Gary Reardon says:

    We had a problem that CU14 exposed. Our external CA OWA certificate was bound the Exchange BackEnd IIS site on Port 444. The Self-Signed Exchange cert must be used here for Exchange to work properly. Not sure how that had happened but it wasn’t an issue until CU14 and all user connectivity went away. Not a fun day…..

  8. Stephen_nehpetS says:

    I contacted MS support regarding the public folder search issue, spent approx an hour explaining the situation and demonstrating the problem. Search index was verified healthy (I rebuilt them during pre escalation troubleshooting) and all other configuration checked OK.

    MS confirmed that there is nothing publicly available regarding this problem and after insisting on a response, I received an email confirming the issue. In a nutshell, their response was to wait for CU15. They did not suggest a workaround / recovery plan. I will send anyone that wants a copy of this confirmation by email. The cost of my case was refunded.

    I did come up with a work around that may work for some people, it’s simple but blunt. In my case I need to search public folders, in addition to mailboxes, so I engaged cache mode in outlook, marked the public folder as a favorite and then added the public folder favorite to the local outlook cache. Search is working. Here are the steps:

    Switch to Outlook’s Folder List.
    Expand Public Folders and find the folder you want indexed.
    Right-click the folder and click Add to Favorites….
    Set any options or rename the favorite and click the Add button.
    Right-click on Public Folders and click Properties for “Public
    Folders”….
    Click the Advanced… button.
    Click the Advanced tab.
    Check Use Exchange Cached Mode and Download Public Folder
    Favorites.
    Click OK to close the Advanced property sheet.
    Click OK to close the Properties property sheet.
    Select your favorite folders or otherwise synchronize your client to
    download all items. This can take quite some time and is not recommended
    over remote connections.

    Now to script and deploy.

    Credit to https://blogs.msdn.microsoft.com/heaths/2006/02/16/fast-searching-in-public-folders/ for the configuration steps. I only used the steps above in my fix. Now to script this.

  9. Shannon says:

    Hi Paul,

    With this issue raised in CU 14, would it be best to stay at CU 13 ? I wasn’t sure if they would end up pulling a CU due to customer issues. We don’t typically like to go to the latest CU so we usually stay at N-1.

  10. Alexander Poettinger | xame gmbh says:

    We have an Exchange 2013 on which we installed CU14.
    We now have the issue, that the ActiveSync protocol does not accept any attachments.
    The error seems to lie on the IIS side. I activated all the protocols. When I send an email with an attachment the only log where I see the connection is the main IIS LOg.
    This is one example:

    2016-11-18 08:44:19 W3SVC1 core 172.16.0.34 POST /Microsoft-Server-ActiveSync/default.eas Cmd=SendMail&User=XXX%5Calp&DeviceId=SEC108XXXC4DB91D&DeviceType=SAMSUNGSMG935F 443 – 213.XXX.108.94 Android-SAMSUNG-SM-G935F/101.60001 – 413 0 0 229 115104 203

    In the ActiveSync protocol these connection do not appear at all.
    Only sending without attachments are delivered and appear in the ActiveSync Log.

    All the Settings about eMail sizes and ContentLengths and so forth are set to 60 MByte.

  11. Mark B says:

    Please let me know when microsoft has fix for the content index issue. I am running single echange 2013 cu14 on windows 2012 r2. I have turned off the content index for now.

  12. frank says:

    the temporary fix for this is :

    add the public folder to a favorite ( favorites can be (properties -> download favorites ) downloaded localy and the outlook search will work on them.

    BUT and this is a big BUT.. no pun intented..

    if your PF’s are 2 TB… you better think not to fav the root or you will get a 2TB local cache file and i doubt many workstations have a 2tb free for that..

    so pick and choose the PF folder you need search and fav those.

    like i said this is the MSFT turnaround untill they figure it out in cu 15

  13. Piet v.d. Hout says:

    I tried to create a DAG on two exchange 2016 CU3 on Windows 2012 R2 servers but as soon as I want to install it on the second member I get these errors in the Highavailability eventviewer on the second server:

    Cluster batch writer failed while attempting operation ‘GetClusterKey’ (error: An error occurred while attempting a cluster operation. Error: Cluster API failed: “GetClusterKey failed with 0x46. Error: The remote server has been paused or is in the process of being started”).

  14. Jason says:

    Exchange 2013 standalone Cu14 and content index failing after fail…..
    Thanks Microsoft for those wonderfull Cu’s lately and the extensive testing you do….. NOT.

Leave a Reply

Your email address will not be published. Required fields are marked *