In my Exchange Server 2010 lab environment I unwittingly created a problem for the Database Availability Group. In preparing to consolidate all of the server roles onto just two servers and implement a hardware load balancer I went ahead and decommissioned the two CAS/HT servers that previously made up the CAS array in the site.

Naturally one of those CAS/HT servers also happened to be the File Share Witness for my two-member DAG. Whoops!

Now my DAG displays a warning when I check the health of it.

WARNING: Database availability group ‘dag-headoffice’ witness is in a failed state. The database availability group requires the witness server to maintain quorum. Please use the Set-DatabaseAvailabilityGroup cmdlet to re-create the witness server and directory.

In this real world this situation may also arise if the server hosting the File Share Witness was being decommissioned, or if it had failed. Fortunately we can resolve the problem by specifying a new FSW for the DAGm which I will demonstrate here.

I’m going to use another member server within the site as my FSW, which allows me to demonstrate a related problem. The server is named HO-MGT so using the Set-DatabaseAvailabilityGroup cmdlet to configure the FSW would mean I run this command.

[PS] C:\>Set-DatabaseAvailabilityGroup dag-headoffice -WitnessServer ho-mgt -WitnessDirectory C:DAGFSW

However in this case I get an error.

WARNING: The Exchange Trusted Subsystem is not a member of the local Administrators group on specified witness server
ho-mgt.
WARNING: Insufficient permissions to access file shares on witness server ‘HO-MGT.exchangeserverpro.net’. Until this problem is corrected, the database availability group may be more vulnerable to failures. You can use the Set-DatabaseAvailabilityGroup cmdlet to try the operation again. Error: Access is denied
Unable to change the quorum for database availability group dag-headoffice. Witness server ‘\HO-MGT.exchangeserverpro.netdag-headoffice.exchangeserverpro.net’ network name wasn’t found. This may be due to firewall settings.
+ CategoryInfo : InvalidArgument: (:) [Set-DatabaseAvailabilityGroup], DagTaskProblemC…ptionBadNetName
+ FullyQualifiedErrorId : 75321C4E,Microsoft.Exchange.Management.SystemConfigurationTasks.SetDatabaseAvailabilityGroup

If you were running the same command but specifying another Exchange 2010 server to be the FSW you would not receive that error. This is because Exchange servers trust each other to perform this type of administration, thanks to a group called Exchange Trusted Subsystem.

All of the Exchange 2010 servers have this group as a member of their local Administrators group, for example here the local Administrators group of one of my DAG members.

Exchange Trusted Subsystem group in local Administrators
Exchange Trusted Subsystem group in local Administrators

So the solution is to add the Exchange Trusted Subsystem group to the local Administrators group on my HO-MGT server, and then run the Set-DatabaseAvailabilityGroup command again.

After running the command you can see that Exchange has created the folder and shared it on the FSW server, no need to manually create the folder or set any permissions yourself.

Database Availability Group File Share Witness directory
Database Availability Group File Share Witness directory

Now when checking the health of the Database Availability Group you should not receive any warnings about missing File Share Witness servers.

[adrotate banner=”49″]

About the Author

Paul Cunningham

Paul is a former Microsoft MVP for Office Apps and Services. He works as a consultant, writer, and trainer specializing in Office 365 and Exchange Server. Paul no longer writes for Practical365.com.

Comments

  1. sachin

    I ran into a situation where one one of my DAG member server has disk failures on direct attached storage since this is quite old hardware we are not going to replace it rather than that we planned to reduce the 3 member DAG to 2 member DAG however i noticed that cluster quorum is configured as node majority to achieve 2 member DAG do i have to first remove failed member from DAG and then do reconfiguration of cluster quorum to node and file share majority ?

  2. Ozz

    I have tried to change the Witness server and have an error.
    I used the following command:

    Set-DatabaseAvailabilityGroup -Identity dag1 -AlternateWitnessServer fmdwbes2 -AlternateWitnessDirectory c:fsw

    I’m getting an error that makes no sense to me, as there is no Account ‘FMDDAG1$’ :
    A database availability group administrative operation failed. Error: File share witness path ‘\FMDBES2.ad.fasken.fmd
    DAG1.ad.fasken.fmd’ exists, but permissions are not correctly set. Account ‘FMDDAG1$’ must have full access.
    + CategoryInfo : InvalidArgument: (:) [Set-DatabaseAvailabilityGroup], DagTaskFswNeedsCnoPermissionExcept
    ion
    + FullyQualifiedErrorId : 7EE051D,Microsoft.Exchange.Management.SystemConfigurationTasks.SetDatabaseAvailabilityGroup

    1. Avatar photo
      Paul Cunningham

      FMDDAG1$ is probably the computer account associated with the DAG/cluster.

      You’ll need to configure permissions on the new FSW. Have you added Exchange Trusted Subsystem to the local Administrators group of the new FSW?

  3. Edwin Mendez

    When I create a DAG replica on another server, what directory on the replica server is used for the replicated DB? In other words, where does Exchange put the replica and is there a way to change the target location of the replica?

    1. Avatar photo
      Paul Cunningham

      Each DAG member stores its copy of the database in the same path. So all DAG members must have the same storage configured. There’s no way to change that behavior.

  4. tariq

    Unable to change the quorum for database availability group GABDAG1. Witness server ‘\gab-dhcp1.gab.gov.saGABDAG1.gab.gov.sa’ network name wasn’t found. This may be due to firewall settings.

    ——————————————————————————–

    The Exchange Trusted Subsystem is not a member of the local Administrators group on specified witness server gab-dhcp1.gab.gov.sa. Error: Access is denied

    any help please

    1. Avatar photo
      Paul Cunningham

      The Exchange Trusted Subsystem group needs to be made a member of the local Administrators group on the file share witness server.

  5. Nick Boulamatsis

    Hi Paul. This is my first time asking a question but have read a lot of your postings and PDF books. Also, I see that there has been a bit of discussion about this but I did not see a reason or resolution to the issue.
    I have moved my FSW to a new server successfully using the below command:
    Set-DatabaseAvailabilityGroup -Identity MYDAG -WitnessDirectory C:mydag-witness -WitnessServer NewWitnessServer -Verbose

    When I run this command, the correct, new, FSW is listed:
    Get-DatabaseAvailabilityGroup mydag | fl

    However, when I run any of the following commands, the listing is the old server still.
    Get-Cluster nfapdag | Get-ClusterQuorum | FL
    Cluster.exe MYDAG /quorum

    When I run this command I get a mix of results, Old and New server are listed:
    cluster MYDAG res “File Share Witness (\OldWitnessServerMYDAG.mydomain.com)” /priv
    Results:
    Listing private properties for ‘File Share Witness (\OldWitnessServer.mydomain.comMYDAG.mydomain.com)’:

    T Resource Name Value
    — ——————– —————————— ———————–
    S File Share Witness (\OldWitnessServer.mydomain.comMYDAG.mydomain.com) SharePath \NewWitnessServer.mydomain.comMYDAG.mydomain.com
    D File Share Witness (\OldWitnessServer.mydomain.comMYDAG.mydomain.com) ArbitrationDelay 6 (0x6)

    I should also mention that in my Failover Cluster Manager, when I click on my domain name, the “Quorum Configuration” lists the correct “Node and File Share Majority” and the NewWitnessServer.

    I’m hoping you could shed some light on why there would be this set of mixed results, and does this clear up or do I have to intervene somewhere?

    1. Nick Boulamatsis

      Sorry, \NewWitnessServer.mydomain.comMYDAG.mydomain.com should actually be:
      \NewWitnessServer.mydomain.comMYDAG.mydomain.com

      1. Avatar photo
        Paul Cunningham

        The name/label in cluster manager doesn’t get updated when you use Set-DatabaseAvailabilityGroup to move the FSW. As long as the new witness server and path appear in Get-DatabaseAvailabilityGroup you should be fine.

  6. Edgar Valle

    Hi Paul

    I’ve tried to add members to my DAG so many times. I’ve my DC, my first machine has HT, CA and MB the other one has MB. So at the end my error is

    The operation wasn’t successful because an error was encountered. You may find more details in log file “C:ExchangeSetupLogsDagTasksdagtask_2016-03-17_19-33-45.729_add-databaseavailabiltygroupserver.log”.
    [2016-03-17T19:33:51] WriteError! Exception = Microsoft.Exchange.Cluster.Replay.DagTaskOperationFailedException: A database availability group administrative operation failed. Error: The operation failed. CreateCluster errors may result from incorrectly configured static addresses. Error: An error occurred while attempting a cluster operation. Error: Cluster API ‘”CreateCluster() failed with 0x13c1. Error: The cluster IP address is already in use”‘ failed. —> Microsoft.Exchange.Cluster.Replay.AmClusterApiException: An Active Manager operation failed. Error An error occurred while attempting a cluster operation. Error: Cluster API ‘”CreateCluster() failed with 0x13c1. Error: The cluster IP address is already in use”‘ failed.. —> System.ComponentModel.Win32Exception: The cluster IP address is already in use
    — End of inner exception stack trace —

    My NIC configuration in

    DOMAIN CONTROLLER
    IP 192.168.160.2
    SUBMASK 255.255.255.0
    GATEWAY 192.168.160.1.
    DNS 192.168.160.2

    EXCH2010-01 HT, CA and MB
    IP 192.168.160.3
    SUBMASK 255.255.255.0
    GATEWAY 192.168.160.1
    DNS 194.168.160.2

    NIC FOR DAG
    IP 10.0.0.1
    SUBNET 255.255.255.0
    GATEWAY
    DNS

    EXCH2010-02 MB
    IP 192.168.160.4
    SUBMASK 255.255.255.0
    GATEWAY 192.168.160.1
    DNS 194.168.160.2

    NIC FOR DAG
    IP 10.0.0.2
    SUBNET 255.255.255.0
    GATEWAY
    DNS

    I’ve tried to change my IP so many times.

    Could you help me.

    1. Avatar photo
      Paul Cunningham

      First, you don’t need those “NIC for DAG” interfaces. DAGs will run fine with a single NIC on each member, and it simplifies your deployment quite a lot.

      Secondly, this seems to be your problem:

      “The cluster IP address is already in use”

  7. Jeff Kamen

    Hello Paul,

    Simple question regarding the Exchange 2010 FSW. I need to move the location of the FSW to a different server, and my question is simply does/will this take the DAG offline at any point? Straightforward question, but all of my searching has turned up empty.

    Thanks!

    1. Avatar photo
      Paul Cunningham

      No, moving the FSW will not take the DAG offline. As long as the DAG still has quorum (majority of nodes online) it will stay up.

  8. Patrick Cronin

    Hey, Paul – I am running two Exchange 2010 servers on Windows 2008 R2. Both are configured with Mailbox, CAS and Hub Transport roles. Also have a DAG. Are you aware of any circumstances where more than one witness server would be needed or desirable?

  9. Alan

    I’m having an issue when trying to move my witness server. The existing witness server is due to be decommissioned, and we want to move the witness share to a new server. However, when we do, Exchange says all is ok, but the Failover cluster manager says the file share witness is offline. If I try to bring it online, it simply says “the cluster resource could not be brought online by the resource monitor”. If I move it back to the old server, it’s fine. The permissions and group membership on the new server are all correct, and it does create the files in the new share. What could be causing this?
    thanks

    1. Avatar photo
      Paul Cunningham

      The name of the resource in Cluster Manager doesn’t change even when you change the witness server.

      Other than that you’ll need to explain the steps you’ve followed to configure the DAG to use a different witness server before I can offer any suggestions.

      1. Alan

        I modified the witness file location in the DAG properties GUI (both server name and drive path). Exchange thinks about if for several minutes, then completes without any errors or warnings. Refreshing the EMC shows the new location is set. The new file share is populated with a long-named folder, and a ‘failoverclusters.temp’ folder. All look fine. But when I open the failover cluster manager, in the cluster core resources section, the File Share Witness is listed as off-line: the new file share is listed.
        It’s Exchange 2010 running on Server 2008 R2.

        thanks

        Alan

  10. Mica Roth

    Hi Paul,
    I was hoping you could help me with a strange issue I am having. this past weekend we performed our first DR test of our Exchane environment, and I ran into a strange issue. we have a 6 node DAG with 4 nodes + FSW in primary site, and 2 nodes + AFSW in DR site in DAC mode. I powered off the primary site servers, and attempted to bring the DR site online. everything appeared to go pretty well, but then I get an error stating that the AFSW was in ‘failed’ state.

    Microsoft.Exchange.Cluster.Replay.DagTaskFileShareWitnessResourceIsStillNotOnlineException: A database availability group administrative operation failed. Error: File share witness resource ‘\AlternateFSWShareName’ couldn’t be brought online. The current state is ‘Failed’

    I verified that the share was accessible from both DR DAG nodes. I searched everywhere but can’t find any documentation on a similar situation. Any help you could provide would be much appreciate.

    1. Avatar photo
      Paul Cunningham

      Is Exchange Trusted Subsytsem group in the local Administrators group for the alt FSW?

      1. Mica Roth

        Yes, the AFSW is the CAS in the DR site. I checked to ensure Exchange Trusted Subsystem was in local admins group

        1. Mica Roth

          Also, I tried using a non-exchange server as the AFSW and added the Exchange Trusted Subsystem to the local admins group, but that did not work either. Is there a way to test the ability to bring up the AFSW without performing an actual failover test, or causing any service outages?

        2. Avatar photo
          Paul Cunningham

          You can try bringing the resource online using the Failover Cluster Manager. Or look at the event log for any evidence of why the resource won’t come online.

  11. charlie

    hi paul,

    great site!

    quick question on this topic, does the FSW have to be Windows Server 2008 for Exchange 2010? i have tried with a Windows 2012 server and although I don’t get any errors when executing the directory never gets created.

    thanks again and keep up the great work!

      1. charlie

        I run Set-DatabaseAvailabilityGroup dag-dagname -WitnessServer servername -WitnessDirectory C:DAGFSW

        I don’t get any errors and a get-databaseavailabiltygroup – status | fl shows the new witness server and directory but the actual folder never gets created on the member server.

        1. charlie

          there were originally 2 and now there is 3!

  12. Andrew

    Hey Paul,
    Yesterday, my FSW failed to do its job. I have a two node EX2010 cluster running on Win2008R2. Each node is in a different physical datacenter. The design is intended to be active/passive (not active/active), with DC A being primary/active. The two datacenters lost communication for a brief period, and all mailbox databases (hosted at DC A) were put offline. The cluster event at the time registered a critical event: “File share witness resource … failed to arbitrate for the file share …” My FSW is on a physically separate server in DC A Exchange in DC A and the FSW did not lose communication.

    I have seen a few sites talk about the importance of extending permissions on the FSW to include Exchange Trusted Subsystem (in addition to the DAG$ group). At the time of failure, the subsystem permission was not in place (the cluster wizard created all of this automatically). I have since extended the permissions are recommended for the subsystem.

    However, I have to lingering thoughts that are bothering me.
    1) Before extending the permissions, I ran a cluster validation, which included checking the FSW. The cluster passed.
    2) Before extending the permission, I looked at the write dates of the FSW files, and it was clear given the timestamps that the files are being actively written to – which is what you expect in a healthy environment.

    How can it be that the two points above could also co-exist with the failure of the FSW to arbitrate a potential split-brain when it becomes “crunch time”?

    1. Avatar photo
      Paul Cunningham

      Did you validate that your DAG performed as expected in that failure scenario before putting it into production?

      1. Andrew

        I inherited this setup, so I don’t know whether it had been previously validated.

        1. Avatar photo
          Paul Cunningham

          Ok. You’ve already made some changes so maybe you’ve already solved the problem. Or maybe not.

          It’s possible the FSW resource at the time of the incident was in a failed state. Maybe the FSW server itself was offline or had some other issue. All speculation.

          In your position I would go back over all the setup steps for a DAG and inspect whether they have been performed or not (eg the FSW permissions/ACLs as you’ve already done). Once the config and current health have been verified then I would arrange a planned test and simulate failure of the secondary server (power it off basically) and see how it goes.

  13. Dale

    Hi Paul,

    This is my first comment on your website, first off thank you for a great website! Every time I Google for Exchange help this is the first site that pops up in the results 🙂

    My question is, if a primary witness server FQDN is spelt incorrectly for one of my customers, can I go in and correct it without any adverse effects? Or should I update it and provide a valid FQDN under change control?

    Thanks,

    Dale

    1. Avatar photo
      Paul Cunningham

      You’re planning to change the server name itself, or the configuration of the DAG?

      1. Dale

        Well, the server name is correct, it’s the FQDN part that is wrong for example:

        CORRECT-SERVER-NAME.MISSPELT-DOMAIN-NAME.LOCAL

        I need to change the misspelt domain name to the correct domain name (Basically someone missed a character out of the domain name).

        If I go in and add the missing character, how will that affect the DAG once I click apply? Or will it just do nothing as I’ve checked the server and there are FSW directories and all the pre-requisites are there.

        What I’m wondering is, once I add the missing domain name character and click apply will there be a stop in services, or will databases failover? I’m sure they won’t as the server is there, pingable and ready to host the FSW.

        Thanks for your reply!

        1. Avatar photo
          Paul Cunningham

          I’m still unclear how this situation came about in the first place, but anyway…

          The FSW can be unavailable as long as the DAG can still maintain quorum (a majority of nodes online). This is no different to, for example, patching and rebooting the FSW.

          If you’re changing the name or FQDN of the FSW in some way, and that in some way affects the DAG members’ ability to resolve the FSW in DNS and connect to it, then you should update the DAG config or DNS to avoid future problems.

          If in doubt, you can run Set-DatabaseAvailabilityGroup to specify a valid FSW at any time.

  14. David Tabib

    Hi Paul,

    I am running into an issue which is don’t know what to do. Here is my situation. I have 4 member. Two DAG + witness server is on one network and 2 dag server in another network.

    About 2 months ago I moved the Witness server to a new server the old server was called ‘mmwitness’ the new Witness server is called “License”

    I ran this command to verify:
    Get-DatabaseAvailabilityGroup -Identity DAG -Status | FL

    Today we had a network issue and databases moved from one network to another, it really should not have. It really should not have moved because the 2 dag members plus witness resides on the same network. When I looked at the log I saw a surprising error. It said:

    File share witness resource ‘File Share Witness (\mmwitness.xxxxx.comdag.xxxxxcom)’ failed to arbitrate for the file share ‘\licensesrv.xxxxxx.comdag.xxxxx.com’. Please ensure that file share ‘\licensesrv.xxxxx.comdag.xxxxx.com’ exists and is accessible by the cluster.

    I do not understand why there is something in my dag setup that still pointing to my old \mmwitness server. Can you assist?

    1. Avatar photo
      Paul Cunningham

      Unless you’ve blocked activation on the server in the second site there is nothing to stop it from being the DAG member that Active Manager chooses for a failover.

      For your other question, did you run Set-DatabaseAvailabilityGroup to change the FSW location?

      1. David Tabib

        I don’t have blocked activation but being the 2 DAG members and Witness servers are in same network, the DAG should not have switched because I have 3 servers on one side and on the other network I have only 2 servers. So, the issue comes in about the Witness server.

        I used the GUI to set the new witness server it shows that my FSW is set to licenserv and NOT the old server that used to be called mmwitness. I also can see in the licenserv a directory is created and that Exchange Trusted Subsystem is the member of local administrator group. But, the error that I got when we had the network issue is wired; somehow Exchange is looking to the old server somewhere. Again the error is:

        “File share witness resource ‘File Share Witness (\mmwitness.xxxxx.comdag.xxxxxcom)’ failed to arbitrate for the file share ‘\licensesrv.xxxxxx.comdag.xxxxx.com’. Please ensure that file share ‘\licensesrv.xxxxx.comdag.xxxxx.com’ exists and is accessible by the cluster.”

        1. Avatar photo
          Paul Cunningham

          Where the DAG members are located in the network is not a factor taken into account by Active Manager when it initiates the Best Copy Selection process to determine which database copy to mount in a failover scenario. If you don’t want databases activating on that server you will need to block activation.

          The FSW issue appears to be just how it is. When you create the DAG and specify an FSW the cluster resource is named “File Share Witness (\servernamesharename)” to match the server name and share name at that time. However, when you specify a different witness server, even though the FSW path updates, the name of the FSW resource in the cluster does not change.

        2. Ryan Sastradinata

          Hey Paul Great article you have here.

          Unfortunately, i have pretty much the same error, I’m currently having 2 DAG Member across 2 subnet (192.168.6.0/24 and 172.20.7.0/24) and a witness server which a Client Access Server (IMTV-000031) that resides on 192.168.6.0/24 network. My DAG name is “IMTV-000032-DAG”, I’ve setup the DAG to have 2 IP address, each one for each subnet.
          Few days ago we got an electricity problem within the server rack, which cause all of the servers lost its power. When i turn it back on, i was surprised that on the “Server Manager Dashboard” i got an error that say, “refresh failed”, so i do a refresh but it still failed with the error message of

          “IMTV-000032-DAG : Configuration refresh failed with the following error: The metadata failed to be retrieved from the server, due to the following error: WinRM cannot process the request. The following error with errorcode 0x80090322 occurred while using Kerberos authentication: An unknown security error occurred.”

          Then i see on Failover Cluster Manager, I got the server (IMTV-000032-DAG) to be online, but the File Share Witness is in Failed status, it also state that the File Share Witness is not configured. So I dive in to the log to see the problem, it recorded error and critical event

          The critical event I got is
          “File share witness resource ‘File Share Witness (\imtv-000031.company.comIMTV-000032-DAG.company.com)’ failed to arbitrate for the file share ”. Please ensure that file share ” exists and is accessible by the cluster.”

          The error event I got is
          “Cluster resource ‘File Share Witness (\imtv-000031.company.comIMTV-000032-DAG.company.com)’ of type ‘File Share Witness’ in clustered role ‘Cluster Group’ failed. The error code was ‘0xc000006b’.”

          I’ve check the share path, and its still there.
          I’ve check the database copy status, everything is healthy, the database is all mounted correctly.
          Even though everything looks fine, I’m afraid that this will cause a major problem in the future. Could you please help me on this? It will be very much appreciated

        3. Avatar photo
          Paul Cunningham

          Server Manager can’t connect to your DAG computer object so I wouldn’t worry too much about that.

          The FSW needs to be online. You can try to manually force it online in Failover Cluster Manager. If that won’t work try setting the FSW again using Set-DatabaseAvailabilityGroup, maybe even try setting it to be a different server.

          Also check clock skew. A simple thing that causes many things to break.

        4. Ryan

          Hey Paul, thank you for replying my comment. I’ve try your suggestion about the time skew and running set-databaseAvaibilityGroup for the witness server and path, but yet still no luck. I’ve try the approch suggested by this article (“http://exchangeexpertscommunity.blogspot.com/2013/08/how-to-reconfigure-failed-fileshare.html”), i got pretty much the same error with When i run the Set-DatabaseAvaibilityGroup command from EMS, which say:

          “There was a problem changing the quorum model for database availability group IMTV-000032-DAG. Error: An error
          occurred while attempting a cluster operation. Error: Cluster API failed:
          “ClusterResourceControl(controlcode=CLUSCTL_RESOURCE_SET_PRIVATE_PROPERTIES) failed with 0xc000006b. Error: Unknown
          error (0xc000006b)”
          + CategoryInfo : InvalidArgument: (:) [Set-DatabaseAvailabilityGroup], DagTaskProblemChangingQuorumExcept
          ion
          + FullyQualifiedErrorId : [Server=IMTV-000029,RequestId=84b52516-8e45-4483-9bf0-a6553c81d715,TimeStamp=6/29/2014 9
          :21:21 AM] [FailureCategory=Cmdlet-DagTaskProblemChangingQuorumException] 3B41C8F3,Microsoft.Exchange.Management.S
          ystemConfigurationTasks.SetDatabaseAvailabilityGroup
          + PSComputerName : imtv-000029.company.com”

          When I run test-ReplicationHealth, I got all *Passed* except for “Quorum Group” and “FileShareQuorum” which say that the DAG and the File Share path can’t be reached

          i’m currently waiting for my supervisor to point which server that could be use to replace to current witness server. In the mean time, i was wondering about the do’s and don’ts in my environment (2 node DAG with different subnet and 1 witness server)?
          cause i read some articles saying that i should be very cautious about rebooting server when there’s only 2 node and a witness server in the DAG

          1. Avatar photo
            Paul Cunningham

            Yes, if you restart a DAG node in your situation you risk quorum being lost and the entire DAG going offline.

  15. Wayne Hutchinson

    Paul – Is it possible to use something like a NAS volume to act as a witness server of does it have to be a Windows Server? I use several NFS and CIFS shares hosted off of my NAS for process targets and would have more faith in that as a highly available source to migrate my Witness directory to.

    1. Avatar photo
      Paul Cunningham

      To the best of my knowledge using a non-Windows Server system is not supported.

      The FSW doesn’t need any special HA.

  16. David

    Hi paul…

    I notice that the “everyone” has full control to my fsw directory……I didn’t set this server up but this seems like a security hole. does the everyone account need full control the the fsw directory.IF i change this to read only will it break the Dag???

    thanks

    Dave

      1. Dave

        paul it looks like both share and ntfs…the everyone group have full permission..

        thanks

        Dave

        1. shishir

          Hi Paul,
          I have 2 exchange 2010 Sp3 (MBX/CAS/HT) servers in a DAG installed on windows 2008 R2. I have configured FSW on non exchange server on the same site which is windows 2012 server.The cluster is online and failover/failback happens properly. When we perform test-replicationhealth for any of the members of this DAG from the other exchange server in the same organisation, I get the result as file share quorum failed with error “couldn’t access the file share witness share”.

          Where as exchange trusted subsystem group already has admin rights on the server where we have configured FSW.

  17. Umang Shah

    I have 4 MBX servers, 2 CAS & 2 Hub servers in my environment. The 4 mbx servers are in a DAG and the FSW is on HUB1. When I reboot HUB 1, the FSW goes into a failed state. Even when after the server comes back up and is functional, the FSW stays in a failed state. The only way I can bring the FSW online, is by manually right-clicking the resource and bring it on-line. Any idea how to fix this?

    Thanks!

  18. Abhi

    I have Windows 2008 R2 server with Exchange as nodes + one another Windows 2008 R2 server to be configured as Witness server.

    All the 3 systems are under “Exchange Trusted Subsystem” security group. However, my DAG configuration constantly fails with the error as below: I have possibly added every place, in all these 3 servers that they are part of “Exchange Trusted Subsystem”

    “Warning:
    Insufficient permissions to access file shares on witness server ‘win-52-239.interopexchange.com’. Until this problem is corrected, the database availability group may be more vulnerable to failures. You can use the Set-DatabaseAvailabilityGroup cmdlet to try the operation again. Error: Access is denied

    Exchange Management Shell command completed:
    New-DatabaseAvailabilityGroup -Name ‘InteropDAG1’ -WitnessServer ‘win-52-239.interopexchange.com’ -WitnessDirectory ‘c:witness’

    Elapsed Time: 00:00:00”

    Any help possible on this please?

    1. Abhi

      There are 2 Exchange nodes and 1 Witness server all under one test domain “interopexchange.com”

  19. Sandeep Sawant

    We have successfully run DR of Exchange 2010 SP2. But we revert back to Primary location. We have observed that file share witness still using DR HT/CAS server. So how to force to use Primary site witness server. Whenever DC-DR link has issue our database get dismounted & once link established database automatically mounted.
    Thanks In advance!

    1. Louise Pasqualle

      Hi

      @Sandeep,

      I have exactly the same issue.

      Did you manage to revert back ot the primary location. Creating a new primary FSW or pointing it to the existing share does not work. It still reverts back to the Alternate.

      Thanks

  20. Jose Parocua

    Paul,

    Another question….

    What brand of hardware load balancer have you had good luck with and why? We may be going down the path of purchasing a hardware load balancer for our CAS array. I know Microsoft offers an article on supported hardware load balancers but I’d like to get your opinion on this.

    Thanks!

    1. Avatar photo
      Paul Cunningham

      I’m happy with my experience of Kemp and F5 but I recommend you take full advantage of their trial periods and do your own assessment and evaluation.

  21. Gowhar Rashid Zargar

    Thanks a million Mr. Cunningham… Appreciate your help.

  22. Gowhar Rashid Zargar

    OK Great !!! I have one more server for McAfee Antivirus Server running on hypervisor in AD Domain. Shall I use that one as a Witness Server ?

    1. Avatar photo
      Paul Cunningham

      I don’t see a problem with that, as long as the server is reliable and not constantly overloaded already. FSW doesn’t add any significant load to a server but the FSW should generally be reliable.

  23. Gowhar Rashid Zargar

    Hi…
    Environment:
    2 Servers installed with MAILBOX+HUB TRANSPORT+CAS Roles in AD DOMAIN on physical servers
    2 Servers installed with EDGE TRANSPORT Roles in a WORKGROUP on Hypervisors

    I want to make a DAG for MAILBOX databases for redundancy. Can I make EDGE TRANSPORT Server as a WITNESS SERVER. Please help with that

    1. Avatar photo
      Paul Cunningham

      No your Edge Transports cannot be file share witnesses for the DAG.

      1. Gowhar Rashid Zargar

        Yeah… tried that but didn’t work… I guess I will have to build a HUB+CAS server and that can be made a WITNESS Server…right…?

        1. Avatar photo
          Paul Cunningham

          It doesn’t need to be an Exchange server, though it is generally recommended. But you can also use other non-Exchange servers as long as you first add the Exchange Trusted Subsystem group to the local administrators group on the server.

          This is the Exchange 2013/Windows Server 2012 version of how to do it, basically the same for Exchange 2010/Server 2008 but the screenshots will look different of course.

          https://www.practical365.com/using-a-non-exchange-server-as-an-exchange-2013-dag-file-share-witness/

  24. Chris Nicholas

    I have some serious man-love for you. Nobody writes a better guide.
    When I read you work, I know what I am doing it and more importantly, why.
    Keep up the great work and thanks for all the great articles.

  25. Burjal

    Unfortunately 4 me it doesn’t work. In producton i had to move cluster witness on another server, where I already put exchange subsystem in local admins. The share is created. Inside are two files with 0 kb. The share witness doeasn’t work. In GUI i see success, but in powershell there is an error that exchange subsystem is not local admin (but aparently it is). Just as is stated here:
    http://blogs.technet.com/b/scottschnoll/archive/2011/06/08/witness-server-warning-message-when-using-certain-database-availability-group-tasks.aspx
    Is this a bug or smth I am missing?

    1. Avatar photo
      Paul Cunningham

      Scott explains in that article that the error is a bug in Exchange and can be disregarded if you have put the ETS group into the local admins group on a non-Exchange FSW.

  26. Jose Parocua

    Paul,

    We have 2 CAS/HTS servers in the same AD Site. When my primary CAS (which also hosts the FSW) goes offline, my outlook clients get prompted to provide credentials and they won’t automatically connect to the secondary CAS until i manually update DNS records. Even then they have to provide credentials to log on to the secondary CAS server.

    Do i possibly need to move the FSW role to another member server?

    Thanks!

    1. Avatar photo
      Paul Cunningham

      The CAS role (and the RPCClientAccessService) has nothing to do with the File Share Witness for a DAG. They may happen to be on the same server, but they are completely different things.

      Your issue is that the RPCClientAccessServer that Outlook users are connecting to is going offline. You would solve that problem with a CAS Array and load balancing solution.

Leave a Reply