Recently I was testing the Datacenter failover of Database Availability Group (DR), and observed few things/errors when adding back the DAG members after the successful failover. During the failback, I could add one of the 2 servers in primary datacenter but failed to add another server. The Start-Databaseavailabilitygroup failed to complete with error as pasted below,
WARNING: Server 'EX-01' failed to be started as a member of database availability group 'DAG-01'. Error: A
server-side database availability group administrative operation failed. Error: The operation failed. CreateCluster
errors may result from incorrectly configured static addresses. Error: An error occurred while attempting a cluster
operation. Error: Node ex2010-01 is already joined to a cluster. [Server: Ex-DR.fabrikam.com]
WARNING: The operation wasn't successful because an error was encountered. You may find more details in log file
Start-DatabaseAvailabilityGroup failed to start server(s) 'EX-01' in database availability group 'DAG-01'.
+ CategoryInfo : InvalidArgument: (:) [Start-DatabaseAvailabilityGroup], FailedToStartNodeException
+ FullyQualifiedErrorId : 811A6BB8,Microsoft.Exchange.Management.SystemConfigurationTasks.StartDatabaseAvailabilit
Other symptoms are,
It means, the ex-01 is cleanly evicted from the cluster and does not have any stale entry in the cluster configuration.
How to Fix:
When I verified the DAG nodes Exchange version, I observed that the node which is failing to add is in a higher rollup version though at the same service pack level. I have raised the other nodes (ex-dr and ex-02) into the similar rollup level as ex-01 (server fail to add) and could able to execute the Start-Databaseavailabilitygroup cmdlet without any issues. That means, when you try to add a dag node using Start-Databaseavailabilitygroup, the PAM (Primary Active Manager) should be at the same or above rollup level.
So, please ensure the below if you are facing the similar error,
First 3 steps are easy to follow, step 4 and 5 are explained below.
How to verify the current started and stopped mailbox server in a DAG,
Execute the below cmdlet,
Get-DatabaseAvailabilityGroup -Status | fl st*,pr*,op*
Once you confirm that the server which are trying add back to DAG is listing in StoppedMailboxServers list, you can safely execute. If the server you are trying to add is already in the startedMailboxServers list then you might have to stop the server before you can add it back. The configuration might have updated due to multiple execution of Start-Databaseavailabilitygroup cmdlet in the past.
Do not forget to execute Set-DatabaseAvailabilityGroup cmdlet to properly set the cluster properties in case you run the Stop DAG command.
How to verify the node status in cluster configuration,
Execute Get-ClusterNode to list all the available nodes in the DAG. If the cluster node you are trying to add is listing out there other than status “up”, you might have to remove it forcefully before executing the Start-Databaseavailabilitygroup cmdlet.
You will be able to execute the Start-Databaseavailabilitygroup cmdlet successfully after these checks, see below,
Share your comments, and errors if any.