You may experience similar problems in version from Exchange 2010, here I have showcased an issue I faced in Exchange Server 2016 infrastructure.

You will see one or more of the following events/status on the server,

Explanation:

This issue can cause due to many factors, and most cases it can be mitigated by Solutions mentioned in the next section.

The reasons for this issues could be one or more of the following, but not limited to

1. Issues with “Microsoft Exchange Search” (MSExchangeFastSearch) Service
2. Issues with “Microsoft Exchange Search Host Controller” (HostControllerService) service.
3. Corrupt Content Index
4. Hard disk errors (read/write)

How to mitigate this issue, follow as suggested below!

Solution 1

Generally, this issue can be mitigated in one of the two list options below,

Option 1:

Consider you do not have any database copies with healthy Index State or the only server that host the database.

Stop the following services,

Delete the folder similar to the following from database path/location.

Start the Services in order,

The Exchange Server will initiate the process to rebuild the whole index. Depends on the data/size the process may take extended period of time. Wait for the process to finish, and in 90% of the case it will fix the issue.

Option 2:

If you have one “healthy” Content Index state database copy, reseed the database copy that has the unhealthy copy of content Index. In some cases when you add database copy in the DAG, the content index can go into unhealthy status (such as failed, unknown, failedandsuspended etc). To recover the content Index back to healthy, you can reseed the content index by running the “Update-MailboxDatabaseCopy” command with switch “-CatalogOnly”.

Update-MailboxDatabaseCopy LAB-DB03\EX16-01 –CatalogOnly

If you already tried this command, you may get the warning “Seeding cannot be requested for the same database copy until the failed request has been cleaned up by the server …”. Please say “y” to proceed with the reseed operations.

Verify the ContentIndexState after sometime, you will see a “healthy” state.

Solution 2

If the above steps did not go through fine, and if you receive errors similar to below even if all these services are healthy then it will be not an easy task to bring back your ContentIndexstate to healthy.

WARNING: Seeding of content index catalog for database 'LAB-DB03' failed. Please verify that the Microsoft Search (Exchange) and the Host Controller service for Exchange services are running and try the operation again. Error: An error occurred while processing a request on server 'EX16-03'. Error: An Exception was received during a FAST operation..

In some cases, there are disk errors but in my case the server was recovered from a unexpected failure which lead to run recovery steps on the database. Post the recovery the ContentIndexState was in healthy status in some database copies, but did not work quite fine in others.

Please note that the Content Index State only impact the end user experience, you are still allowed to mount the database on server where the Content Index State is not healthy. Hence the recommended approach is to create additional database and move the mailboxes gradually to get rid of the problematic database.

In my case, I also had the following event (Event ID 40025) logged on the server where this database was active.

Please share you comments or issues.

-Praveen

You may receive the following error when you tried to re-add a recovered Exchange 2016 Mailbox server back to Database Availability Group (DAG).

-
A server-side database availability group administrative operation failed. Error The operation failed. CreateCluster
errors may result from incorrectly configured static addresses. Error: An error occurred while attempting a cluster
operation. Error: Node EX16-01 is already joined to a cluster..
-

You may also see that the following error when trying to run the command "Set-DatabaseAvailabilityGroup". Please note I ran the command from the cluster node which I was trying to add to DAG.
-
The following servers in the Windows Failover Cluster are not in Active Directory: ex16-01. This is usually the result
of an incomplete membership change (add or remove) of the database availabilty group.
-

The above error usually the result of disabled cluster service, or AD replication errors.

Solution:

It is evident that, some stale entries or communication error stops from adding recovered exchange server to the DAG cluster. Before you start any process to forcefully remove or add, try to do a force replication of Active Directory and repeat the usual procedures. If the results are same, follow the below process.

Run the following command from a healthy DAG Member.

1. Forcefully Remove cluster node which you failed to join the DAG cluster. In my case, I was trying to re-add the node ex16-01 back to the DAG02 Database Availability Group.

Get-ClusterNode ex16-01 | Remove-ClusterNode -Force

Wait for 10 - 15 minutes, so that the replication happens.

2. Run the command Add-DatabaseAvailabilityGroupServer with the node we want to add to cluster.

Add-DatabaseAvailabilityGroupServer -Identity DAG02 -MailboxServer EX16-01

3. Verify the Database Availability Group status,

[PS] C:\Windows\system32>Get-DatabaseAvailabilityGroup DAG02 | fl Name,Started*

Name : DAG02
StartedMailboxServers : {ex16-01.lab.ed.com, EX16-03.lab.ed.com, EX16-02.lab.ed.com}

Following screen shot shows the chronology for your better understanding.

Post this, you may proceed with adding database copies or additional cluster operations.

-Praveen

This video help you understand the process of introducing or installing new version of Exchange, Exchange Server 2016 to existing environment. The LAB I have is with Exchange 2010 and not with Exchange 2013. But the process are the same to introduce the new exchange server. 

Ensure you went through the pre-requisites such as the minimum patch level, AD requirements etc. This video walks you through only the installation procedure and the time is not actual. 

The actual process may take anytime between 40 min to 90 minutes, do not disturb the process when feels no progress. You may just drag the Command windows to refresh the progress if any. 

I preferred to choose command line installation, much faster an easier.

-Praveen