In the past, upgrading a Windows failover cluster to use a new server operating system was a big undertaking. But...
Microsoft has put in quite a bit of work with Windows Server 2016 to make it an easier procedure to transition failover clusters to use the new OS.
Before I explain how this process works, this article only discusses failover clustering from the OS standpoint. There may be additional steps needed depending on what applications are running on the failover cluster. Similarly, the upgrade process does not generally require any hardware upgrades; if you decide to perform a hardware upgrade as a part of the upgrade process then that hardware upgrade may require some additional steps.
With that said, Microsoft allows what is known as a rolling upgrade from Windows Server 2012 R2 to Windows Server 2016. This rolling OS upgrade works in a similar manner to rolling patching, in which cluster nodes are patched individually so the cluster remains online throughout the patching process.
Microsoft has designed the cluster upgrade process to be safe. Up until you reach the final step in the upgrade process, the upgrade can be aborted and the system reverted to its previous state.
Back up the cluster and its workloads
The first step to upgrade a Windows Server 2012 R2 failover cluster to run Windows Server 2016 is to back up the cluster and the clustered workloads. Even though the upgrade process is designed to be safe, there is always a chance of an issue during the upgrade; you will want a backup if things go awry.
Once the backup is complete, pick a cluster node to begin the upgrade process and move all its workloads to other cluster nodes. Once the node has been drained, evict the node from the cluster. At that point, the node becomes a standalone server rather than a failover cluster node.
With the node removed from the cluster, it's time to install Windows Server 2016. While currently unconfirmed, the prerelease documentation from Microsoft suggests an in-place upgrade of the server's operating system will not be supported. The documentation indicates the server's system drive needs to be reformatted and the new operating system installed from scratch. After installing Windows Server 2016, there are a number of tasks required to prepare the server to participate in the cluster. Some of these tasks include configuring network connectivity, configuring connectivity to the cluster shared volume, installing the failover cluster feature and installing any necessary infrastructure software, such as backup agents or antimalware software.
Upgrade can be reversed -- up to a certain point
At this point, the cluster node should be ready to use -- in a production environment you may have to install an application, such as Hyper-V or SQL Server. You can use the Add-ClusterNode cmdlet to join the upgraded server to the failover cluster. The failover cluster is now operating in mixed mode due to the combination of Windows Server 2012 R2 nodes and Windows Server 2016 nodes.
Nodes can be upgraded to Windows Server 2016 one by one, but nodes can be reverted to Windows Server 2012 R2 by repeating the procedure, but installing Windows Server 2012 R2 instead of Windows Server 2016.
The upgrade process becomes irreversible after raising the cluster's functional level which is similar to raising an Active Directory domain functional level. The process allows the new features introduced in Windows Server 2016 to be used by raising the functional level after all cluster nodes have been updated. Raising the cluster functional level is irreversible without restoring the cluster from backup.
Microsoft has made it less of a chore to upgrade a failover cluster from Windows Server 2012 R2 to Windows Server 2016. The added benefit is the process keeps highly available workloads online and available throughout the upgrade process.
The first step to prepare for a Windows Server 2016 upgrade
Calculating the move to Windows Server 2016
Making the move to Windows Server 2012