hi All,
We have a SQL 2005 SP4 mirroring setup of 15 DBs with Principal(P), Mirror(M) & Witness (W).
We have now seen abrupt DB failovers for some of the databases (yest it was 4 out of 15) from P to M.
Errors were seen on Witness server as follows for all Dbs that failed over:
Date
07/01/2015 11:07:48 PM
Log
SQL Server (Current - 08/01/2015 12:00:00 AM)
Source
spid19s
Message
The mirroring connection to "TCP://<server.domain.com>:5022" has timed out for database "<DBName>" after 10 seconds without a response. Check the service and network connections.
Actions taken:
1. Network and Firewall team reverted that no error detected and no network traffic between the witness server and db server during the db auto failover period.
2. On the system side, we have verified that no hardware error found on either VM or SAN storage, and no Symantec SQL backup jobs running nor anti virus scanning during the db auto failover period too.
3. We did see some high amount of IO activity on P server around failover time. Some IO errors similar to below were seen, however point to note is these errors were not only for the DBs that failed over, but for others including TEMPDB:
Date
07/01/2015 11:07:38 PM
Log
SQL Server (Current - 08/01/2015 4:06:00 AM)
Source
spid2s
Message
SQL Server has encountered 1 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [R:\SQLDATA\MSSQL.1\MSSQL\Data\<DBName>.mdf] in database [DBName] (5). The OS file handle is 0x000000000000095C. The offset of the
latest long I/O is: 0x0000054ff22000
Questions:
1. I assumed that the Witness keeps polling P & M on DB mirroring endpoints (in our case 5022) to check that the DBs are online, but Network team says there is no activity on that port, is my understanding correct?
2. Is there any other reason for DB failover ?
Link referred:
http://dba.stackexchange.com/questions/22402/what-can-cause-a-mirroring-session-to-timeout-then-failover-sql-server-2005
http://msdn.microsoft.com/en-us/library/ms179344(v=sql.90).aspx
Any help is highly appreciated!!!
Regards,
Mandar