Server Update Fails During Database Backup

Description

Update of the server component consist from these steps:

  1. Update confirmation
  2. Download and verification of files
  3. Turning off web service and eWayTimer service
  4. Database backup
  5. Update of application files
  6. Update of database
  7. Creation of CLR procedures
  8. Turning on web service and eWay Timer

The most common error during update is related to database backup. Administration Application will show this error:

eWay.ServerUpdater.Exceptions.CommandRunningException: Command 'eWay.WebServiceDeploy.Wizard.WizardFormCommands.BackupDatabaseCommand' wasn't successful. The result was 'Error'.
at eWay.ServerUpdater.DeployWizard.RunnedCommand.Run(ExceptionCatcher exceptionCatcher)
at eWay.ServerUpdater.Executables.Executable.RunCommands(IEnumerable`1 commands)
at eWay.ServerUpdater.Executables.ProcessTheUpdateExecutable.Execute()
at eWay.ServerUpdater.Program.Main(String[] args)

Alternatively the error may look like this:

eWay.ServerUpdater.Exceptions.CommandRunningException: Command 'eWay.WebServiceDeploy.Wizard.WizardFormCommands.BackupDatabaseByEwayProcedureCommand' wasn't successful. The result was 'Error'.
   at eWay.ServerUpdater.DeployWizard.RunnedCommand.Run(ExceptionCatcher exceptionCatcher) in C:\CC.NET\Projects\NAnt\Sources\DeployManagement\ServerUpdater\ServerUpdater\DeployWizard\RunnedCommand.cs:line 38
   at eWay.ServerUpdater.Executables.Executable.RunCommands(IEnumerable`1 commands) in C:\CC.NET\Projects\NAnt\Sources\DeployManagement\ServerUpdater\ServerUpdater\Executables\Executable.cs:line 73
   at eWay.ServerUpdater.Executables.ProcessTheUpdateExecutable.Execute() in C:\CC.NET\Projects\NAnt\Sources\DeployManagement\ServerUpdater\ServerUpdater\Executables\ProcessTheUpdateExecutable.cs:line 73
   at eWay.ServerUpdater.Program.Main(String[] args) in C:\CC.NET\Projects\NAnt\Sources\DeployManagement\ServerUpdater\ServerUpdater\Program.cs:line 61

CommandRunningException tells that one of commands (steps) fails. It is important to know which one. eWay.WebServiceDeploy.Wizard.WizardFormCommands.BackupDatabaseCommand specifies that it is database backup. The second error logs BackupDatabaseByEwayProcedureCommand - the error is caused by the backup setting.

Administration Application does not specify the cause of the backup error. You need to go to update process logs, phase 2 - e.g. Update_2017-12-14_Phase2.txt. It looks like this:

2017-12-11 13:56:40: ERROR: : [MessageProcessor] [Error, ] Backups database '' to 'eWay_beforeUpdate_2017-12-11_13-56-34.bak'. Exception message: Simple Sql command 'backup database eWay' failed with messaqge: Cannot open backup device 'c:\Program Files\Microsoft SQL Server\MSSQL10_50.EWAYSQL\MSSQL\Backup\eWay_beforeUpdate_2017-12-11_13-56-34.bak'. Operating system error 3(failed to retrieve text for this error. Reason: 15100).
BACKUP DATABASE is terminating abnormally.
Ex. stacktrace:
  at eWay.WebServiceDeploy.Wizard.Utils.DatabaseUtils.RunSimpleSqlNonQueryCommand(String connectionString, String sqlText, Int32 commandTimeout)
  at eWay.WebServiceDeploy.Wizard.Utils.DatabaseUtils.BackupDbToFile(String instanceConnectionString, String databaseName, String filePath)
  at eWay.WebServiceDeploy.Wizard.WizardFormCommands.BackupDatabaseCommand.Run()
Inner ex. stacktrace:
  at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
  at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
  at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
  at System.Data.SqlClient.SqlCommand.RunExecuteNonQueryTds(String methodName, Boolean async, Int32 timeout, Boolean asyncWrite)
  at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
  at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
  at eWay.WebServiceDeploy.Wizard.Utils.DatabaseUtils.RunSimpleSqlNonQueryCommand(String connectionString, String sqlText, Int32 commandTimeout): 
2017-12-11 13:56:40: DEBUG: : Result: None, : 
2017-12-11 13:56:40: ERROR: : The updater execution ended with an exception.: Command 'eWay.WebServiceDeploy.Wizard.WizardFormCommands.BackupDatabaseCommand' wasn't successful. The result was 'Error'.

Version

All versions of eWay-CRM

Difficulty

Medium

Resolution

Before the moment when updater tells that BackupDatabaseCommand was not successful, the error is logged right from the proceeding command. In this case, SQL Server was not able to realize access to the file on this path: c:\Program Files\Microsoft SQL Server\MSSQL10_50.EWAYSQL\MSSQL\Backup\eWay_beforeUpdate_2017-12-11_13-56-34.bak. The reason was missing folder.

Other possible reasons of database backup failure are:

  • Missing write permissions to filesystem
  • Lack of space on the target hard drive
  • Incorrectly recovery model of SQL Server database (eWay-CRM requires FULL recovery model)

The error must be resolved on the server due to its character. In most cases, it is simple action (make more space on hard drive, set permissions, change / create target folder, etc.). If you proceed the action, you need to repeat the update process.

As you can see from the update steps above, when the update fails on the backup step, there are still no changes on the server side of eWay-CRM. Application data and files are not in inconsistent state. Server component can be relaunched in these steps:

  • Turn on web service - rename App_Offline.htm to _App_Offline.htm
  • Turn on eWayTimer service

Now, eWay-CRM is in the same state as it was before update. Now, you can go back to the Administration Application and start the update again.