File System Quotas

Through normal usage, the Framework will consume additional disk space over time. On the web/app server, space is consumed by items such as letters, credit reports, archived import files, etc. This consumption is much more prevalent on the database server. The database will continue to grow as new accounts are imported and worked (creating workflows, comments, promise to pays, etc.). Over time, it is possible that a server will run out of disk space. When this happens, the Framework can no longer function and will be down until the space issue is resolved. Most often this problem occurs on the database server and results in the following error:

Through the use of file system quotas, which is a feature built into Windows, a system administrator can be notified via email when the disk usage exceeds a predefined threshold. This proactive monitoring will enable the administrator to take action to resolve the space issue before it affects the Framework and end users.

• Create a file system quota by opening the File Service Resource Manager. If this program is not available, it can be installed through Windows Features in Programs and Features. Under Server Role select File Services and under Role Services select File Server Resource Manager.

• Open the File Server Resource Manager

• To receive email notifications, you must ensure File Server Resource Manager has the proper SMTP settings.  Rick-click on the File Server Resource Manager and select Configure Options.

• Enter the information on the Email Notifications tab according to your environment.
• Click the OK button.
• Right-click on Quota Management > Quotas and select Create Quota.

• Enter the drive to monitor for this quota in the Quota Path field.

If the server uses multiple logical/physical drives, you must set up separate quotas for each.

• Select the Define customer quota properties option.
• Click Custom Properties.

• Under Space limit, enter the total drive capacity.  The image above shows 80 GB because that is the maximum capacity of the C:/ drive.
• Select Soft Quota option.
• Under Notification thresholds, click Add.

• Enter 75 for the usage percentage
• Check the email administrator option and enter your email address.
• Click OK.

• Add two more notification thresholds for the usage percentages of 85% and 95%.

• Click OK.
• Click Create.

The set up is complete.  You will now receive email notifications when the drive(s) reaches 75%, 85%, and 95% of capacity.

Schedule and execute weekly maintenance of your Web, Application and Database servers. This general maintenance will ensure your servers are running efficiently. The maintenance should include the following:

• Run Windows Update and install all “High” and “Critical” patches and hotfixes to ensure all Windows related components and services are up to date and all known vulnerabilities and issues are resolved. If updates are installed, reboot the server.
• Complete maintenance by rebooting the server. This ensures the application runs at peak performance.

Consistent periodic maintenance of the Framework database is important in keeping the Framework running at peak performance. This maintenance should be conducted once per week during off hours. This section will walk you through the steps required in setting up an effective maintenance plan.

Create the maintenance plan by using the SQL Server Agent.

• Download the Akcelerant provided SSIS File found here.
• Save the file where it is accessible from the SQL Server machine.
• Start SQL Server Agent if it is not currently running.
• Right click on SQL Server Agent (Agent XPs disabled) and click Start.

• Right-click on Jobs and click New Job.

• On the New Job dialog, fill out the Name of the new job to Akcelerant Maintenance Plan.

• Click Steps in the left navigation pane and click New at the bottom of the form.
• New Job Step dialog box displays. Complete the following fields:
  • Step Name: Akcelerant Maintenance Step
  • Type: SQL Server Integration Services Package
  • Package source: File System
  • Package: Browse to the .dtsx file that was saved on the SQL Server machine.

• Click Set Values tab.  Complete the following:
  • Property Path:  \Package.Variables[User::DBName].Properties[Value]
  • Value: Enter in the name of your Akcelerant Framework Database

The Value must match to the Database you are running Akcelerant Framework on.

• Click OK. 
• Click on Schedules on the left menu pane and click New.

• A New Job Schedule dialog displays.  Enter in the following information:
  • Name: Akcelerant Maintenance Schedule
  • Schedule Type: Recurring
  • Select Enabled check box
  • Complete Frequency

Akcelerant recommends running this maintenance once per week on Sunday at 12:00 AM. The customer should consider their daily imports when creating the schedule. Import times can vary for each customer depending on their size, environment, etc. Customers should make sure that the maintenance and imports are not scheduled so that they are running at the same time.

• Click OK.
• Click OK on the Job Properties screen.

• To verify that the job ran successfully, double-click Job Activity Monitor.  The dialog box displays the job that was created.  Verify the Last Run (Succeeded, Failed, Unknown) and Last Run Date.

High availability is achieved through hardware redundancy. Consider a scenario where the Framework is installed on a single web server. If that server goes down, the Framework is unavailable until the server is back online. If that single server experiences a hardware failure, it could be hours or days before the Framework is operational again. Implementing hardware redundancy will make the Framework application highly available. Consider another scenario where the Framework is installed and load balanced across two web servers. If server 1 goes down, server 2 will continue to serve requests while server 1 is brought back online or replaced. The end user experiences no interruptions in this scenario.

Load Balancing

Redundancy on the web/application servers is achieved through load balancing. This solution distributes traffic among two or more identical servers. Choosing and implementing a load balancing solution is the responsibility of the customer. Two of the more popular solutions are Windows Network Load Balancing and f5 Load Balancing. Utilizing session state is a common roadblock for applications in a load balanced environment. The Framework application does not utilize session state so this is not a concern.

The following is required when implementing the Framework in a load balanced environment:

• All servers in the load balanced environment must be running the exact same Framework release. When you install a new Framework release, it must be installed on every server.
• The Framework Application server utilizes Windows Services and Scheduled Tasks. These components must be enabled on only one server. Consider a scenario where the Framework is installed on three load balanced servers. If the services and tasks are enabled on all servers, the daily imports from the Core will run on all three servers at the same time. This will cause issues since the imports (and many other processes) must only run once per day. To avoid these issues, you must choose one server out of the three as the “primary server.” Your secondary server(s) must disable the following components:

SQL Server Failover Clustering

Redundancy on the database server is achieved through SQL Server Failover Clustering. This solution is a collection of two or more physical servers with identical access to shared storage. Implementing a clustering environment is the responsibility of the customer. This article will walk you through the basics of clustering configuration options and installation procedures.

The following is required when implementing the Framework in a clustered environment:

• During the initial Framework installation, enter the cluster name when prompted for the SQL Server name.
• The Framework must be configured to use the cluster name and not a specific IP Address for all database connections.

Database Backups

A committed back up strategy protects you in the event of unexpected data loss. A key factor devising your backup strategy is to determine the acceptable amount of data loss. If you can accept the risk of losing a day’s worth of data, you should implement the simple backup strategy. If this type of data loss is not acceptable, you should implement the advanced backup strategy.

Simple Backup Strategy	The simple back up strategy is simple to implement and requires less server resources to maintain.  By implementing this strategy, you accept the risk of losing one day's worth of data in the event of disaster. Set the Recovery Model database option to "Simple." Create a full backup of the database once per day during off hours.
Advanced Backup Strategy	The advanced back up strategy is a bit more complicated to implement and requires more server resources to maintain.  By implementing this strategy, you have complete control  over the amount of data loss in the event of a disaster. Set the Recovery Model database option to "Full." Create a full backup of the database once per day during off hours. Create an incremental backup of the database every four hours. Create a transaction log backup at a minimum every hour.  You may increase the frequency of a transaction log backup to suit your needs.  In the event of a disaster, you will  be able to recover your database up to the point of your last transaction log backup.
Important Considerations	Here are important considerations regardless of the backup strategy implemented above: Always keep several days’ worth of backup files. Backup files can become corrupted, lost or accidentally overwritten after they are created. It is wise to keep several days’ worth in the event that the most recent backup files are not available. Store backups on a separate physical disk. In the event of disk failure, the database and all its backups are lost if they are on the same physical disk. If possible, store backups at an external site. This protects against natural disasters at your site.

Share Folder Backups

The Akcelerant Framework stores important files in the share folder. These files include letters, documents, log files, credit reports and archived import files.

• Create a full back up of the share folder at a minimum once per week during off hours. Increase the frequency of this backup to suit your needs.
• Store backups on a separate physical disk. In the event of disk failure, the share folder and all its backups are lost if they are on the same physical disk.
• If possible, store backups at an external site. This protects against natural disasters at your site.

Stand-by Servers

A stand-by server is beneficial in the event of a server failure. It will greatly reduce down time after the failure occurs. With a few configuration changes, the stand-by server can be brought online and begin serving requests.

• Create a stand-by server for your web server and another for your application server.
• Before conducting the initial installation of the Framework on your stand-by server, create a full back up of your production database. Restore this back up over the production database after the installation is complete. This step is required because the installation of the stand-by server will change configurations stored in the database. The backup/restore will ensure the configurations are identical before and after the installation. This step is not required when running an upgrade on your stand-by server.
• Stand-by servers must be kept in sync with the production servers as upgrades are installed.
• After the Framework is installed on the on the stand-by servers, disable tasks and services to prevent processing duplication and accidental requests.

Web Server	IIS
Application Server	IIS Akcelerant Process Manager (scheduled task) Akcelerant Lending Service (Windows service)

Web Server Failure Procedures

In the event of a web server failure, follow these procedures to bring the stand-by server online.

• Update the configurations stored in the database which use the server name

dbo.tblGlobalParameter	ParameterCode = SITEROOT
dbo.tblGlobalParameter	ParameterCode = SITEROOT2

• Reset IIS on the stand-by server.
• Distribute the new URL to end users.

Application Server Failure Procedures

In the event of an application server failure, follow these procedures to bring the stand-by server online.

• Update the configurations stored in the database which use the server name

Table Name	Row Identifier
dbo.tblGlobalParameter	ParameterCode = SHAREPATH
dbo.tblSolutionParameter	ParameterCode = CORE_SERVICE_ROOT
dbo.tblSolutionParameter	ParameterCode = COLLECTIONS_SERVICE_ROOT
dbo.tblSolutionParameter	ParameterCode = LENDING_SERVICE_ROOT
dbo.tblSolutionParameter	ParameterCode = HOST_SERVICE_URL
dbo.tblSolutionParameter	ParameterCode = WEBLOAN_SERVICE_URL
Lending.DisbursementProvider	All rows
Core.EventListener	All rows

• Enable the following tasks and services:
  • Akcelerant Process Manager (scheduled task)
  • Akcelerant Lending Service (Windows service)
• Reset IIS on the stand-by server.

Reporting Services

Disaster recovery procedures for Microsoft SQL Server Reporting Services are not included in this document. It is the customer’s responsibility to plan and implement procedures for high-availability and redundancy of the Reporting Services system.

The Framework stores configuration information for Reporting Services in the database in the dbo.tblGlobalParameter table. In the event of a disaster with Reporting Services, these configurations can be changed to point to a backup/restored instance:

Parameter Code	Parameter Description
RS_ReportExecutionPath	Virtual folder where the RS Web Service resides
RS_ReportFolderName	Name of the Folder in the RS Server that holds all reports and reporting DSNs
RS_ServerName	Name of the Server where Reporting Services Runs
RS_SQLServerVersion	Tells the systemtwhar version of SQL Server the client is running

Database Mirroring

Database mirroring in SQL Server allows you to keep a copy, or mirror, of a SQL Server database on a standby server. Mirroring ensures that two separate copies of the data exist at all times, providing high availability and complete data redundancy.

Requirements

Configuration Options

Configuration Recommendation

Setting up the Framework for mirroring

Database Failover Procedures

Application Failover Procedures