No one likes downtime. It is stressful for managers, operations, and developers. Downtime is frustrating and confusing for users of a site, whether or not the “down for maintenance” page is live. Most of the popular deployment strategies for ASP.NET apps hosted in IIS do not even try to deploy without downtime. Any change to
.dll files on a live site can cause significant downtime (I’ve seen 5-10 minutes for large apps). There is a better way. It is possible to achieve zero downtime with the “blue green deployment” strategy in IIS using Application Request Routing, and URL Rewrite. Let’s get started.
The general idea of “blue green deployment” is that there is an entry point (load balancer) that routes requests to a site that is up. An application is deployed to a site that is down, that application is warmed up, then the entry point is notified to route new requests to the newly warmed up site instead of the old one. The entry point needs to be able to make the switch with no hiccups, no lost connections, low CPU/memory overhead, and definitely without any perceived downtime by end users. Application Request Routing can act as that load balancing entry point for us.
- IIS 7 or higher
- Application Request Routing
Start with a basic static site called
alwaysup. Keep in mind that this site could be any IIS hosted application (ASP.NET, HttpPlatformHandler, iisnode, PHP, etc). A static site will be enough for this example. There will need to be two instances of
alwaysup in IIS, so duplicate the folder that
alwaysup lives in to create
/alwaysup-blue. The application files are simple. Just some
index.html files to indicate which application is being hit.
Create corresponding IIS sites for each application and name them
alwaysup-blue. These sites need to be bound to a unique port that is not port
Also create some host entries for each site so that when the Server Farm is created later each unique host name will act as a server address in the Server Farm. Use
alwaysup-green for the host names, and also add a host name for the server farm that will act as the entry point for the application
127.0.0.1 alwaysup 127.0.0.1 alwaysup-blue 127.0.0.1 alwaysup-green
This is the complete configuration for the
Next create a Server Farm in IIS to route traffic between the two sites. Add a Server Farm named
Next add a server for each host name binding of the
After creating the servers, IIS will ask prompt to create a URL rewrite rule to route all incoming requests to this server farm automatically. Select “No” because a rule will be created later manually with the URL Rewrite module.
Next add some health checks to the server farm. This is how Application Request Routing will know which site is “up” to route requests to. A simple way to do this is to add an
up.html file in the root of
alwaysup-green with the text “up” in one, and “down” in the other.
Then add a health check to the
alwaysup Server Farm that makes sure the response received from
/up.html contains the text “up”. Set the polling interval to be quick so that when changes are made to the health check files very little wait time is required for the health checks to fail or pass. Notice also that the
alwaysup HOSTS entry is used here.
The “Monitoring and Management” page of the Server Farm shows that one of the Server Farm servers is marked as unhealthy - which is to be expected.
This is the complete configuration for the
alwaysup Server Farm.
Now route the actual traffic to our Server Farm
alwaysup. To do this use the URL Rewrite module in IIS. Requests will be made on host name
alwaysup, so add a URL Rewrite rule to match on
alwaysup on port 80. Route this traffic to the
alwaysup Server Farm. One thing to note is that IIS needs at least one site listening on port
80 for our URL Rewrite to work.
Test everything done so far by making a request to
Make one of the sites fail the health test and the other pass and see the site content change.
At this point any deployment strategy could be used to deploy to a site in IIS, except now it is necessary to check which site is down and deploy to that one instead of the one that is up. Deployment can happen via WebDeploy/
msdeploy, FTP, Dropbox (yes - some people deploy with Dropbox), an Octopus Deploy Tentacle, etc. After new code has been deployed to the site that is down, the down site needs to be warmed up before its health check can be changed to a passing state. To do this, make a request to the site manually. In this example, a request would be made to
http://alwaysup-blue:8001 to initiate a warm up in IIS. Then edit the health check file to make it pass the health check by changing
up. Then once the server’s health status is “Healthy”, the initial site that was up can safely be brought down. All of this is done without introducing any downtime.
No more 4:30am deployments!
Automating with PowerShell
Doing all of this manually can be tedious, and should be automated so that steps aren’t accidentally skipped or done incorrectly. Here are some PowerShell snippets to achieve what was done manually in this example.
Check which site is unhealthy and ready for deployment
The quickest way would be to check the contents of the
up.html file with
Get-Content, essentially mimicking the health check in the IIS Server Farm.
We can also check the status of the Server Farm itself in IIS through
First define a function to return a Server Farm by name.
Then get the
isHealthy status from the ARR counters.
At this point use whetever deployment method necessary to deploy to the site that is down.
Warming up after deployment
I’m still trying to figure out how best to warm up a site without setting a timeout, but the idea so far has been to
Invoke-WebRequest to the down site until a response comes back within a short enough time period.
Then change the content of the
up.html file to pass the health check, wait a couple of seconds and then bring the up site down. Bringing the up site down can be done with or without draining the server in the Server Farm first.
To disallow new connections on the server that is up before bringing it down, we can call the
SetState method with the following values.
After setting the state to
Drain it would be possible to loop until the server has
0 connections, then make health check fail and make the server “Available” and “Unhealthy” - ready for the next deployment.
If SSL needs to be terminated in IIS it will now have to be done at the Server Farm. To do this a new URL Rewrite rule has to be created to match port
443. Make sure that the certificate to be used is bound to at least 1 site in IIS, or use IIS’s centralized certificates.
When ARR terminates SSL it will add a server variable of
HTTP_X_ARR_SSL. Checking this server variable is useful for preventing redirect loops in applications that need to do HTTPS redirection.
Other things to consider
- If a deployed application uses ASP.NET
InProcsession, all session data will be lost when requests are routed to a different site. Use
SQLServer, or a custom session state provider in order to retain session state across deployments.
- Don’t autogenerate or use a different
machineKeyin blue and green ASP.NET sites so that Session, ViewState, and Forms Authentication will be decrypted the same for each.
- Make sure the proxy timeout is greater than or equal to your application’s timeout setting. This is often defined in an ASP.NET application’s
- Caching on the Server Farm is enabled by default. Consider disabling it if you have a different caching strategy.
- If you choose not to use a health check and instead servers are only set to “Unavailable” between deployments, note that when IIS restarts, all sites (both blue and green) will be “Available” and “Healthy”. As a precaution to this, it is recommended to enable one of the Server Affinity options to make requests “sticky”.
- Martin Fowler’s 2010 definition of “Blue green deployment”
- Server Fault answer from Matt Bathje outlining a “blue green deployment” process
Microsoft.Web.Administration.ServerManagerPowerShell samples for setting state of servers in a Server Farm
- “Blue green deployment” PowerShell scripts from yosoyadri (includes useful functions for creating IPs, creating Sites, creating Server Farms, and archiving previous deployments in
- Documentation on “blue green deployments” from Octopus Deploy