Kevin Reed

Updating Rows with Window Functions in Postgres

Fri, 08 Sep 2023 00:00:00 +0000

Limits of `DISTINCT ON`

I had a problem recently where I needed to modify a script that would update 1 row per distinct values of a column to instead update 2 rows per grouping. The data in the table looked something like this.

Id	TypeA	TypeB
1	A1	B1
2	A1	B1
3	A2	B1
4	A2	B1
5	A3	B1
6	A3	B1
7	A1	B2
8	A1	B2
9	A2	B2
10	A2	B2
11	A3	B2
12	A3	B2

This meant that the DISTINCT ON in the following SQL was too limiting, and I would have to do something more creative.

UPDATE "Table"
SET "Flag" = '✔️'
WHERE "Id" IN (
    SELECT DISTINCT ON (s."TypeB") s."Id"
    FROM "Table" s
    WHERE s."TypeA" = 'A1'
);

UPDATE "Table"
SET "Flag" = '✔️'
WHERE "Id" IN (
    SELECT DISTINCT ON (s."TypeB") s."Id"
    FROM "Table" s
    WHERE s."TypeA" = 'A2'
);

This would update the table to look like this

Id	TypeA	TypeB	Flag
1	A1	B1	✔️
2	A1	B1
3	A2	B1	✔️
4	A2	B1
5	A3	B1
6	A3	B1
7	A1	B2	✔️
8	A1	B2
9	A2	B2	✔️
10	A2	B2
11	A3	B2
12	A3	B2

But what I really wanted was to update 2 rows per TypeB grouping, so that the table would look like this. Notice we don’t want to touch the A3 rows, nor any other TypeA values other than A1 and A2.

Id	TypeA	TypeB	Flag
1	A1	B1	✔️
2	A1	B1	✔️
3	A2	B1	✔️
4	A2	B1	✔️
5	A3	B1
6	A3	B1
7	A1	B2	✔️
8	A1	B2	✔️
9	A2	B2	✔️
10	A2	B2	✔️
11	A3	B2
12	A3	B2

Using Window Functions

To update more than 1 row per TypeB grouping, we’ll need both of these Postgres functions

ROW_NUMBER() (see Window Functions Docs docs)
PARTIION BY (see Window Functions Tutorial tutorial)

To start we can number each row by TypeB like this

SELECT ROW_NUMBER() OVER (PARTITION BY "TypeB"), "TypeB"
FROM "Table"

This returns something like

ROW_NUMBER	TypeB
1	B1
2	B1
…	B1
1	B2
2	B2
…	B2

Knowing that PARTITION BY creates subsets of each group, and that ROW_NUMBER() can index each row in each group starting with 1, then we can do this in our update statement

UPDATE "Table"
SET "Flag" = '✔️'
WHERE "Id" IN (
    SELECT t."Id"
    FROM (
        SELECT "Id", ROW_NUMBER() OVER (PARTITION BY "TypeB") AS row_number
        FROM "Table"
        WHERE "TypeA" = 'A1'
    ) t
    WHERE t.row_number < 3
);

UPDATE "Table"
SET "Flag" = '✔️'
WHERE "Id" IN (
    SELECT t."Id"
    FROM (
        SELECT "Id", ROW_NUMBER() OVER (PARTITION BY "TypeB") AS row_number
        FROM "Table"
        WHERE "TypeA" = 'A2'
    ) t
    WHERE t.row_number < 3
);

This does it!

Reducing Repetition

How can I combine the two UPDATE statements for both A1 and A2? Postgres does have a FOR loop that looks like this:

FOR i IN 1..10 LOOP
	-- ...
END LOOP;

The FOR loop does work, but using it is not as simple as you would hope. Postgres can run the FOR inside of a DO block that runs code for a given language. In the following snippet, the language is plpgsql which is Postgres’ own procedural language. The DECLARE block is where we can declare variables, and the BEGIN block is where we can run our FOR loop.

DO $$
DECLARE a_idx integer;
BEGIN 
    FOR a_idx IN 1..2 LOOP 
        UPDATE "Table"
        SET "Flag" = '✔️'
        WHERE "Id" IN (
            SELECT s."Id"
            FROM (
                SELECT "Id", ROW_NUMBER() OVER (PARTITION BY "TypeB") AS row_number
                FROM "Table"
                WHERE "TypeA" = 'A' || a_idx
            ) s
            WHERE s.row_number < 6
        );
    END LOOP;
END $$;

This is much better than having to write out the UPDATE statement twice, and it’s also more flexible. If we wanted to update more rows per grouping, we could just change the WHERE clause to WHERE s.row_number < 6 and so on. And if we wanted to update more TypeA values, we could just change the FOR loop to FOR a_idx IN 1..3 LOOP and so on. If we wanted to loop over a list of strings instead of integers we could do that too, though, counterintuitively it does require a bit more verbosity.

DO $$
DECLARE
    a_arr text[] := ARRAY['A1', 'A2'];
    a text;
BEGIN
    FOREACH a IN ARRAY a_arr LOOP
        UPDATE "Table"
        SET "Flag" = '✔️'
        WHERE "Id" IN (
            SELECT s."Id"
            FROM (
                SELECT "Id", ROW_NUMBER() OVER (PARTITION BY "TypeB") AS row_number
                FROM "Table"
                WHERE "TypeA" = a
            ) s
            WHERE s.row_number < 6
        );
    END LOOP;
END $$;

Conclusion

DISTINCT ON can be used to select 1 row per distinct values of a column
ROW_NUMBER() is a window function that can index rows in a given group
PARTITION BY can be used to create groups of rows
FOR loops can be used to run the same code multiple times
DO $$ can be used to run code in a given language
DECLARE blocks can be used to declare variables
FOR ... IN ... LOOP can be used to iterate over a range of values
FOREACH ... IN ARRAY ... LOOP can be used to iterate over an array of values
|| can be used to concatenate strings
text[] := ARRAY[...] can be used to declare and create an array of strings

Creating a Command-Based CLI in PowerShell

Wed, 14 Apr 2021 00:00:00 +0000

Lately, I’ve become obsessed with automating local development environment tasks, especially when it comes to onboarding, bootstrapping, and managing local machine environments for various projects. When something like a web application starts to add external dependencies like services running in docker-compose, quickly that project’s onboarding steps can get quite lengthy. Managing issues around onboarding steps can become a pain, and troubleshooting them, especially when those tasks are expected to be run across operating systems, can be very difficult and time consuming. PowerShell Core, being cross-platform and JIT runnable, offers a friendly scripting experience for creating command-based command line interfaces for whatever application you need.

Prerequisites

Install PowerShell Core on your operating system of choice: https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell
All the examples can also be found here. Download/clone this repo to follow along with each step: https://github.com/kavun/ps-cli

Goal

Let’s build a script with these commands:

.\cli.ps1 up
.\cli.ps1 down
.\cli.ps1 build
.\cli.ps1 test
.\cli.ps1 migrate <name>
.\cli.ps1 ip

Validate the commands

To enable these ad-hoc command names (without the default - prefix that PowerShell params normally require), we can use a combination of ValidateSet and Parameter(Position=0) attributes on the script’s first param.

1-validate.ps1

param(
  [Parameter(Position=0, Mandatory=$True)]
  [ValidateSet("up", "down", "build", "test", "migrate", "ip")]
  [string]$Command
)

Write-Host $Command

This ensures that when you pass an incorrect command or if you exclude the command, you get a detailed error message.

Handle the commands

Now we need to handle the commands and run functions for each one. For this, we’ll use a switch.

2-handle.ps1

param(
  [Parameter(Position=0, Mandatory=$True)]
  [ValidateSet("up", "down", "build", "test", "ip")]
  [string]$Command
)

function Command-Up    { iex "docker compose up" }
function Command-Down  { iex "docker compose down" }
function Command-Build { iex "dotnet build" }
function Command-Test  { iex "dotnet test" }
function Command-Ip {
    $ip = iwr "https://api.ip.sb/ip" `
        | Select-Object -Expand Content
    
    Write-Host "Your IP address is: " -N
    Write-Host $ip -F Green
}

switch ($Command) {
    "up"    { Command-Up }
    "down"  { Command-Down }
    "build" { Command-Build }
    "test"  { Command-Test }
    "ip"    { Command-Ip }
}

Now, when a command is passed, the associated function is run.

Support `Get-Help`

This is great, but it’s not immediately known what commands are available without opening up the .ps1 file and reading it. We could add a command help and then spit out some help content with Write-Host "help string", but there’s a better way. PowerShell scripts can hook into the Get-Help command to provide structured help documentation for the whole script. Let’s do both!

3-help.ps1

<#
.SYNOPSIS
This script is an example of how any .ps1 script can provide content to `Get-Help`.

.DESCRIPTION
USAGE
    .\3-help.ps1 <command>

COMMANDS
    up          run `docker-compose up`
    down        run `docker-compose down`
    build       run `dotnet build`
    test        run `dotnet test`
    ip          get your local ip
    help, -?    show this help message
#>
param(
  [Parameter(Position=0)]
  [ValidateSet("up", "down", "build", "test", "ip", "help")]
  # The command to run
  [string]$Command
)

function Command-Help { Get-Help $PSCommandPath }

if (!$Command) {
    Command-Help
    exit
}

function Command-Up    { iex "docker compose up" }
function Command-Down  { iex "docker compose down" }
function Command-Build { iex "dotnet build" }
function Command-Test  { iex "dotnet test" }
function Command-Ip {
    $ip = iwr "https://api.ip.sb/ip" `
        | Select-Object -Expand Content
    
    Write-Host "Your IP address is: " -N
    Write-Host $ip -F Green
}

switch ($Command) {
    "up"    { Command-Up }
    "down"  { Command-Down }
    "build" { Command-Build }
    "test"  { Command-Test }
    "ip"    { Command-Ip }
    "help"  { Command-Help }
}

Let’s look at what we changed:

Added <# #> help content at the top of the file. And added .SYNOPSIS and .DESCRIPTION sections.
Removed Mandatory=$True from the $Command param, so that we can show the help content when you call the script with no params.
Added the help command, which ends up calling Get-Help $PSCommandPath

This will show help for all 4 of these commands. The goal here is to make finding help foolproof.

.\3-help.ps1

.\3-help.ps1 help

.\3-help.ps1 -?

Get-Help .\3-help.ps1

Handle nested commands

What we have is great for simple one-liners, but eventually we’ll want to pass params to the commands and also support nested commands.

4-nest.ps1

<#
.SYNOPSIS
An example of how to next commands.

.DESCRIPTION
USAGE
    .\4-nest.ps1 <command>

COMMANDS
    speak       speak some text
    logs        view logs
#>
param(
    [Parameter(Position=0)]
    [ValidateSet("speak", "logs")]
    [string]$Command,

    [Parameter(Position=1, ValueFromRemainingArguments=$true)]
    $Rest
)

if (!$Command) {
    Get-Help $PSCommandPath
    exit
}

function Command-Speak {
    param (
        [Parameter(Position=0, Mandatory=$True)]
        [string]$Text
    )

    if (!$IsWindows) {
        Write-Error "Sorry, the 'speak' command is only supported on Windows"
        exit
    }

    Write-Host "Speaking the text: '$Text'"
    $sp = New-Object -ComObject SAPI.SpVoice
    $sp.Speak($Text) | Out-Null
}

function Command-Logs {
<#
.SYNOPSIS
Show logs for an application.

.DESCRIPTION
USAGE
    .\4-nest.ps1 logs <command>

COMMANDS
    app-one     show logs for app-one
    app-two     show logs for app-two
#>
    param(
        [Parameter(Position=0)]
        [ValidateSet("", "app-one", "app-two")]
        [string]$App
    )

    if (!$App) {
        Get-Help Command-Logs
        exit
    }

    Write-Host "Showing logs for " -N
    Write-Host $App -F Green
}

switch ($Command) {
    "speak" { Command-Speak $Rest }
    "logs"  { Command-Logs $Rest }
}

Let’s look at what we did:

We used ValueFromRemainingArguments to capture any trailing params in the $Rest param and passed those along to the inner functions.
We used ValidateSet on an inner function’s param, just like we did on the script’s params.
Added help docs to one of the inner functions.

These are just examples, but you can see the range of possibilities that open up when we start nesting commands and forwarding params. Here’s what this example looks like in use:

Resources

Scripts from examples: https://github.com/kavun/ps-cli
ValidateSet: https://docs.microsoft.com/en-us/powershell/scripting/developer/cmdlet/validateset-attribute-declaration
ValueFromRemainingArguments: https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_functions_advanced_parameters#valuefromremainingarguments-argument
about_Comment_Based_Help (<# #>): https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_comment_based_help

How to Run Dotnet Test Alongside Docker Compose in GitLab CI/CD

Tue, 05 Jan 2021 00:00:00 +0000

To run dotnet test in GitLab and have it connect to and use containerized services like databases, caches, or in this case Selenium Hub, you can use the following pattern:

Rely on GitLab CI’s built-in docker:dind (“docker in docker”) service to run docker commands as job scripts.
Install docker-compose manually (if you know of a way to somehow use an image instead for this, let me know!)
Run docker-compose using a test-specific docker-compose.yml.
Run dotnet test using docker run [...] mcr.microsoft.com/dotnet/sdk:5.0 dotnet test [...].

Example xUnit Test Class

This is using xUnit, pulling the Selenium Hub URL from appsettings.json, and grabbing the title of google.com.

SomeBrowserTests.cs

public class SomeBrowserTests
{
    [Fact]
    public void GetTitleTest()
    {
        var env = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT");
        var configuration = new ConfigurationBuilder()
            .AddJsonFile("appsettings.json")
            .AddJsonFile($"appsettings.{env}.json", optional: true)
            .Build();
        var webDriverHubUrl = configuration.GetSection("Selenium")["HubUrl"];
        var options = new ChromeOptions();
        var hubUri = new Uri(webDriverHubUrl);
        using var driver = new RemoteWebDriver(hubUri, options);

        driver.Url = "https://www.google.com";
        var title = driver.Title;

        Assert.Contains("Google", title);

        driver.Quit();
    }
}

Example `appsettings.json`

This is the appsettings.json that GitLab CI will use. For local development it’s possible to use a separate appsettings.json pointed to localhost, since a local docker-compose setup for development would expose the necessary ports on localhost. However, GitLab CI creates a new container for each job script, so anything running there will have to connect from the a test container to the Selenium Hub container using the container name that docker provides via it’s magic DNS stuff. (this took me a long time to figure out, and adding a job script of docker network ls helped.)

appsettings.CI.json

{
  "Selenium": {
    "HubUrl": "http://some-selenium-hub:4444/wd/hub"
  }
}

Example `docker-compose.yml`

Nothing fancy here (this is close to what the Selenium docs show), other than to note what network name (bridged by default) is used (some-tests) because it will be used later when running dotnet test.

docker-compose.some-tests.yml

version: "3.3"

networks:
  some-tests:

services:

  some-selenium-chrome:
    image: selenium/node-chrome:4.0
    restart: always
    volumes:
      - /dev/shm:/dev/shm
    environment:
      - SE_EVENT_BUS_HOST=some-selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
    entrypoint: bash -c 'SE_OPTS="--host $$HOSTNAME" /opt/bin/entry_point.sh'
    networks:
      - some-tests
    depends_on:
      - some-selenium-hub

  some-selenium-hub:
    image: selenium/hub:4.0
    restart: always
    ports:
      - "4442:4442"
      - "4443:4443"
      - "4444:4444"
    networks:
      - some-tests

Example `.gitlab-ci.yml`

Here’s where the rubber meets the road.

Alpine now allows apk add docker-compose (so there’s no need to mess with python/pip installation of docker-compose 🙌).
After docker-compose is installed, start the Selenium Hub.
Then run the dotnet tests:
- Notice it runs the dotnet/sdk image in the same some-tests network (so that it can connect to http://some-selenium-hub:4444/wd/hub)
- Notice it sets the ASPNETCORE_ENVIRONMENT=CI (so that the test will read from the appsettings.CI.json configuration)
- Notice it mounts -v $PWD:/app so that the output from --logger "junit" will be readable by the GitLab CI task runner to capture test results. This requires the JUnitTestLogger nuget package to be installed in the test project(s).
- Notice it calls docker-compose down, and I’m not entirely sure that’s necessary – I just like to be a good citizen.

.gitlab-ci.yml

image: docker:latest
services:
  - name: docker:dind
    entrypoint: ["dockerd-entrypoint.sh"]
    command: ["--max-concurrent-downloads", "6"]

stages:
  - test

variables:
  DOCKER_DRIVER: overlay2

some-tests:
  stage: test
  before_script:
    - apk add docker-compose
  script:
    - docker-compose -f docker-compose.some-tests.yml up -d --build --remove-orphans
    - docker run --network some-tests -v $PWD:/app -w /app -e "ASPNETCORE_ENVIRONMENT=CI" mcr.microsoft.com/dotnet/sdk:5.0-alpine dotnet test SomeSolution.sln --logger "junit"
    - docker-compose -f docker-compose.some-tests.yml down
  artifacts:
    reports:
      junit: ./**/TestResults/TestResults.xml

How to become a Master Text Wrangler with Regular Expressions

Fri, 05 Feb 2016 00:00:00 +0000

I wrote a post as an intro to Regular Expressions over on the madebyspeak.com blog. ☛ Check it out!

Low Quality Image Placeholders with PowerShell and ImageMagik

Sat, 12 Dec 2015 00:00:00 +0000

Lazy loading images can be great for page performance, but what if we want low quality image placeholders to load before the full lazy loaded image? ImageMagik’s convert can be used to create LQIPs, and PowerShell can be used to call convert and generate base 64 strings of the LQIPs for inlining into web pages. Here are some PowerShell functions that will:

Find all .pngs files or files with an $extension in a given $path.
If LQIP doesn’t already exist, pass the image to convert $($file) -resize 1% -gaussian-blur 0.05 -quality 1 $($file).lqip.jpg (these parameters will need to be tweaked based on the end goal, for example you may not want to resize to 1%)
Pass LQIP to [convert]::ToBase64String((Get-Content $path -Encoding Byte)) to generate base 64 string. Output base 64 string, along with the width and the height of the original image to a file ending in .b64.

Make sure that ImageMagik is installed and that convert is available on the PATH.

Function Generate-LQIP {
    Param (
        [string]$path=".",
        [string]$extension=".png"
    )

    Add-Type -AssemblyName System.Drawing
    Set-Location $path

    Get-ChildItem $path -Filter "*$($extension)" | % {
        $nameStub = $_.Name -replace '\$($extension)$', ''
        $newFileName = "$($nameStub).lqip.jpg"
        $oldLength = (Get-Item $_).Length
        $png = New-Object System.Drawing.Bitmap $_.FullName

        If ((Test-Path $newFileName) -eq $true) {
            $newLength = (Get-Item $newFileName).Length
            Write-Host "Already exists: $($newFileName) $($oldLength)b > $($newLength)b"
        } Else {
            Invoke-Expression "convert $($_.Name) -resize 1% -gaussian-blur 0.05 -quality 1 $($newFileName)"
            Generate-B64 $newFileName >> "$($newFileName).b64"
            $png.Width >> "$($newFileName).b64"
            $png.Height >> "$($newFileName).b64"
            $newLength = (Get-Item $newFileName).Length
            Write-Host "Created $($newFileName) $($oldLength)b > $($newLength)b"
        }
    }
}

Function Generate-B64 {
    Param (
        [String]$path
    )

    [convert]::ToBase64String((Get-Content $path -Encoding Byte))
}

To be honest, I’m not actually using LQIPs on this blog or anywhere else for that matter. Usually just lazy loading images is quick enough that LQIPs are uneccessary and actually degrade performance since it’s extra bytes that the page has to load before rendering. Nevertheless, PowerShell and ImageMagik prove to work nicely together making it easy to do automated bulk image processing.

Cherry Logging Git Branches

Sat, 12 Dec 2015 00:00:00 +0000

In a previous post I showed how to compare branches with git log master..topic. This method is great, but it only shows commits in topic not yet in master based on a sha1 comparison. If we have another branch called shared that has been merged into master and topic, git log master..topic would show the commits that were merged from shared even though they have been merged into topic and master. If we use git cherry -v topic master or git log --cherry master..topic, the actual diffs from each commit will be compared instead of only the sha1. This will make sure the commits from shared will not show up. Here is an example.

Notice how the commit Merge branch 'shared' into topic doesn’t really need to show up as part of topic when comparing to master because master also has that merge. --cherry compares the diff of the merge so that it’s not included in the command git log --oneline --cherry master..topic. This would also ignore shared cherry picked commits.

Analogy for the Web

Sat, 12 Dec 2015 00:00:00 +0000

What is the web made of? This is how I explain it to those who have never built on the web before.

HTML elements are the bones
Javascript is the muscle
CSS is the skin

Comparing Git Branches

Sun, 08 Nov 2015 00:00:00 +0000

The majority of the time I manage Git repositories in SourceTree because wrangling lots of Git projects on the command line can get hairy. The visibility of statuses across mutiple local repositories that a GUI like SourceTree provides is not available on the command line. However, some things are often simpler and quicker on the command line than in GUI Git clients. One of those things is comparing branches.

git log <since>..<until>

This will show any commits that happened in the <until> branch that are not in <since>. Often I will use this command to compare a feature branch with a master branch before merging, just to make sure I’m not missing anything. Note that <until> is optional and if omitted will default to the current checked out branch.

git log master..

This also works with tags.

git log 2.4.0..

This is useful as a pre deployment task to check what’s about to be deployed, create changelogs for a new tag, etc. git shortlog can also be useful for changelogs.

Useful flags for `git log`

`--oneline`

Condense log messages to a single line to be more readable.

`--stat`

Show files that have changed. This can be extremely useful when combined with <since>.. and grep. The following will output filenames of all .png files in the current checked out branch that are not in master.

git log --oneline --stat master.. | grep .png

Usually this is accurate enough, but grep could also accept a regular expression of -e ".*\.png\b"

Resources

git log documentation
Prettier git log with --graph and --decorate

How to Deploy Anything in IIS with Zero Downtime on a Single Server

Sat, 07 Nov 2015 00:00:00 +0000

No one likes downtime. It is stressful for managers, operations, and developers. Downtime is frustrating and confusing for users of a site, whether or not the “down for maintenance” page is live. Most of the popular deployment strategies for ASP.NET apps hosted in IIS do not even try to deploy without downtime. Any change to Web.config or .dll files on a live site can cause significant downtime (I’ve seen 5-10 minutes for large apps). There is a better way. It is possible to achieve zero downtime with the “blue green deployment” strategy in IIS using Application Request Routing, and URL Rewrite. Let’s get started.

The general idea of “blue green deployment” is that there is an entry point (load balancer) that routes requests to a site that is up. An application is deployed to a site that is down, that application is warmed up, then the entry point is notified to route new requests to the newly warmed up site instead of the old one. The entry point needs to be able to make the switch with no hiccups, no lost connections, low CPU/memory overhead, and definitely without any perceived downtime by end users. Application Request Routing can act as that load balancing entry point for us.

Prerequisites

IIS 7 or higher
Application Request Routing

Tutorial

Start with a basic static site called alwaysup. Keep in mind that this site could be any IIS hosted application (ASP.NET, HttpPlatformHandler, iisnode, PHP, etc). A static site will be enough for this example. There will need to be two instances of alwaysup in IIS, so duplicate the folder that alwaysup lives in to create /alwaysup-green and /alwaysup-blue. The application files are simple. Just some index.html files to indicate which application is being hit.

Create corresponding IIS sites for each application and name them alwaysup-green and alwaysup-blue. These sites need to be bound to a unique port that is not port 80, like 8001 and 8002.

Also create some host entries for each site so that when the Server Farm is created later each unique host name will act as a server address in the Server Farm. Use alwaysup-blue and alwaysup-green for the host names, and also add a host name for the server farm that will act as the entry point for the application alwaysup.

0.0.1 alwaysup
0.0.1 alwaysup-blue
0.0.1 alwaysup-green

This is the complete configuration for the alwaysup-blue and alwaysup-green sites.

<site name="alwaysup-blue" id="3" serverAutoStart="true">
    <application path="/" applicationPool="DefaultAppPool">
        <virtualDirectory path="/" physicalPath="C:\Projects\alwaysup\alwaysup-blue" />
    </application>
    <bindings>
        <binding protocol="http" bindingInformation="*:8001:" />
    </bindings>
</site>
<site name="alwaysup-green" id="4" serverAutoStart="true">
    <application path="/" applicationPool="DefaultAppPool">
        <virtualDirectory path="/" physicalPath="C:\Projects\alwaysup\alwaysup-green" />
    </application>
    <bindings>
        <binding protocol="http" bindingInformation="*:8002:" />
    </bindings>
</site>

Next create a Server Farm in IIS to route traffic between the two sites. Add a Server Farm named alwaysup.

Next add a server for each host name binding of the alwaysup-green:8001 and alwaysup-blue:8002 sites.

After creating the servers, IIS will ask prompt to create a URL rewrite rule to route all incoming requests to this server farm automatically. Select “No” because a rule will be created later manually with the URL Rewrite module.

Next add some health checks to the server farm. This is how Application Request Routing will know which site is “up” to route requests to. A simple way to do this is to add an up.html file in the root of alwaysup-blue and alwaysup-green with the text “up” in one, and “down” in the other.

Then add a health check to the alwaysup Server Farm that makes sure the response received from /up.html contains the text “up”. Set the polling interval to be quick so that when changes are made to the health check files very little wait time is required for the health checks to fail or pass. Notice also that the alwaysup HOSTS entry is used here.

The “Monitoring and Management” page of the Server Farm shows that one of the Server Farm servers is marked as unhealthy - which is to be expected.

This is the complete configuration for the alwaysup Server Farm.

<webFarm name="alwaysup" enabled="true">
    <server address="alwaysup-blue" enabled="true">
        <applicationRequestRouting httpPort="8001" />
    </server>
    <server address="alwaysup-green" enabled="true">
        <applicationRequestRouting httpPort="8002" />
    </server>
    <applicationRequestRouting>
        <healthCheck url="http://alwaysup/up.html" interval="00:00:01" responseMatch="up" />
    </applicationRequestRouting>
</webFarm>

Now route the actual traffic to our Server Farm alwaysup. To do this use the URL Rewrite module in IIS. Requests will be made on host name alwaysup, so add a URL Rewrite rule to match on {HTTP_HOST} alwaysup on port 80. Route this traffic to the alwaysup Server Farm. One thing to note is that IIS needs at least one site listening on port 80 for our URL Rewrite to work.

<rule name="alwaysup" stopProcessing="true">
    <match url=".*" />
    <conditions>
        <add input="{HTTP_HOST}" pattern="^alwaysup$" />
        <add input="{SERVER_PORT}" pattern="^80$" />
    </conditions>
    <action type="Rewrite" url="http://alwaysup/{R:0}" />
</rule>

Test everything done so far by making a request to http://alwaysup/.

Make one of the sites fail the health test and the other pass and see the site content change.

At this point any deployment strategy could be used to deploy to a site in IIS, except now it is necessary to check which site is down and deploy to that one instead of the one that is up. Deployment can happen via WebDeploy/msdeploy, FTP, Dropbox (yes - some people deploy with Dropbox), an Octopus Deploy Tentacle, etc. After new code has been deployed to the site that is down, the down site needs to be warmed up before its health check can be changed to a passing state. To do this, make a request to the site manually. In this example, a request would be made to http://alwaysup-blue:8001 to initiate a warm up in IIS. Then edit the health check file to make it pass the health check by changing down to up. Then once the server’s health status is “Healthy”, the initial site that was up can safely be brought down. All of this is done without introducing any downtime.

No more 4:30am deployments!

Automating with PowerShell

Doing all of this manually can be tedious, and should be automated so that steps aren’t accidentally skipped or done incorrectly. Here are some PowerShell snippets to achieve what was done manually in this example.

Check which site is unhealthy and ready for deployment

The quickest way would be to check the contents of the up.html file with Get-Content, essentially mimicking the health check in the IIS Server Farm.

$bluePath = "C:\Projects\alwaysup\alwaysup-blue"
$greenPath = "C:\Projects\alwaysup\alwaysup-green"

$upPath = @($bluePath, $greenPath) | Where {
    (Get-Content "$($_)\up.html") -contains "up"
}

$downPath = if ($upPath -eq $bluePath) {
    $greenPath
} else {
    $bluePath
}

Write-Host "$($upPath) is up"
Write-Host "$($downPath) is down"

We can also check the status of the Server Farm itself in IIS through Microsoft.Web.Administration.ServerManager.

First define a function to return a Server Farm by name.

function Get-ServerFarm {
    param ([string]$webFarmName)

    $assembly = [System.Reflection.Assembly]::LoadFrom("$env:systemroot\system32\inetsrv\Microsoft.Web.Administration.dll")
    $mgr = new-object Microsoft.Web.Administration.ServerManager "$env:systemroot\system32\inetsrv\config\applicationhost.config"
    $conf = $mgr.GetApplicationHostConfiguration()
    $section = $conf.GetSection("webFarms")
    $webFarms = $section.GetCollection()
    $webFarm = $webFarms | Where {
        $_.GetAttributeValue("name") -eq $serverFarmName
    }

    $webFarm
}

Then get the isHealthy status from the ARR counters.

$serverFarmName = "alwaysup"
$webFarm = Get-ServerFarm $serverFarmName
$servers = $webFarm.GetCollection()

$servers | % {
    $arr = $_.GetChildElement("applicationRequestRouting")
    $counters = $arr.GetChildElement("counters")
    $isHealthy = $counters.GetAttributeValue("isHealthy")
    if ($isHealthy) {
        $healthyNode = $_
    } else {
        $unhealthyNode = $_
    }
}

$healthyAddress = $healthyNode.GetAttributeValue("address")
$unhealthyAddress = $unhealthyNode.GetAttributeValue("address")

Write-Host "$($healthyAddress) is up"
Write-Host "$($unhealthyAddress) is down"

Deploy!

At this point use whetever deployment method necessary to deploy to the site that is down.

Warming up after deployment

I’m still trying to figure out how best to warm up a site without setting a timeout, but the idea so far has been to Invoke-WebRequest to the down site until a response comes back within a short enough time period.

$blueSite = "http://alwaysup-blue:8001"
$greenSite = "http://alwaysup-green:8002"
$minTime = 400

@($blueSite, $greenSite) | % {
    Write-Host "Warming up $($_)"
    Do {
        $time = Measure-Command {
            $res = Invoke-WebRequest $_
        }
        $ms = $time.TotalMilliSeconds
        If ($ms -ge $minTime) {
            Write-Host "$($res.StatusCode) from $($_) in $($ms)ms" -foreground "yellow"
        }
    } While ($ms -ge $minTime)
    Write-Host "$($res.StatusCode) from $($_) in $($ms)ms" -foreground "cyan"
}

Then change the content of the up.html file to pass the health check, wait a couple of seconds and then bring the up site down. Bringing the up site down can be done with or without draining the server in the Server Farm first.

$siteBlue = "http://alwaysup-blue:8001"
$siteGreen = "http://alwaysup-green:8002"
$pathBlue = "C:\Projects\alwaysup\alwaysup-blue"
$pathGreen = "C:\Projects\alwaysup\alwaysup-green"
$pathBlueContent = (Get-Content $pathBlue\up.html)
$serverFarmName = "alwaysup"
$webFarm = Get-ServerFarm $serverFarmName
$webFarmArr = $webFarm.GetChildElement("applicationRequestRouting")
$webFarmHeathCheck = $webFarmArr.GetChildElement("healthCheck")
$healthCheckTimeoutS = $webFarmHeathCheck.GetAttributeValue("interval").TotalSeconds

$siteToWarm = $siteBlue
$pathToBringDown = $pathGreen
$pathToBringUp = $pathBlue

if ($pathBlueContent -contains 'up')
{
    $siteToWarm = $siteGreen
    $pathToBringUp = $pathGreen
    $pathToBringDown = $pathBlue
}

Write-Host "Warming up $($siteToWarm)"
Do {
    $time = Measure-Command {
        $res = Invoke-WebRequest $siteToWarm
    }
    $ms = $time.TotalMilliSeconds
    If ($ms -ge 400) {
        Write-Host "$($res.StatusCode) from   $($siteToWarm) in $($ms)ms" -foreground "yellow"
    }
} While ($ms -ge 400)
Write-Host "$($res.StatusCode) from $($siteToWarm) in $($ms)ms" -foreground "cyan"

if ($res.StatusCode -eq 200) {
    Write-Host "Bringing $($pathToBringUp) up" -foreground "cyan"
    (Get-Content $pathToBringUp\up.html).replace('down', 'up') | Set-Content $pathToBringUp\up.html

    Write-Host "Waiting for health check to pass in $($healthCheckTimeoutS) seconds..."
    Start-Sleep -s $healthCheckTimeoutS

    Write-Host "Bringing $($pathToBringDown) down"
    (Get-Content $pathToBringDown\up.html).replace('up', 'down') | Set-Content $pathToBringDown\up.html
} else {
    Write-Host "Cannot warm up $($siteToWarm)" -foreground "red"
}

To disallow new connections on the server that is up before bringing it down, we can call the SetState method with the following values.

$serverFarmName = "alwaysup"
$serverAddress = "alwaysup-green"

$webFarm = Get-ServerFarm $serverFarmName
$servers = $webFarm.GetCollection()

$server = $servers | Where {
    $_.GetAttributeValue("address") -eq $serverAddress
}

$arr = $server.GetChildElement("applicationRequestRouting")
$method = $arr.Methods["SetState"]
$methodInstance = $method.CreateInstance()

# 0 = Available
# 1 = Drain
# 2 = Unavailable
# 3 = Unavailable Gracefully
$methodInstance.Input.Attributes[0].Value = 1
$methodInstance.Execute()

After setting the state to Drain it would be possible to loop until the server has 0 connections, then make health check fail and make the server “Available” and “Unhealthy” - ready for the next deployment.

Enabling SSL

If SSL needs to be terminated in IIS it will now have to be done at the Server Farm. To do this a new URL Rewrite rule has to be created to match port 443. Make sure that the certificate to be used is bound to at least 1 site in IIS, or use IIS’s centralized certificates.

<rule name="alwaysup HTTPS" stopProcessing="true">
    <match url=".*" />
    <conditions>
        <add input="{SERVER_PORT}" pattern="^443$" />
        <add input="{HTTP_HOST}" pattern="^alwaysup$" />
    </conditions>
    <action type="Rewrite" url="http://alwaysup/{R:0}" />
</rule>

When ARR terminates SSL it will add a server variable of HTTP_X_ARR_SSL. Checking this server variable is useful for preventing redirect loops in applications that need to do HTTPS redirection.

Other things to consider

If a deployed application uses ASP.NET InProc session, all session data will be lost when requests are routed to a different site. Use StateServer, SQLServer, or a custom session state provider in order to retain session state across deployments.
Don’t autogenerate or use a different machineKey in blue and green ASP.NET sites so that Session, ViewState, and Forms Authentication will be decrypted the same for each.
Make sure the proxy timeout is greater than or equal to your application’s timeout setting. This is often defined in an ASP.NET application’s Web.config as <httpRuntime executionTimeout="120"/>
Caching on the Server Farm is enabled by default. Consider disabling it if you have a different caching strategy.
If you choose not to use a health check and instead servers are only set to “Unavailable” between deployments, note that when IIS restarts, all sites (both blue and green) will be “Available” and “Healthy”. As a precaution to this, it is recommended to enable one of the Server Affinity options to make requests “sticky”.

Other resources

Martin Fowler’s 2010 definition of “Blue green deployment”
Server Fault answer from Matt Bathje outlining a “blue green deployment” process
Microsoft.Web.Administration.ServerManager PowerShell samples for setting state of servers in a Server Farm
“Blue green deployment” PowerShell scripts from yosoyadri (includes useful functions for creating IPs, creating Sites, creating Server Farms, and archiving previous deployments in .zip files)
Documentation on “blue green deployments” from Octopus Deploy

How to Start Linting Javascript

Tue, 28 Oct 2014 00:00:00 +0000

Javascript is wild. The popular and incredibly accommodating language is never selfish. It will never put you in a box, nor will it keep you a leash. On the contrary, it should be the one on a leash, but instead walks calmly by your side, and only veers when it craves to show you something you’ve never seen before. Problems begin when we are not attentive to its deviations, and quickly become bridled ourselves by our own selfishness and negligence.

Developers and Javascript need checks and balances to keep each other under control. Whether that be guarding against typos, bad syntax, or anti-patterns, linting tools like ESHint, JSCS, JSLint, and JSHint can provide the safety net needed to write clean, lint-free code.

What is “linting”?

The process of linting will analyze uncompiled code for potential known errors. Since Javascript is uncompiled, the only way to check for errors without a linter is to run the code and hope for the best. We can do better.

ESLint

ESLint offers a flexible solution for linting Javascript. ESLint is highly configurable and extendable out of the box, so it can be used in many scenarios. It works well with teams that work on a shared code bases as well as the solo Javascript ninja. Other alternatives are JSLint (less configurable, more strict), and JSHint (configurable, but not extendable). At the time of writing ESLint has not yet reached v1.0.0, but it is established enough to use for linting production code and has implemented all of the JSHint rules plus many more features.

ESLint in Sublime Text 3

Before using ESLint in Sublime Text 3 we will need a variety of frameworks and plugins so that ESLint can lint our Javascript in real time as we type.

Install Node.js.
Install eslint globally via npm with the command npm install -g eslint.
Make sure Package Control for Sublime Text 3 is installed.
Install the package SublimeLinter, which acts as a framework for actual linting packages (not just Javascript).
Install ESLint as a linter for SublimeLinter with the SublimeLinter-contrib-eslint package.

At this point if we create a new file in Sublime Text 3 and set its syntax to Javascript, once we starting writing Javascript we can see errors popping up.

Right away you’ll notice a lot of errors. This is because ESLint is using its default configuration. To change this we can annotate our file with ESLint specific comments, or we can add a .eslintrc file at the root of our project so that ESLint can use that configuration for all .js files in a folder. For more on configuring ESLint refer to their documentation.

Configuration comments `/* eslint ... */`

To configure ESLint rules inline, we can add Javascript comments containing configuration options. Notice, by setting /* eslint no-unused-vars:1, quotes:1 */ our errors have turned into warnings. To remove the errors and warnings completely, set the rule value to 0. The ability to show warnings instead of errors is a very nice addition to ESLint when compared to JSHint.

Configuration files: `.eslintrc`

An .eslintrc file can be created in either JSON or YAML that will provide configuration for an entire folder and all of its subfolders. Using the three sections of env (environment), globals (global variables), and rules, we can provide a tailored configuration for any project.

A simple .eslintrc file in JSON would look like

{
    "globals": {
        "myModule": true
    },
    "env": {
        "browser": true
    },
    "rules": {
        "no-unused-vars": 1,
        "quotes": 1
    }
}

or in YAML

---
  globals
    myModule: true
  env
    browser: true
  rules
    no-unused-vars: 1
    quotes: 1

Find a full list of rules here.

Helpful rules

If you are just getting your feet wet with linting, here are some helpful rules that are enabled by default.

no-cond-assign This rule disallows assignment in conditional expressions, so a warning will occur when a statement looks like if (x = 0) { ... }. Sometimes a second pair of eyes (like a linter!) can be extremely valuable so bugs like this don’t arise too easily.
valid-typeof This rule makes sure that comparisons with the result of typeof are compared against the correct strings. For example, typeof [] === 'array' would create an error. Rookies will find ESLint helpful while they learn the intricacies of the language.

Alternatives

ESLint is one of the newest linting tools available. Other linters have been around much longer and have seen their fill of poorly written code. Definitely check them out before choosing which linter you will use on your next project.

JSHint (configurable, but not extendable)
JSCS (focuses more on style than correctness)
JSLint (strict linter from Douglas Cockford)
Google’s Closure Linter (follows the Google Javascript Style Guide)

Use multiple linters at once

If you use a build tool like Grunt or Gulp you can easily run multiple linters at once. Here is a sample Gulp setup using ESLint, JSHint, and JSCS, assuming you have an .eslintrc, .jshintrc, and .jscsrc already set up at the root of your project at minimum. Check out this gulpfile.js file in context of a sample project at kavun/multiplelinters.

var gulp = require('gulp');
var jscs = require('gulp-jscs');
var jshint = require('gulp-jshint');
var stylish = require('jshint-stylish');
var eslint = require('gulp-eslint');

var files = {
    js: 'src/*.js'
};

gulp.task('jscs', function () {
    return gulp.src(files.js)
        .pipe(jscs());
});

gulp.task('jshint', function() {
    return gulp.src(files.js)
        .pipe(jshint())
        .pipe(jshint.reporter(stylish));
});

gulp.task('eslint', function () {
    return gulp.src(files.js)
        .pipe(eslint())
        .pipe(eslint.format());
});

gulp.task('lint', ['jscs', 'jshint', 'eslint']);

Id	TypeA	TypeB
1	A1	B1
2	A1	B1
3	A2	B1
4	A2	B1
5	A3	B1
6	A3	B1
7	A1	B2
8	A1	B2
9	A2	B2
10	A2	B2
11	A3	B2
12	A3	B2

Id	TypeA	TypeB
1	A1	B1
2	A1	B1
3	A2	B1
4	A2	B1
5	A3	B1
6	A3	B1
7	A1	B2
8	A1	B2
9	A2	B2
10	A2	B2
11	A3	B2
12	A3	B2

Kevin Reed

Updating Rows with Window Functions in Postgres

Limits of DISTINCT ON

Using Window Functions

Reducing Repetition

Conclusion

Creating a Command-Based CLI in PowerShell

Prerequisites

Goal

Validate the commands

Handle the commands

Support Get-Help

Handle nested commands

Resources

How to Run Dotnet Test Alongside Docker Compose in GitLab CI/CD

Example xUnit Test Class

Example appsettings.json

Example docker-compose.yml

Example .gitlab-ci.yml

How to become a Master Text Wrangler with Regular Expressions

Low Quality Image Placeholders with PowerShell and ImageMagik

Cherry Logging Git Branches

Analogy for the Web

Comparing Git Branches

Useful flags for git log

--oneline

--stat

Resources

How to Deploy Anything in IIS with Zero Downtime on a Single Server

Prerequisites

Tutorial

Automating with PowerShell

Check which site is unhealthy and ready for deployment

Deploy!

Warming up after deployment

Enabling SSL

Other things to consider

Other resources

How to Start Linting Javascript

What is “linting”?

ESLint

ESLint in Sublime Text 3

Configuration comments /* eslint ... */

Configuration files: .eslintrc

Helpful rules

Alternatives

Use multiple linters at once

Limits of `DISTINCT ON`

Support `Get-Help`

Example `appsettings.json`

Example `docker-compose.yml`

Example `.gitlab-ci.yml`

Useful flags for `git log`

`--oneline`

`--stat`

Configuration comments `/* eslint ... */`

Configuration files: `.eslintrc`

Id	TypeA	TypeB
1	A1	B1
2	A1	B1
3	A2	B1
4	A2	B1
5	A3	B1
6	A3	B1
7	A1	B2
8	A1	B2
9	A2	B2
10	A2	B2
11	A3	B2
12	A3	B2