Archives for October 2014

SQL 2014 Always-on AG Pt. 12: Kerberos

We are nearing the end of the SQL 2014 AlwaysOn Availability Group series, with just Kerberos and SSL left to configure. In this installment we tackle Kerberos. Depending on your environment, you may or may not need Kerberos configured. Kerberos is only effective when using Windows authentication to your SQL server, not SQL authentication. Should Kerberos authentication fail, it will fail back to NTLM. In case you do have a multi-tiered application that needs Kerberos, let’s get it configured. Microsoft has made some of the Kerberos configuration easy, via a nice GUI tool they created. Unfortunately it is not AAG aware, so there’s still a bit of manual configuration needed. But it helps reduce human error.

Blog Series

SQL 2014 Always-on AG Pt. 1: Introduction
SQL 2014 Always-on AG Pt. 2: VM Deployment
SQL 2014 Always-on AG Pt. 3: Service Accounts
SQL 2014 Always-on AG Pt. 4: Node A Install
SQL 2014 Always-On AG Pt. 5: Unattended Node B
SQL 2014 Always-on AG Pt. 6: Cluster Configuration
SQL 2014 Always-on AG Pt. 7: TempDB
SQL 2014 Always-on AG Pt. 8: Max Mem & Email
SQL 2014 Always-on AG Pt. 9: SQL Maintenance
SQL 2014 Always-On AG Pt. 10: AAG Setup
SQL 2014 Always-On AG Pt. 11: File Share Witness
SQL 2014 Always-On AG Pt. 12: Kerberos
SQL 2014 Always-On AG Pt. 13: SSL

Kerberos Configuration

1. Download the Microsoft Kerberos Configuration Manager for SQL server here. Install it one of your SQL servers.

2. Navigate to C:\Program Files\Microsoft\Kerberos Configuration Manager for SQL Server and launch KerberosConfigMgr.exe

3. Since we are connecting locally you won’t need to enter any connection information.


4. Once connected, navigate to the SPN page. All the way on the right there is a Status column. Unless it says Good with a green checkmark, your SPNs are not configured. You should see a “Missing” status.

5. Click on the two Fix buttons. You should now see a status of Good.

6. Repeat the process on the second SQL server.

7. Go to an Active Directory domain controller and locate the database engine service account. In the advanced view open the Attribute Editor tab and locate the servicePrincipalName entry. Open it, and you should see four entries.

8. We need to add two SPNs, for the AAG listener. Follow the same format as the existing entries, but use the FQDN of your listener computer object. Add a second entry with the port number. You should now have a total of six SPNs.


9. Download my Kerberos PowerShell test script from here. Copy it to a non-SQL server and run it. Enter the FQDN of the first SQL host and the FQDN of the AAG listener. You should see two Online statements. Do not run this from the SQL server, or the authentication method will be shown as NTLM. Run this on a non-SQL AAG server, please.


10. Download my SQL authentication script from here. Open it in SQL Studio and run it. At the bottom of the screen you should see two Kerberos entries. This corresponds to the two connections made from my PowerShell script. If they are shown as NTLM, then Kerberos is not working. Re-run the PowerShell script, but this time connect to the second SQL server and validate Kerberos is working on there as well.

SQL Auth


At this point you’ve now configured Kerberos for both SQL nodes and your AAG listener. You’ve also tested it as well to ensure the configuration works. Sometimes Kerberos can be touchy, and I’ve had situations where even with all the SPNs setup correctly the listener won’t authenticate with Kerberos. So YMMV, and you may need to open a ticket with MS should it not work for you. But at least you know the process that should work for everyone.

Next up in Part 13 is configuring SSL.

Adding a GUI Back to Windows Server Core

The other day I had the occasion where I wanted to add back the Windows Server 2012 R2 GUI to a server core installation. This was a test environment, and for what I was testing I felt the GUI provided a more streamlined experience. Server core certainly has its places, and is great as a hypervisor, appliance, or in high security environments. Installing the GUI, while not difficult, it look quite a bit of Googling and trial and error to find a command that actually worked.

1. RDP into your Core install or use your server’s IPMI/VM console feature, and a command prompt should open. Type powershell.

2. From your original install ISO, copy the \sources\install.wim to your core server.

3. Type the following command and wait several minutes for the install to complete. Include the full path to where you copied your install.wim file.

install-windowsfeature -name server-gui-shell -includemanagementtools -source:wim:c:\install.wim:2



4. After the installation is complete, reboot the server. The reboot process will be quite slow, as it will be configuring the new features for several minutes. Be patient.


SQL 2014 Always-on AG Pt. 11: File Share Witness

Now that we have our SQL AAG up and running, there’s still some configuration left to do. In this installment I cover SQL 2014 file share witness confiugration. In my example I’m doing a 2-node AAG, which means that we need a file share witness to help establish quorum. If you have a NAS appliance, you can easily create a share on there and use it. In my case I’m assuming 100% Windows, so we will be using a third member server as our FSW. There’s nothing too special about this FSW, except for some permissions. Storage space is very minimal.

Blog Series

SQL 2014 Always-on AG Pt. 1: Introduction
SQL 2014 Always-on AG Pt. 2: VM Deployment
SQL 2014 Always-on AG Pt. 3: Service Accounts
SQL 2014 Always-on AG Pt. 4: Node A Install
SQL 2014 Always-On AG Pt. 5: Unattended Node B
SQL 2014 Always-on AG Pt. 6: Cluster Configuration
SQL 2014 Always-on AG Pt. 7: TempDB
SQL 2014 Always-on AG Pt. 8: Max Mem & Email
SQL 2014 Always-on AG Pt. 9: SQL Maintenance
SQL 2014 Always-On AG Pt. 10: AAG Setup
SQL 2014 Always-On AG Pt. 11: File Share Witness
SQL 2014 Always-On AG Pt. 12: Kerberos

Create File Share

1. On a WS2012/R2 member server (not either SQL server) open Server Manager, go to File and Storage Services, click on Shares, then from the Tasks menu select New Share. If you don’t have that option, add the File Server Role and wait for the installation to complete. No reboot is needed.


2. Select the SMB Share – Quick file share profile.

3. Select the appropriate volume the share will be created on.

4. Use a share name in the format of: <Cluster name>-FSW (e.g. SDSQL03-FSW).

5. Enter a description in the format of: <Cluster Name> Cluster File Share Witness.

6. Uncheck allow caching of share and enable encrypt data access.

7. Customize the permission, disable inheritance and remove all inherited permissions.

8. Give the cluster computer object (e.g. SDSQL03) full control. If you want, you could also give administrators access so they can peek inside.


9. Finish the wizard and wait for the share to be created.

SQL 2014 File Share Witness Configuration

1. On a SQL server launch the Failover Cluster Manager.

2. Right click on the root cluster object (e.g. SDSQL03.contoso.local), select More Actions and then click Configure Cluster Quorum Settings.

3. Select Select the quorum witness.

4. Select Configure a file share witness.

5. Enter the file share path. Click through the remainder of the wizard and verify the FSW was successfully configured.

6. Verify Quorum configuration is now using a file share witness.



In this installment we configured the SQL cluster to use a file share witness. This is needed when you have an even number of servers in the SQL AAG. You can use either a NAS appliance, or another Windows member server. In the final two installments we will configure Kerberos and SSL. Check out Part 12 for Kerberos details.

SQL 2014 Always-on AG Pt. 10: AAG Setup

2014-10-12_18-34-23After a brief pause in this SQL 2014 AlwaysOn Availability Group series, we are resuming with the server configuration. In this installment we will configure the Windows firewall to allow the two SQL servers to communicate to each other, configure a shared folder for the SQL replication, and finally configure the AlwaysOn feature. Yippee!

Throughout this series I’ve maintained a standard naming convention for all of the computer objects needed for this installation. In the end we will end up with five, yes five, computer objects for this cluster. Two are the SQL server nodes themselves, one for cluster aware updating (-CAU), one for the AlwaysOn Availability group (-AG1), and one for the AlwaysOn Availability group listener (-AG1L). The listener is the name which you would use in ODBC connections to talk with the SQL AlwaysOn cluster and the clustered databases. Don’t use any of the other computer names, as they may appear to work but would result in problems when failing over between nodes.

You can host non-AAG enabled databases on these SQL servers, but would of course not be protected by AAG. In that case you would use SQL server node name in the ODBC connection name, not the AAG listener. Mix and match as you wish, depending on your business requirements.

Blog Series

SQL 2014 Always-on AG Pt. 1: Introduction
SQL 2014 Always-on AG Pt. 2: VM Deployment
SQL 2014 Always-on AG Pt. 3: Service Accounts
SQL 2014 Always-on AG Pt. 4: Node A Install
SQL 2014 Always-On AG Pt. 5: Unattended Node B
SQL 2014 Always-on AG Pt. 6: Cluster Configuration
SQL 2014 Always-on AG Pt. 7: TempDB
SQL 2014 Always-on AG Pt. 8: Max Mem & Email
SQL 2014 Always-on AG Pt. 9: SQL Maintenance
SQL 2014 Always-On AG Pt. 10: AAG Setup
SQL 2014 Always-On AG Pt. 11: File Share Witness

Windows Firewall

1. Download my basic SQL AlwaysOn firewall script from here. Open a Windows command prompt and type the following command:

SQL-Firewall-AG.cmd <AlwaysOn Partner IP>

2. Repeat the same process on the other SQL node, using the other node’s IP address.

Always-On Availability Group Preparation

1. On S drive (or another drive of your choosing) of your first SQL node create a folder called Replication. This is where SQL will stage the replication data.

2. Change the NTFS permissions on the folder and add the SQL service account with full permissions.


3. Right click on the Replication folder and select the Sharing tab. Click Share. Verify the Database engine account is listed as a read/write member. Click on Share.

4.  Launch SQL Manager, right click on Databases and select New Database. Create a new database called SeedDB and leave all other parameters at their default.

5. Expand the SQL Server Agent node in the left pane and locate the DatabaseBackup — User_Databases – Full job. Open the properties of the job, click on Steps in the left pane, and edit Step 1. Modify the path of the backups to S:\Backup.


6. If you want to use the other backup jobs, open them and modify the backup path as well.

7. Run the DatabaseBackup — User_Databases – Full job by right clicking on it and selecting Start job at step.

8. Open SQL Server Configuration Manager and locate the SQL Server service. Open the properties click on the AlwaysOn High Availability. Check the box to enable AlwaysOn.


9. Restart the SQL service.

Always-On Availability Group Configuration

1. Open Active Directory Users and Computers (ADUC) and create a new computer object using the cluster name with a suffix of –AG1 (e.g. SDSQL03-AG1). Change the description of the object to SQL Cluster <Cluster Name> Availability Group.

2. On the first SQL server cluster node in SQL Server Management Studio navigate to AlwayOn High Availability, right click and select New Availability Group Wizard.

3. For the availability group name use the format of: <Cluster Name>-AG1 (e.g. SDSQL03-AG1). This should be the same name as step 1 in this subsection.

4. On the Select Databases screen the SeedDB database should be listed, and it should meet the prerequisites. Check the box next to the database and click Next.

5. On the Specify Replicas screen add the second SQL server cluster node. Configure the automatic failover and synchronous commit options as shown below. Make the second node a readable secondary.


6. On the Listener tab create an availability group listener. For the DNS name, use the same name as the AG (e.g. SDSQL03-AG1) but with an “L” suffix added (e.g. SDSQL03-AG1L), using port 1433. Assign an IP address on the same subnet as the SQL server. If you don’t see the Add button under the network mode box expand the size of the window.

2014-10-12_17-56-247. Configure the data replication synchronization using the Replication file share on the first SQL node, as shown below.


8. Review the validation results and verify all are successes.


9. Proceed through the rest of the wizard and validate all steps are successful.

10. In SQL Server Management Studio right click on the AG and select Show Dashboard. Verify all objects are in a healthy state, as shown below.



Now that we have a SQL AlwaysOn Availability group up and running, there’s still a bit of configuration left to do. So future installments will cover configuring a file share witness, Kerberos, and SSL certificates. Yes, the all important SSL! Don’t quit now and think that you are done. While we are almost there, you will want to follow through the rest of the configuration. Check out Part 11 here.

Nutanix Announces All Flash Array

In very exciting news Nutanix announced the industry’s first hyperconverged all flash appliance, the NX-9240.  What makes this puppy special? Well it’s a 2U appliance which contains two compute nodes, each with 6 SSDs. The SSDs can be either 800GB or the 1.6TB models. This means up to 19.2TB of RAW flash in a 2U appliance. Fill half a rack with these monsters and you have nearly 200TB of high performance flash at your disposal.


Nutanix has always had great performance, due to our hybrid approach of combining SSDs with HDDs. This works great for a large majority of workloads including VDI, Exchange, SQL, Oracle and general purpose server VMs. With integrated ILM, hot data is moved up to SSDs while colder data is moved down to the HDDs. But sometimes there are exceptional workloads, such database OLTP which need consistent I/O latency and predictable performance across the entire working set that only flash can bring. These workloads will feel right at home on the NX-9000 series platform.


Why Nutanix?

One big advantage that Nutanix all flash has over other competing all flash arrays is our scale out controller architecture of one CVM per server node. As shown in my Running IOMeter on Nutanix blog post, we saw that I/O performance scales out as you add more controllers. Driving flash requires a lot of processing power, and dual controller arrays may start to peter out when pushing all flash performance. If you are doing array firmware upgrades or have a hardware malfunction and lose a controller your performance can be cut in half. With Nutanix, if you take a node down for maintenance then you only lose up to 1/n performance from your cluster. For example, in a 16 node cluster you would only suffer a 6% drop (assuming the cluster was being pushed to the maximum). In reality you probably wouldn’t even notice a node is down during normal workloads.

Another Nutanix advantage is the lack of rip and replace when you want to increase your storage performance. Instead of a fork lift controller upgrade to get better performance you merely add new nodes through the PRISM GUI in a few clicks. You’ve now just added many 10s of thousands of IOPS capacity to your cluster with minimal fuss. As new platforms come out, such as those based on Haswell, just add them to existing clusters and take advantage of the latest Intel CPU performance benefits.

Even More Models

In case you missed it, Nutanix also very recently announced the NX-8000 series, which has four SSDs and 20 HDDs. This is ideal for large scale Microsoft Exchange workloads (think thousands of mailboxes), or medium sized databases. From the NX-1000 series for small deployments through the NX-9240 for all flash, we have all of your hyperconverged bases covered.

Running IOmeter on Nutanix

Sometimes Nutanix customers or prospective customers will want to perform IOmeter testing to ensure the block is performing up to par. Or maybe they are upgrading to a new NOS release or doing a POC and want to ballpark performance. Performing benchmarks on a Hypeconvered platform like Nutanix is a bit different than the traditional 3-tier architecture. Nutanix has a concept of  CVMs (controller VMs) that run on each node (server). These CVMs service I/Os, and scale out 1:1 as you add more nodes. This gives rise to the great linear performance of the Nutanix platform. No more legacy two-controllers architecture that don’t scale up well.

In order to accurately assess the ballpark performance of a Nutanix cluster you will need to deploy at least one IOmeter VM per node, and configure that VM in a particular manner. Specifically, you will get the best I/O performance when spreading the workload across 8 virtual disk (VMDKs in the VMware world) and across multiple SCSI controllers. You also need to increase the number of vCPUs to 8, so Windows can better service all of the I/Os. And this might be news to some, but you don’t want to perform tests on NTFS formatted disks. What? Yes, you want IOMeter to use raw disks. Using NTFS formatted disks introduces an additional layer of caching and even lazy writes can be introduced into the picture.

Now that you understand a bit of the IOmeter configuration background, let’s deploy IOmeter and walk through all these steps.

Deploying IOMeter on Nutanix

1. Use the vSphere client and connect to the vCenter that is managing your Nutanix cluster.

2. Deploy your favorite Windows server VM template. This will be our IOMeter master, which we will clone later on.

3. While the VM is powered off, increase the number of vCPUs to 8.


4. Add 8 VMDKs to the VM, spreading them evenly across the four SCSI controllers (e.g. 0:x, 1:x, 2:x, 3:x.). I would suggest 8x 20GB thinly provisioned disks.

5. Change the new SCSI controllers (controllers 1 to 3) to Paravirtual (PVSCSI). You can use LSI Logic SAS as well, but the PVSCSI controller is more optimized for high IOPS and lower CPU utilization. Leave controller 0 as LSI Logic SAS so Windows can boot.

6. Install IOMeter. You can download it from here. Download the IOmeter configuration from here. Unzip the configuration files. Power down the master VM.

7. Clone the IOMeter master VM so that you have one VM per Nutanix node. Disable DRS on the cluster (if enabled) and vMotion the new VMs so that you have one IOmeter VM per node. Power on the IOmeter VMs.

8. Launch IOmeter and load the 32k-sequential-write ICF file via the yellow folder icon in the upper left. Ignore any errors.

9. Click on the Computer name in the left pane then shift click and select all disks. Change *each* disk’s maximum disk size to 8000000.


10. Click on the Green flag to start the test. Wait a few minutes for the green flag to become unghosted. This will write data to all of the drives. Exit IOmeter.

11. Locate the benchmark that you want to run from the IO configuration files that you downloaded. In my example I’ll use 4k-random-read-2-outstanding-IO. Double click the configuration file to launch IOmeter.

12. You should see 1 worker, and 8 VMware disks in right pane. Click on the Disk icon to add 7 more workers, for a total of 8. For each worker select the corresponding VMware virtual disk, as shown below. Worker 2 uses VMware disk 2, etc. Change the maximum disk size to 8000000.

2014-10-09_13-10-5713. Click on the Access Specifications tab and select the 4K random read and add it. Click on the green flag to start the test. Change the update frequency from the drop down menu to a reasonable value like 5. Wait a few minutes for the performance to stabilize. Perform the same benchmark in the other IOMeter VMs and wait for the IOPS to peak and level out. You are now testing the total cluster IOPS capacity. For example, if you have three nodes and each IOMeter instance is pushing 20K IOPS, then the cluster can sustain 60K IOPS.

Outstanding I/Os

One parameter within IOMeter that can have a big effect on the total IOPs is the number of outstanding I/Os per target. This parameter adjusts how many I/Os are “in flight” to the storage controller. As you increase the number of outstanding I/Os you increase storage latency, but it can also increase performance. There’s a balance between the number of outstanding I/Os and latency. For example, during a test run if I set the outstanding I/Os to 4, latency during 100% read tests was right at 1ms with 33K IOPS for a single node. Jacking it up to 16 outstanding IOs increase IOPS to 58K but latency went up to 2.1ms. Using 64 outstanding IOs latency jumped up to 7.6ms and IOPS peaked at 67K. For this particular test 16 outstanding I/Os gave a good balance between latency and IOPS.



As you can see from this setup, the Nutanix performance scales linearly as you add more nodes. Each CVM processes IOs and as you add more nodes, you scale the CVMs and the total cluster performance. You can also see the scale-out IOPS performance within the VM by starting with one worker, benchmarking, then incrementally add workers. As you will see once you get to about 8 VMDKs and workers the performance tops out. You can also affect IOPS performance by changing the number of outstanding IOs. Latency increases as you increase the number of in flight IOs.

This design principal, of distributing workloads across multiple VMDKs within a VM carries across through our Exchange, SQL, Oracle and other best practices that Nutanix has published. This enables those applications to take full advantage of the web-scale architecture and linear performance that Nutanix brings to the table. Go hyperconvergence!

Sample VCDX-DCV Architecture Outline

This post covers my approach to writing my VCDX-DCV Architecture Guide. I’ve been debating in my mind for a while whether I should write this post or not. I hesitated for a few reasons. First, I’m just a regular guy that happened to jump through the VCDX hoops and have no “insider” information on how they score. Those that do know the scoring rubric can’t disclose it anyway. Second, there are 1000 different ways to write your VCDX-DCV architecture document. Third, there’s no “magic template” or “sure fire” outline that ensures your design gets accepted. Do not view this post as shortcut or cheat sheet.

What matters is your content, how it aligns to the VCDX blueprint, and that you convey expert level knowledge to the reader. It’s NOT about speeds and feeds, but rather the full traceability of customer requirements, constraints,  assumptions and risks throughout your design. Who cares if you’ve thrown every VMware product and feature at a solution if you haven’t met the business requirements? #Fail

So why did I publish this article? I know when I started the VCDX process it was a bit daunting to read the DCV blueprint and try to come up with an architecture guide that hit all the areas in a logical manner. I’ve heard from other candidates they experienced the same “VCDX writer’s block.” In fact several of us have scrapped our first attempts, and started over. Bottom line is you need to do what feels right to YOU, and what works for YOUR design while covering all the blueprint areas. You may not like my methodology or outline, which is perfectly fine and a valid way to feel.

I’ve also heard comments from VMware customers (like myself when I went through the process) that think since they aren’t a partner and don’t have access to the VMware SET templates that they are at a disadvantage. That’s not true,  IMHO. Yes the VMware SET docs are structured and may help you, but they aren’t directly aligned to the VCDX blueprint and need augmentation.

With all those caveats, I wanted to share my DCV architecture guide outline. Maybe it will help someone with writer’s block, or enable you to see some the areas that a VCDX design could cover. Your design may need additional areas, or less coverage. This is certainly not all inclusive, and it’s guaranteed your outline will be different. It is your responsibility to ensure your documents cover all blueprint areas, makes sense for your design, and something you feel comfortable with. Own your documentation.

Before I go any further, let me state that how I chose to incorporate the specific VCDX bootcamp book recommendations is somewhat unique to my style. Of the submissions I’ve seen none did it exactly this way, which proves that there is no “magic” template or style for VCDX submissions. I just felt it gave a better overall flow to the document.

You will see some common sub-sections in all design areas (e.g. cluster, storage, compute, etc.). For example, in most areas I had specific conceptual, logical and physical sections. This helped me show the traceability of customer input through the entire design process. Each major section also concludes with a Design Justification which is a summary of how I met the customer requirements and sites all of the applicable requirements, assumptions, constraints, and risks.

At the end of the Design Justification section I had two tables to help distill down the critical information. First, I had a summary table, shown below. All of the design quality items (e.g. C02) were referenced elsewhere in that section as applicable. Possibly overkill, but I liked the compact summary.


The second table was that of the applicable design decisions, each with the decision, impact, decision risks (after all, nearly every decision has a risk), and risk mitigation. A sample design decision is below.


WordPress was not cooperating with me for a clean outline format, so I’ve inserted a series of screen captures to maintain formatting.

Sample VCDX-DCV Architecture Outline


2014-10-06_16-50-08  2014-10-06_16-52-34