It’s had several titles, but currently its “How to be a more successful architect” and I originally created it as an overview of my Microsoft Certified Architect journey, although I’ve taken it on a different course since then.

Hopefully of interest to everyone

The session is hopefully interesting to everyone in the IT industry, not just a specific type of SQL Server professional.  It’s foundation is the idea that regardless of your job title, you’re probably informally, or formally, performing the role of an architect at some point in your working day.

It could be you have to clarify someone’s requirements so you can understand how whatever you create for them demonstrates its value the most.  Or, you could you to justify to someone, or yourself, why the solution design you’ve come up with is the right design as you’ve taken a structured path to designing it.

Presentation slides

The slides from my most recent delivery of the session at SQLSaturday #194 in Exeter are available here.   Like all architectures its evolving, so I’m sure there’ll be newer versions in the future.

Presentation links

Within the presentation, and during my delivery of it, I mention numerous online resources that I recommend reviewing as you develop your IT architect skills.

Architect Certifications

Global IT Architect Association’s CITA-P here.

The Open Group’s Certified Architect here.

Microsoft’s Certified Architect here.

Architect Resources

Microsoft’s The Architect Journal here.

Perspectives Based Architecture for requirements gathering here.

ATAM for prioritising requirements here.

Waterfall vs. Agile methodologies here.

Influencing Without Authority book here.

MSDN Patterns and Practices library here.

Microsoft Application Architecture Guide here.

Solution Lifecycle Management

ITIL Application Life Cycle Process here.

Hopefully something somewhere in this post or the web links has caught your imagination, in which case, please share just what in the comments section.

I’ve saved it as a PDF file as WordPress doesn’t like .sql or .txt files, but you can still copy and paste from it.

The file is here.

Virtualisation of yesterday

In the last few years, the capabilities of virtualisation have increased dramatically.  It was only a few years ago that virtual servers were limited to 4 vCPUs, had only average performance storage, and never had enough memory.  That explains why for a long time it was only ever our “small footprint” servers that we virtualised.

Virtualisation in 2013

Today, the enterprise virtualisation platform vendors deliver capabilities that we only dreamt of a few years ago; 32 vCPUs, 1TB of memory, and high availability by default, for every virtual server.

That’s a lot of engine power we can put behind our virtual servers, allowing us to virtualise hopefully most of our data centres if we choose to these days.

Watch for the subtle limitations

However, despite these great headline capabilities, there are still inherent capability limits that the small print carries.  I found one of these during a cluster build recently and that prompted me to write this article.

Our project was using VMware 5 to host some fairly large nodes of a SQL Server failover cluster instance.  Each virtual server had 16 vCPUs and 128GB of memory, something that VMware can adequately provide, but it was the storage configuration where we hit a limitation by surprise.

The SQL Server cluster design we initially created required upwards of 60 mount points to be presented to each of the cluster nodes, however we then discovered that VMware has a limit of 45 clustered/shared SAN volumes (raw device mappings) being presented to a virtual server.

What it quickly demonstrates is that while our focus was initially on whether or not the virtual server would run fast enough, the limitation that actually mattered to us was actually related to storage provisioning.

Published configuration limitations

The links below provide details from VMware and Microsoft about the complete limitations of their virtualisation software, and are worth reading if you’re planning to deploy virtual servers with more than the default resources.

VMware 5 Configuration Maximums

Windows Server 2012 Hyper-V Scalability

I’ve been using Windows Server 2012 for a while now, not just because it’s the newest and shiniest Microsoft operating system, but also because some of its new features are useful to me as both an architect and implementer of SQL Server platforms.

The most significant of these are in the failover clustering feature where some of SQL Server 2012’s requirements for Availability Groups can now be done in the GUI rather than the deprecated cluster.exe command line.  For example, modifying quorum voting, dynamic quorum models, and handling asymmetric storage.

Subtle Issues

Previously, I’d implemented clustered instances of SQL Server 2012 on Windows Server 2012 without any problems using the native RTM releases of both, in a production and development environments and all went well.  However, this week I began a new clustered platform installation and came across some real-world issues that I hadn’t seen before and thought I’d share.

Installing the .net framework 3.5

SQL Server 2012 requires that the .net framework 3.5 is installed prior to installation, however in Windows Server 2012 the install files for it are not shipped as part of the operating system.  Instead when you go to add the Windows feature, it will ask for the path to the source files on the Windows installation DVD to perform what it calls a side-by-side installation.  Sometimes, through luck, you’ll put the exact path details in correctly first time and it’ll work; the rest of the time you’ll find what is reported to be a bug where it’ll never accept anything you enter.

To get around this, you can use one line of PowerShell instead:

Install-WindowsFeature NET-Framework-Core –Source X:\Sources\sxs

Problem solved!

.net framework hotfix KB2750149

This hotfix is one of many that get installed automatically through the Windows Update service, however it has a side effect of breaking the Windows Failover Cluster manager, specifically when you click on the Roles branch (used to be known as Services).  You’ll receive a big error message screen starting with the text:

“A weak event was created and it lives on the wrong object,
there is a very high chance this will fail, please review 
and make changes on your code to prevent this issue”.

This is a known issue that Microsoft has acknowledged and is investigating.  In the meantime, the only workaround is to un-installed the hotfix KB2750149.

DNS Registration Failure

Another random error that I received was the DNS record for the cluster failing to get registered correctly during the creation of the cluster.  Windows Server 2012’s management tools now integrate with the other nodes in a cluster a lot more closely than they used to so even trying to make disk storage changes caused random server connection errors.  The main error I got when trying to bring the cluster itself online was the following:

Cluster network name resource ‘Cluster Name’ failed registration 
of one or more associated DNS name(s) for the following reason:
The handle is invalid.

I’m still not sure of the cause, apparently it could have been related to Active Directory permissions, but the solution that worked for me was to Repair the cluster’s virtual network name with the following two steps:

Firstly, taking the virtual network name offline.

Secondly, repairing the virtual network name.

Finally, bring the virtual network name back online.

I’ve no idea if this will fix similar issues you may get, but knowing the option to repair the cluster’s DNS record is useful to know.

Summary – A Powerful Combination of Platforms

Implementing a clustered instance of SQL Server 2012 on Windows Server 2012 is still a very capable and impressive combination, and something I’d still recommend be the default option these days.

This blog post covers what’s true about all failover cluster installations and that’s that they touch every part of the server’s infrastructure: storage, networking, operating system, active directory, as well as SQL Server.  Consequently, minor tweaks to any of these areas can leave the SQL Server professional with a new area of the infrastructure to master.

My presentation’s purpose, and demos, were to show how this type of database engine index, that’s new in SQL Server 2012, significantly improves the performance of data warehouse style queries. 

Columnstore indexes, and their complimenting Batch-Mode Query Processing feature, are used transparently by SQL Server once they’re created so the benefit for SQL Server professional is that that there’s no need to change existing data warehouse applications or queries to benefit from these performance improvements. 

Slides

The presentation’s slides are available here.

Session Questions

Below are some of the answers to questions that I remember giving during the session:

• Even though the levels of internal compression of data within a Columnstore index are affected by the cardinality of the index’s data, the performance benefits of Columnstore indexes are so great compared to a non-clustered index that your data’s cardinality is not likely to influence whether you use Columnstore indexes or not;
  
• You can use Columnstore indexes with AlwaysOn Availability Groups;
  
• The SQL Server query optimiser will automatically consider using a Columnstore index and Batch-Mode Query Processing itself, you don’t have to re-write queries to use start using them;
  
• Batch-Mode Query Processing only works with Columnstore indexes;
  
• The column order, and number of Columnstore index keys, do not affect the performance of a query that using the Columnstore index.

Microsoft Certified Architect : SQL Server – Giving back

Preparing for my MCA Board was the hardest, yet in hindsight the most rewarding and interesting, thing I’ve ever done.  The subjects it covers still interest me to the extent that I’m now contributing to the MCA programme itself, allowing the next people through the certification’s process to benefit from someone else’s experiences and inputs.

For those of you with the pre-requisite SQL Server MCM certification and who are interested in taking the next level of certification, there are more details here:
http://www.microsoft.com/learning/en/us/mca-certification.aspx

Additionally, free feel to email me if you have any questions about my experience of the programme gavin@coeo.com

Accelerating Data Warehouses with Columnstore Indexes – SQL Santa – 14th December

The fact that Columnstore indexes work in a totally different way to how we’re used to SQL Server working makes me curious about them.  I’m sharing the learnings from my curiousness, and outcome from a customer proof of concept test, in a hour’s presentation at a London event in mid-December.  SQLSanta is a day long even at Microsoft’s London venue, and has a range of Microsoft Product Marketing, and industry professionals, presenting about contemporary SQL Server industry topics.

Although the event is now fully booked, you can find more details about the sessions here http://www.sqlserverfaq.com/events/452/SQL-Santa-2012.aspx as I suspect some will be presented at future community events, I’ve certainly submitted mine for some already.

Mission Critical Data Platforms on Windows Server 2012 Pre-con – SQL Saturday 194 – Exeter – 8th March

Christian and I are hosting this one day workshop with the agenda of un-leashing the new capabilities of Windows Server 2012 to make SQL Server deployments even more highly available in even more scenarios.  Rather than reviewing the introductory level aspects of Failover Clustering or Availability Groups, the day looks at how previous platform limitations can now be overcome using native features in the latest Windows operating system.  These include subjects such as the merging of high-availability and disaster recovery design patterns, the use of new storage formats such as SMB file shares, and using Hyper-V virtual machines in HA and DR configurations.

You can read more about our pre-con, and book, here http://sqlsouthwest.co.uk/SQLSaturday_precon.htm

Coeo were lucky to have four of us present a staggering five sessions on the day; so thank you to the SQLSaturday team for selecting our sessions, and to those who chose to attend them.

I’ve put a link to the presentation slides for all of our sessions below:

• I want to be a better architect – Gavin Payne
  Slides here

• NUMA internals of SQL Server 2012 – Gavin Payne
  Slides here

• Deploying a highly available BI environment – Chris Testa-O’Neill
  Slides here

• SQL Server 2012 certification – Chris Testa-O’Neill and Pete Harris
  Slides here

• Troubleshooting SQL Server with PowerShell – James Boother
  Slides here

This is part one of three blog articles about my initial experiences of them, but more interestingly, a quick look under the hood at how they work.

What are columnstore indexes?
In summary, the Columnstore indexes feature is both a type of non-clustered index AND a new query processing mode built into the SQL Server 2012 database engine. Their aim is to massively increase (approx. 10x) the performance of the types of queries used within a data warehouse, or in simpler terms – querying very large tables as quickly as possible.

There are many explanations of what columnstore indexes actually are, how they’re different to row-based indexes, and how SQL Server’s implementation works so I’ll point you to perhaps the best links for these topics:

• How do Column Stores work? – Thomas Kejser here
• Inside SQL Server’s implementation – Conor Cunningham here

How would I use them?
Hopefully, the two links above will give you an idea as to what the feature was designed to do and the circumstances in which you might use them. The next section focuses on how you would use them and for this series of blog articles I’m going to use the two simple tables shown below:

Having created and populated the tables, I used random number based values to create 35 million rows in the fact table for 10 products, I then created regular clustered indexes on them:

create table tblFact (rowID bigint identity(1,1),
custID int, prodID int, Sale money); 
create table DimProd (prodID int,prodName varchar(200)); 
create unique clustered index idx_cls_tblFact on tblFact(rowID);
create unique clustered index idx_cls_Prod on DimProd(prodID);
insert into DimProd (prodID, prodName) 
values (1, 'Car'), (2, 'Van'), (3, 'Truck'), 
(4, 'Taxi'), (5, 'Horse'), (6, 'Helicopter'), 
(7, 'BizJet'),(8, 'JumboJet'),(9, 'Ferry'), 
(10, 'Hovercraft')
insert into tblFact 
(custID, prodID, Sale) 
(select convert(int,rand()*10), convert(int,rand()*100),
convert(money,rand()*10000)) 
go 1000000 

Followed by:

insert into tblFact (custID, prodID, Sale) 
(select top 500000 custID, prodID, Sale from tblFact) 
go 58

This will give you 30 million rows for about 1GB of disk space.

I then ran a simple query against the table that’s designed to simulate a simple star schema and the type of query a data warehouse might use:

select DimProd.prodName, 
sum(tblFact.Sale) 
from tblFact, DimProd 
where tblFact.prodID = DimProd.prodID 
group by DimProd.prodName 

I could have added supporting non-clustered indexes, but traditional index design isn’t the point of this article, but with the 35 million rows I had in my fact table the query takes 4 seconds to run.

Adding a columnstore index
The next step is to actually add a columnstore index, this is quite a big step as:

• You can only have 1 columnstore index per table
• Creating the columnstore index makes the table read-only

The following command adds a columnstore index:

create columnstore index idx_cs_tblFact on tblFact(custID, prodID, Sale)

You can tell from the time it takes to create, on my SSD drive about 50 seconds, that there’s a lot of work going on to create it; for comparison, creating the equivalent regular non-clustered index takes about half the time.

If I then execute the query used above again, the query optimizer will use the columnstore index AND the new batch query processing mode that Conor mentioned, and the query will now execute in less than a second.

Although the performance improvements in our example might not seem great, 4 seconds down to 1 second, in further tests I’ve increased the fact table from 30 million to 90 million rows, yet didn’t seen a 3x increase in query runtime.

Sys.indexes
Finally, for this article, you can begin to see how columnstore indexes are both different and similar to normal indexes by looking at their properties in SSMS and sys.indexes with this query:

select * from sys.indexes where type = 6

Books OnLine lists Index type 6 books as a “Nonclustered columnstore index”, while Index type 5 interestingly is “Clustered xVelocity memory optimized columnstore index (Reserved for future use.)” This suggests there might be quite a lot more than just updatable columnstore indexes on the roadmap.

Of course, this is all just my first look at columnstore indexes, if you know something I’ve not mentioned, please feel free to comment on the post!

Earlier this week, I had a client ask about the best way to perform:

The last point might sound un-necessary because you’re restoring a differential backup, but the restore script originally being used meant SQL Server still wanted every transaction log since the full backup restored.  This article explains the background to the situation, the successful restore commands, and identifies what was causing every transaction log to need to be restored.

For this article, let’s imagine we have a database of the following configuration:

And, let’s imagine it has the following backup schedule:

Then, let’s assume we want to restore fgTwo to the point in time labelled above in order to recover data from a table it stores.  Performing a partial database restore which will be required also requires the primary filegroup to be restored so SQL Server will automatically restore it making the future database look like the following:

You’d expect the path to restore fgTwo to look like the path on the left of the diagram below, but for some reason the client was being forced to perform the restore steps on the right, despite their T-SQL restore commands appearing to follow the syntax in Books Online.

To show how to restore the filegroup using the path on the left, and highlight what was causing the path on the right to be required, I’ll use a series of T-SQL restore commands.

This first command restores the parts of the database we’re interested in from the full backup file, labelled “A” in our timeline, and the two most important parameters shown in green are what turns a complete database restore into a partial filegroup restore.

restore
database partial2
filegroup=’primary’, filegroup=’fgTwo’
from disk = ‘c:\parttest\partial1_FULL_A.bak’
with norecovery, partial, replace,
move ‘primary’ to ‘c:\parttest\priamry.mdf’,
move ‘primary_log’ to ‘c:\parttest\primary_log.ldf’,
move ‘fTwo’ to ‘c:\parttest\fTwo.ndf’

The next command restores from the differential backup, labelled “B” in our timeline, and means we don’t need to restore transaction logs A1 and A2.  In the client’s real-world scenario they actually had 120+ transaction logs between the full and differential backups, hence the desire not to have to restore each of them.

It was this command that was the cause of them having to restore every transaction log since the full backup, the crossed out parameters in purple were what was used and causing it to happen.  Telling SQL Server again which filegroups to restore was forcing it to need to restore the entire log chain since the full backup.

restore
database partial2
filegroup=’fgTwo’
from disk = ‘c:\parttest\partial1_DIFF_B.bak’
with norecovery,
move ‘primary’ to ‘c:\parttest\priamry.mdf’,
move ‘primary_log’ to ‘c:\parttest\primary_log.ldf’,
move ‘fTwo’ to ‘c:\parttest\fTwo.ndf’

Finally, with that step performed, you can then restore the subsequent transaction logs from after the differential backup with a relevant STOPAT parameter:

restore
log partial2
from disk = ‘c:\parttest\partial1_Log_B1.trn’
with norecovery, stopat=’2012-05-29 15:13:12.750′

restore
log partial2
from disk = ‘c:\parttest\partial1_Log_B2.trn’
with norecovery, stopat=’2012-05-29 15:13:12.750′

restore database partial2 with recovery

At this point, the database is recovered with just the primary and fgTwo filegroups online, you can look in sys.master_files to see the state of all of the database’s data files and which are currently online.

In summary, this article showed how to perform a partial database restore, and how you can easily have to restore more than you were expecting to due to a simple “over-clarification” in a restore command.

I have a complete demo script of a more thorough test available for download from here.

The Read Committed Snapshot database option in SQL Server, known perhaps more accurately as Read Committed Snapshot Isolation or RCSI, can be enabled to help readers from blocking writers and writers from blocking readers.  However, enabling it can cause two issues with the tempdb database which are often overlooked.

One can slow down queries, the other can cause queries to fail.

Overview of RCSI

Enabling the option changes the behaviour of the default SQL Server isolation level, read committed.  By default, this isolation level means that if one session is updating a range of rows then another session cannot execute a select query against the same set of rows; the shared locks the select query needs are incompatible with the eXclusive or Update locks that the update query will be using; the session executing the select query ends up being blocked until the transaction involving the update query has committed.  That’s a very high level overview of what happens so those wanting to know more or become an MCM should watch this video here.

Configuring a database to use the Read Committed Snapshot option aims to stop this happening by using Row Versioning.  Essentially, before a row is locked in preparation for changes a copy of it is put in a special part of the tempdb database, known as the version store.  When a select query then wants to read that row it uses the copy in the version store rather than waiting for the actual row to have its locks released.  Again, I recommend watching the video linked to above to see just what happens as I’m just giving a high level overview of the feature before I discuss the primary points in this article.

Tempdb Performance

One of the first things people mention when you suggest enabling the Read Committed Snapshot option is that it will put pressure on the tempdb database which stores the version store.

That’s correct it will, but a properly configured tempdb database should never be a bottleneck in your environment.  You can read how super-tuning expert Thomas Kejser put the world’s biggest database workloads through SQL Server and never had to worry about tempdb performance here.

There are two considerations I recommend for configuring your tempdb database in order to ensure good performance:

• The number of tempdb data files – in the absence of your own monitoring for PFS, GAM or SGAM contention within tempdb I hear people at Microsoft these days recommending having 8 equally sized data files which I’d agree with.  They don’t need to each be massive, their total size could still be 8MB(!), but having more increases the number of internal system objects within the database so reduces the changes a single system objects becoming contended.  (The rule in SQL Server generally tends to be that if you have any kind of resource contention add more of that resource; whether it be IOPS, locks or PFS pages etc.).  Having too many data files is much less likely to cause you problems than having too few, just be sensible about how many you have.

• The type of physical storage for the tempdb data files – as always, configure the storage you use for tempdb like you would for your database files.  Fast, reliable and striped arrays, and while Thomas will tell you that using dedicated disks isn’t important for him, others will tell you its crucial.  The only additional comment I have is that some people deploy NAND storage specifically for tempdb, in fact SQL Server 2012 had some changes to support this with failover clusters.

Crucially though, having a poor performing tempdb database won’t break a user’s queries, they may just run slower than they could do, especially if you’re using the read committed snapshot option.

Tempdb Sizing

How much space your tempdb database needs depends on how you use SQL Server, some people need very little space whiles others need lots.  What’s relevant to this article is that the read committed snapshot option doesn’t just like tempdb space, it needs it.

We mentioned how rows get written to the version store in tempdb as they get locked in the source user database, but what happens if there’s no free space in tempdb when that process goes to happen?  I did a test to find out, you can email me for my demo script if you want to try it for yourself.

In summary, when tempdb is full update queries will still execute successfully but select queries which expect to find the rows they need in the version store will fail when they can’t find them there.  You’re likely to see this message:

“Msg 3958, Level 16, State 1, Line 1

Transaction aborted when accessing versioned row in table ‘dbo.t1′ in database ‘s1′. Requested versioned row was not found. Your tempdb is probably out of space. Please refer to BOL on how to configure tempdb for versioning.”

So how do you stop that from happening? 

The safety option I suggest is to place tempdb on a drive with lots of free space although its acceptable to configure tempdb to start with small initial data file sizes.  However, make sure you configure the date file auto-grow settings so that when tempdb runs out of free space SQL Server will automatically allocate it some more.  There are methodologies you can use for setting a good initial size for tempdb but for now enabling the autogrow option is a good failsafe.  Then, after a couple of months see what size the data files have grown to and consider setting that to be their initial sizes. 

General Recommendations, Test

Those are very general recommendations, along with setting the data file auto grow size to 500MB, but in the absence of specific testing or advice for your environment they might be something you want to try using in a test environment to help you learn more about tempdb behaviour in your environment.

Before we start, I should point out a couple of things: this article has been reviewed by the former MCM program manager Robert Davies so don’t think it gives away secret information about how to prepare – none of that is covered by any NDA anyway, and also realise that the MCM is a very high level of certification, its not for everyone.  Read the experience the program assumes you have and ask yourself if that’s like your background.

How I Prepared

There were three specific steps I took for in my preparation and luckily I was able to make most of them part of my day to day role at Coeo. They were:

1. Real-world experience

2. Reading materials

3. MCM readiness videos

Equally as importantly, you need to find a way to make the intense preparation more exciting to you than just sitting down in an evening and reading a book. If you take that approach you’ll run out of mental energy and begin cutting corners in your preparation which will reduce your knowledge and familiarity with features. I introduce the word familiarity there for the first time, that’s an important word for the MCM’s requirements of you.

Familiarity

This certification isn’t about knowing every syntax, command or trace flag ever published, it’s about knowing what features SQL Server has, how they work, how they work with each other and their limitations. No book will teach you all of that, only having used them a lot will you become familiar with them. For example, if the lab exam asks you to meet requirement X then you should immediately know that feature Y is the answer, then is not the time to be teaching yourself the difference between two similar features. Of course, if you also know there’s something which can stop a straightforward implementation of feature Y then your familiarity with it needs to be able to tell you in advance what that issue will be – and how to fix it.

Real-world Experience

Nothing can prepare you better for the MCM exams than real-world experience of having used SQL Server, that’s where your familiarity comes from. Of course, that probably doesn’t mean managing your company’s HR or Intranet databases as the MCM is focused at enterprise environments with requirements and issues which require MCM level experts to resolve them. I was lucky that my day job put me in those enterprise situations almost every day and while you may not think you have that luxury you’d probably be surprised.

My advice is to do the following regardless of what your involvement with SQL Server is:

• become familiar with the MCM’s exam objectives and your understanding of what they require you to know (the readiness videos may help if you’re not sure) in terms of features and best practices, they are available here
• generalise each SQL Server request or requirement you get given in your current role, remove what makes it specific to your environment and most importantly, the limitations your current environment might put on how you would deliver it
• then, in your own time, consider how you might meet that requirement in a way that best shows your understanding of the MCM’s exam objectives. For example, using peer-to-peer replication might not be possible in your environment but it could be the most appropriate way an MCM would implement a solution in an environment with no legacy restrictions
• with your chosen approach planned, implement it on some virtual machines. You don’t need to worry about having complicated database schemas or large volumes of data, nor do your virtual machines have to be fast. You should be able to create the databases and data for your tests in 60 seconds, keep them simple so you can focus on getting the feature to work so you can learn about it. Spending a whole evening just trying to replicate all of the tables in your intranet database is an un-necessary way to waste time
• with the feature installed, remove it and repeat the setup, determine if its best to use the GUI or T-SQL setup steps in case there are options you can’t chose through the GUI, then find out how you troubleshoot and diagnose that feature, are there DMVs or log files etc.

That may seem like a lot of effort but that’s the level of preparation I put into learning about SQL Server for my exams. Of course, there will be some features you might know more than enough about already, great, tick those off and move onto those you don’t know so well.

Reading Materials

The MCM web site gives you a very thorough reading list, here, which covers almost every part of SQL Server in almost every depth you can imagine. How much of that reading list you can ever read is a very personal decision based on time, ability and ultimately your interest in some of the subjects. What I will say is that every little thing you know about SQL Server when you go into the exams will boost your confidence, especially if you find you know more than is required. Some of the books can be expensive and difficult to get hold of so don’t feel they’re compulsory reading. Instead, Books Online and blogs can give you the equivalent knowledge but you will have to spend more time searching for it. I’m not going to recommend any specific blogs, only that the types of blog you read should typically be authored by people who teach pre-cons or deliver level 500 training at conferences.

When you do read a book, make sure you have a good note taking method. There’s no point in reading a book, getting to the end and forgetting what you read 2 days ago at the beginning. I used OneNote but there are many variants of the productivity tool around.

MCM Readiness Videos

I found the videos, available here, very useful but not in the way you’re probably imagining.
I watched them at anywhere between 1.2 and 1.6x their normal speed depending on who was speaking. Why? Because I used them for revision and checking I knew what was being shown, not because I was using them as my only learning opportunity or because I wanted to emulate the classroom experience. If I didn’t know something that was being shown I either stopped the video and researched that subject, or watched that part at normal speed. However, you cannot pass the MCM exams just by watching 40 hours of videos! They are readiness videos, not a complete training course.

That was “all” I did for my preparation. Notice I didn’t call it revision, as I was honest and had to learn new things for the MCM exams, you will need to as well. I also didn’t do any MCM training courses, practice labs, webinars or use any discussion groups etc. However, watching the #sqlmcm hash tag on Twitter though may be of interest.

Knowing When You’re Ready

If someone asked if you’re ready to take the exams you’d always say you need another month or 2 months. In fact, the more you prepare the more you realise you don’t know so the more time you feel you need, that’s just because you’re aiming for a perfection that’ll never exist. Below are three recommendations my mentor, an MCM, gave me to help determine if I was ready and prepared:

• if your preparation plan is going to be more than 6 months you’re not ready to even begin. You can’t sustain the preparation intensity for any longer than that
• you should be able to confidently talk about a feature of SQL Server to someone else who works with the product for 90 seconds and not end wondering if the other person believed what you were saying because everything you said should have been true and something you’d learnt
• you should be able to take all four of the pre-req exams needed for the MCM the day before you take your first MCM exam and not think anything of it. If passing an MCITPro exam is still a big hurdle for you, you’ll need to re-consider your approach to the MCM exams

So, that’s my advice, it’s not right, it’s not wrong, it’s just my approach that I used for taking, and passing first time, the MCM exams. Good luck with yours!

Recently, in the SQL Server space there have been some technologies become available that showed how existing performance bottlenecks were removed by solving the problem in a completely new way to the previous successful but relatively temporary fixes.  Two examples we’ll consider are solid-state storage and Columnstore indexes.

10x faster can’t always come from 10x the power

Tonight I read the keynote presentation that Conor Cunningham gave at SQLBits recently about ColumnStore indexes (available here) but rather than read about how ColumnStore indexes work I read it from an architectural decision making perspective. 

Midway through the presentation is the line “Making something 10x faster is HARD”, followed by a slide questioning how could you make a query run 10x faster.  The answer wasn’t optimising the existing way of doing things, it was to implement a completely new way of working, in this case a new query optimiser mode and index format.  The existing options just couldn’t be tuned anymore to meet the requirement.

Game changing storage

The other example I quoted was the introduction of solid state storage in database servers.  Making storage fast was always a goal of infrastructure teams, they used different RAID levels, different caches and different numbers of disks until the industry realised that 5 and 10% gains weren’t always worth the effort or cost of implementing.  Instead, what we wanted was something that was 10000% faster and the answer wasn’t more disks, it was a totally different approach to persistently storing data, in this case using memory chips instead of spinning platters.

Conclusion

You’re probably never going to be asked to re-write a query optimiser or create a new type of storage hardware, however you maybe asked to keep making something go faster or scale further after having just done all you can do to make it faster.  It could be making reports run quicker or supporting more concurrent users.  My conclusion is that you shouldn’t just think about how to solve today’s problem using today’s options, think about a medium term option that may mean doing something radical but it won’t be a shock to you when you realise you need to do it.

These are just my views and the advice I give people who ask me similar questions, they may differ from yours, they may not.

Which level of certification should you aim for?

For the SQL Server database engine, there are four levels of certification, the BI track will soon have a similar number, and at the same time all of the certification names will change but for now I’m going to use the SQL Server 2008 era titles and list them in increasing order of “difficulty”.  They are: Technical Specialist (MCTS), IT Professional (MCITPro), Certified Master (MCM) and Certified Architect (MCA). 

Which level is right for you depends on your personal goals.  If you have only just started using SQL Server or want to get confidence through external validation that what you know is accurate then the Technical Specialist is a good first goal.  In order to prepare for this you’ll probably need to broaden your knowledge to include some SQL Server features you haven’t used before, preparing you for when you do use them.  This is an important step as it shows how certification will drive your professional development as well as deliver your certification goals. 

If you take a Technical Specialist exam and get a good pass mark then you’re probably already on the way to achieving the IT Professional certification.  I’d recommend this level for anyone who uses SQL Server all day everyday and has done so for 1-2 years.  Passing this level will not only validate your knowledge, but also give you confidence to talk about SQL Server .  If you’re in a role that deals primarily with SQL Server then you’re going to have people asking you questions about it, you’re on the road to becoming your department’s subject matter expert. 

When it comes to the Microsoft Certified Master there’s only one person who can tell you if that level of certification is a suitable goal for you, and that’s you.  Actually, your manager might have a view as well especially if you work for a company that’s passionate about SQL Server.  If you look at the requirements that microsoft.com gives for Microsoft’s highest level of technical certification and think “I can do that” or “I want to pass that” then you’ve just set your next goal.  It’s true that there’s probably a lot more thought that goes into aiming for this difficult and expensive certification but ultimately you have to have that “I want to pass” motivation in you otherwise the 6+ months of preparation will just be something you start tomorrow.  If you’re serious about wanting to pass the MCM then it has to become part of your everyday life.  I’ve met a few people who are “18 months into a 6 month revision plan”, they’ve just got side-tracked apparently.

The Microsoft Certified Architect is Microsoft’s highest level of certification for SQL Server and one which often gets overlooked.  Why?  I put it down to it not being a technical certification.  To me however its equally as important as any technical credentials you might have if you’re positioning yourself as one of the best in the world at deploying or architecting SQL Server solutions.  Being a Microsoft Certified Master is a pre-requisite for taking the MCA program which doesn’t have an exam.  Instead you have to submit a 40 page portfolio and attend a 6 hour Board in Seattle where you’re quizzed on design and deployment methodologies, communication and influencing styles, and risk and resource management amongst other things.  For me personally, its an exciting challenge.  After 2 years of tech-heavy MCTS, MCITPro and MCM preparation I need to balance my skills with the ability to walk into a blue-chip company and say “here’s the right SQL Server architecture for your business and technical requirements, and I’ll lead the implementation of it.”  That to me is powerful.  

Which version of certification should you choose?

Whenever a new major release of SQL Server appears people always starting asking whether they should stop revising for the previous version and start studying for the new version.  Again, that’s a question only you can answer as versions are only old if you start using something newer.  If you work in an environment where your software vendor’s support policy means you’ll never be able to upgrade from version x of SQL Server and you want to learn more about that version then there’s probably no benefit in you taking exams for a newer version that you’ll never use.  If however you want your knowledge to always be relevant and current then you’ll quickly get used to taking exams as soon as they get announced.  However, if you’ve taken that approach in the past make sure you maintain the real-world experience to back up text-book reading otherwise your success with the SQL Server 2012 exams and especially the MCM will be limited.

Strengths and Weaknesses

One of the perceptions newcomers to certification often worry about is that you need to be an expert and get every question in an exam right in order to pass.  If you look at the learning topics and begin to see un-familiar subjects, perhaps impersonation, XML or CLR, you might think you need to be as good at those areas as you are in your stronger areas.  The good news is that to my knowledge, no Microsoft exam requires you to get 100% to pass, in fact the published pass mark for most if not all is 70%.  I’m not suggesting you can ignore entire areas of the syllabus, but at the same time you don’t need to get every question right to pass.  If you go into an exam knowing more about indexing than required but less about XML than needed you might be lucky and pick up extra marks to balance out your overall score.  Of course, there are quite a few topics you’ need to be familiar with the pass but knowing that its ok to get some questions wrong should help remove the fear that you need to be an expert in every part of the exam.  No one is an expert in everything, someone will always have a strength that makes up for a weakness elsewhere.

Knowing what comes in each edition, or more commonly what doesn’t come in Standard Edition especially in the area of high availability, is critical for me when I’m designing solutions for clients.  So, with the recent public release of SQL Server 2012 my hunt for the “official SKU breakdown” began.  There have been licensing datasheets released since late last year but they were too high level for me and the marketing terminology didn’t always translate into technical features. 

Fortunately, TechNet has a feature by edition breakdown well documented on a web page here.  Some of what the page shows you could guess, such as AlwaysOn Availability Groups being Enterprise Edition but 2-node FCIs remaining in Standard, however there were lots of surprises, caveats and footnotes which caught my eye.

I’ve yet to test some of the exact scenarios the web page described, but here is a list of the licensing breakdown which caught my eye:

• Database mail, SSDT, Intellisense, and version control support are not available with the 64-bit version of Standard edition
  
• There are a lot of advanced SSIS features and adapters only in the Enterprise edition and not in the BI edition
  
• Database mirroring is still listed as being supported in Standard and BI editions, just deprecated
  
• Database Tuning Advisor has limited functionality in Standard Edition
  
• The BI and Standard editions are limited to lesser of 4 Sockets or 16 cores
  
• Standard edition also has a “Maximum Compute Capacity” limit, explained in a complex way here
  
• The database engine in BI edition is limited to 64GB of memory but SSAS and SSRS have no memory limitation

I don’t intend to review each observation, instead make a general observation that if you’re looking in Microsoft’s words to deploy more than “non-mission critical” workloads on Standard or BI editions you should check the SKU breakdown first.

Chaining Together Disparate Services

What amazed me was not only how many of the essential services I needed were available for free but also how they integrated with each other.  I’ll now talk you through what I used and how it connected.

Web Site

I have a really really simple web site for the event (link) which just lists information about the next event.  I host this using a free blog provided by WordPress (link) but rather than creating blog posts, I just have a single “page” which I edit with each event’s details.  WordPress have a content management system so I can create and edit the page in a web browser, no need to upload files like a traditional web site.  I should one day make something more of the web site but I’ll save that for the future. 

Domain Name

One of the nice things WordPress also do is sell and host domain names, before integrating them seamlessly with a blog site.  For about $20 a year I have my domain name (sqlserverintheevening.com) linked to my free blog site and hosted by WordPress for me.  That may not sound that impressive today but setting all of this up took about 5 minutes and was done through a couple of simple web pages. Use of the domain name isn’t just limited to the web site though, its also used by email.

Email

WordPress have a setup web page in place that will do all the domain name configuration needed to accept email for your domain.  What’s even better is that they a step by step guide for configuring a Google mailbox to send and receive using your domain name.  This is a pretty complicated process but WordPress’s and Google’s configuration pages together make it a 5 minute process.  5 minutes later I had a Google Mail mailbox setup to send and receive email for sqlserverintheevening.com.

Event Registration

I’ve used EventBrite (link) to meet my venue’s requirements of providing a list of who’s coming to each event.  This is a very slick web site which allows you to customise your event listing, the information you require, and more importantly store and provide the lists of who’s attending each of your events.   

Email Mailing Lists

Finally, after the event I want to email those who attended and tell them about the next event.  For this I use Mail Chimp (link) and their free service is good enough for almost everyone I’d imagine.  The web site does everything you need to create, manage and report on an email “campaign” as they call it and most importantly, people can un-subscribe without my involvement from the list. 

The best part about Mail Chimp for me though was it can integrate into my EventBrite events and extract the names and email addresses of everyone who’s attended one of my previous events.  That was really straight forward and took about 30 seconds, priceless.  I also love being able to see how many emails bounced, were opened and who clicked the web link to the registration site.

The Best Part

There were two best parts for me, firstly all of the services apart from the domain name are free.  The vendors sell premium services but the free versions are more than good enough for my SQL Server community event.  Secondly, setting everything up and sending my first email to everyone took about an hour, and I never saw an error message or have to resort to reading the help pages.  It all just worked.

Finally, I should clarify that for everyone who’s been emailed by me I’ve got multiple ways for them to opt out and to never get an email from me again, you should always do that too.  Mail Chimp is almost too helpful in making sure you don’t spam people.

With SQL Server 2012, your licensing requirements can choose to use or be required to use a per-core model depending on the edition you’re deploying. This is a change to previous editions which used a per-CPU socket model that made counting how many per-CPU licences you needed easier – cores and HyperThreading didn’t influence the CPU socket count.  Any complications which people did have typically came from running SQL Server in a virtualised environment, was a vCPU a socket or did licensing related to the host’s CPU configuration etc.

SQL Server now shares with us for the first time in 2012 its understanding of the server’s CPU configuration on start-up.  In the first few entries in the errorlog you’ll see an entry like these:

SQL Server detected 1 sockets with 4 cores per socket and 4 logical processors per socket, 4 total logical processors; using 4 logical processors based on SQL Server licensing. This is an informational message; no user action is required.

This doesn’t replace the need for accurate licensing purchases based on the hardware architecture you’re deploying but it should at least help you understand what SQL Server’s view of the hardware looks like.

While their contents used regular terms such as node and affinity mask the results were often in an “internals” format that can be distracting to the human reader.  An example of this is the DMV sys.dm_os_nodes (link to Technet here), or more specifically the column cpu_affinity_mask which returns a bigint value that bears no resemblance to the server’s hardware configuration.  Although, as many of you will probably know, the bigint value is a decimal representation of a bitmask that represents the processor value.  To show what I mean, below is an example of the output from a server we manage:

So how do we turn the –281474976710656 value into something meaningful?  The answer comes from a query I found on the SQLServerCentral forums which I then modified, the original link is here, while my version is shown below:

with bits as
(select 7 as n,128 as e union all select 6, 64 union all
select 5, 32 union all select 4, 16 union all select 3, 8 union all
select 2, 4 union all select 1, 2 union all select 0, 1), bytes as
(select 1 m union all select 2 union all select 3 union all
select 4 union all select 5 union all select 6 union all
select 7 union all select 8 union all select 9)
select node_id, cpu_affinity_mask,
(select convert(varchar(3), (( convert(tinyint,
substring(convert(binary(9), cpu_affinity_mask), m, 1))
& e )) / e) as [text()]
from bits
cross join bytes
order by m, n desc
for xml path(”)
) as binaryString
from sys.dm_os_nodes
where node_id < 64

This then gives the more human readable results as follows:

NUMA Node	CPU Mask –Binary
0	0000000000000000000000000000000000000000000000001111111111111111
1	0000000000000000000000000000000011111111111111110000000000000000
2	0000000000000000111111111111111100000000000000000000000000000000
3	1111111111111111000000000000000000000000000000000000000000000000

For those not familiar with the output, each “bit” is a logical CPU within the server.  In this example, the bits set to 1 represent logical CPUs which are assigned to a specific NUMA node, more about that in my SQLBits session!

What is NUMA?

NUMA stands for Non-Uniform Memory Architecture and is a server hardware architecture which any modern server with more than 1 CPU will use.  In two sentences, to make servers scale better each CPU in a server is now only directly connected to specific pieces of the server’s physical memory.  If a CPU needs to access memory which it isn’t directly connected to it needs to go via a memory controllers, this adds latency to the memory request so in an ideal world modern software should do its best to avoid ever needing to read “foreign memory”.  There’s also something called Soft-NUMA which can be misleadingly positioned as being similar to Hard-NUMA, I’ll talk about that more in my session.

What’s this got to do with SQL Server?

Today, SQL Server is regularly deployed on servers with several CPUs and large amounts of memory and as a consequence both SQL Server and Windows are NUMA aware.  They each do their part in trying to ensure these expensive “foreign” memory requests are kept to a minimum, Windows at the CPU instruction level, SQL Server at the SQLOS scheduler level.  

Curiousness develops

You can’t influence how SQL Server works in the NUMA world, it all happens deep within the internals, but you can see NUMA “in action”.  A lot of information about how SQL Sever is internally load balancing its workload and caches between the server’s NUMA nodes is exposed in the DMVs.  And, if you’re anything like me then when you start looking into these DMVs you’ll be curious about what’s actually happening internally.  How do parallel queries run across multiple schedulers when there is a “buffer pool” per NUMA node, why does all the memory seem to be assigned to one NUMA etc etc.

Friday session

It was these questions which made me curious enough to research the topic for myself earlier this year and I hope for those who come to my session on the Friday in March you’ll leave equally as impressed as I was about how SQL Server “floats” across your server’s motherboard.

Since then, I’ve been in discussions with some of the Microsoft product team who are focussed on optimising Windows 8 Hyper-V and SQL Server 2012, specifically Guy Bowerman and Serdar Sutay.  We’ve been talking about some of the challenges SQL Server professionals have today working in virtualised environments and how we could resolve them through future product functionality.  An example of a recent change they’ve engineered into the product has been the support for hot-add memory support in the standard edition of SQL Server 2012 when running in virtual environments, this is still an Enterprise Edition only features for physical servers.  This will allow lower end users of SQL Server to still benefit from the Dynamic Memory feature of Windows Server 2008R2 service pack 1.

In both my presentation and conversations with Microsoft I’ve been discussing the importance of knowing as much as we can about the physical host server, its utilisation, the hypervisor’s configuration as well as what’s happening inside the virtual server – the SQL Server component.  To help, Serdar kindly sent me some information about Hyper-V information exposed inside the virtual machine’s registry.

As a result, I’ve written a Powershell script for DBAs which gives a bit more of an insight about the Hyper-V environment SQL Server is running within.  The script can be found at the bottom of this blog article which gives the following information:

• The hostname of the virtual server the script is running on – for reference purposes.
• The hostname of the physical host server the script is running on – this is useful to be able to tell if the virtual server has been Live Migrated to another physical host server without the DBA knowing.  Why?  Well this maybe an issue if the new physical host server is of a much lower spec than the one it was on previously; if SQL Server jobs start running more slowly than they did yesterday then the problem might have nothing to do with SQL Server at all.  Instead, the new host server might be slower or more highly contended.
• A check to see whether or not its possible for Hyper-‘s Dynamic Memory feature to be used on the current host server.  While we may not be able to tell as easily whether  Dynamic Memory is actually configured for use, knowing whether or not it could be will at least help with memory related performance troubleshooting.
• Finally, the last two checks give information about the physical CPUs in the host server.  This information may not directly be relevant to the performance of the virtual server as CPU resource maybe restricted or contended, but it is a good way to help identify whether you’re using the same physical server as you were yesterday.  If your virtual  server has been moved at least you can tell whether you should be getting the same CPU performance as you were previously.

It’s a fairly basic script which I’m sure could fail badly due to its lack of error handling, however it worked as expected from a Powershell session.

The script is below, apologies if the formatting is slightly odd:

#Get Hyper-V environment data from the registry
$Path = ‘HKLM:\SOFTWARE\Microsoft\Virtual Machine\Guest\Parameters’
$Key = Get-Item -LiteralPath $path

$PhysicalHostname = $Key.GetValue(‘PhysicalHostName’)
$VirtualHostname  = $Key.GetValue(‘VirtualMachineName’)
$OsMajor          = $Key.GetValue(‘HostingSystemOsMajor’)
$OsMinor          = $Key.GetValue(‘HostingSystemOsMinor’)
$ServPack         = $Key.GetValue(‘HostingSystemSpMajor’)

if (($OSMajor -ge ’6′) -and ($OSMinor -ge ’1′) -and ($ServPack -ge ’1′))
{ $DynamicMemory = ‘Supported’}
else
{ $DynamicMemory = ‘Not supported’}

#Get physical CPU information
$CPUName = (Get-WMIObject Win32_Processor).Name
$CPUMHz  = (Get-WMIObject Win32_Processor).MaxClockSpeed

write-host ‘The hostname of this virtual machine is               :’ $VirtualHostname
write-host ‘The hostname of the physical host server is           :’ $PhysicalHostname
write-host ”

write-host ‘Hyper-V Dynamic Memory support on this host server is :’ $DynamicMemory
write-host ”

write-host ‘The CPU type in the physical host server is           :’ $CPUName
write-host ‘The CPU speed in the physical host server is          :’ $CPUMHz ‘MHz’

Moving to SQL Server 2008 is a good move for them and while there maybe more features to use we also hope that out of the box SQL Server 2008 is more reliable, secure and scalable than previous versions. That brings us to the subject of new features. Unless we use those new features we’ll never benefit from them being there, but we can’t use them if we don’t know how to.

That subject is the purpose behind a training course I deliver which is now being made available to the general public for the first time in November. The 2 day course aims to teach developers familiar with SQL Server 2005 and sometimes just 2000 about what’s new for them in 2008. I enrich what could otherwise be just another “what’s new” book chapter by combining feature overviews with demos, labs and experiences from the field.

In summary, upgrading to a new version of SQL Server is a good move, but can be even better if you upgrade your team’s skillsets at the same time. And yes, Denali is coming soon but for a fair number of businesses running SQL Server, 2008 is still either a goal or a recent investment here for a few more years yet.

For those interested in knowing more about my public training course, please see the link here.