Translation: Fail Whale for SQL Server Management Studio and Central Management Server for Policy Based Managment using PowerShell…. lol simple eh?

So this is a Microsoft Connect Item in the making…

I had created a PowerShell script to run a bunch of policies against a list of servers that I defined in a ServerList.txt file.  I decided that I didn’t like the idea of needing to maintain a text file and instead of storing the Instance Names in a table somewhere I decided to hook into the Central Managment Server.

So I logged into the server that is currently running the SQL Server Agent job that was doing all this using a ServerList.txt file.  I changed the PowerShell Script to point to my CMS server instead and ran the .ps1 directly on the server using “sqlps F:\PBM\EvaluatePolicies.ps1″.  The script ran as expected, used my CMS server, and everything was tickidyboo.  Or so I thought….

I decided to do my final check and run the SQL Server Agent job itself… and it failed.

Hmmmm that’s weird… so I confirmed that the SQL Server Agent account had access to the CMS server and even granted it explicit sysadmin rights (just for my troubleshooting).

FAIL

I started tweeting about the problem and got some immediate advice from @afernandez, @russjohnson, @wendy_dance, and @SQLChicken.  While none of the advice solved my problem it was great to have some other “eyes” reading my problem and confirming the stuff that I had already thought of and tested.

The SQL Server Agent Service Account is a secure account that I don’t even have access or the password for so I asked for some help from one of our Server Admins for my troubleshooting… he logged into the server for me and I tried to run the ps1 file….

FAIL

So I opened up SSMS and checked if I could connect to the CMS directly….

SOLUTION:  In order to use a PowerShell script to connect to the Central Management Server using the SQL Server Agent the service account that SQL Server Agent is running under must REGISTER the CMS Server through SSMS on the Server that is running the job.

Did you get all that…. because I had never Remote Desktopped onto the Server using the SQL Server Agent service account and Registered the CMS Server in SSMS using that account when I tried to access the CMS Server through my PowerShell script it failed.

I registered the CMS Server using the SQL Server Agent Service Account and everything started sizzling like bacon on a hot pan.

Enjoy!!

“I’ll Wiki you my pretty… and your little script too!!”

So I submitted my first Wiki article over at SQLServerPedia:

Restoring a User Database to a Different Environment

and I have to admit that when I first started thinking about it I was quite concerned about how to do it, whether the script is “wiki worthy”, am I the only one who would find this useful, etc… So let’s talk about it

How to do it – Well look no further then Brent Ozar to provide an amazing how to video: How to Write and Edit Articles

Is my script “Wiki-Worthy” – This is the great thing about a wiki… even if your script is a just basic script to make your job easier what it provides is a building block for others to take and grow into something that turns into a more robust solution. 

Would Anyone Else Find This Useful – Do YOU find your script useful?  So what makes you think you’re so special that others wouldn’t find it useful or maybe they have a similar script that doesn’t do as much as what your script does… maybe it does more and now someone can edit your wiki article to include their contributions.

At the end of the day a wiki article is a great way to not only give something back to the community but it also provides an excellent mechanism to have others edit and contribute their 2 cents into the script or discuss how they solve a similar problem.  If SQL Server has taught me anything it’s taught me that there are a million ways to solve a problem and the way that YOU solve your problem might work but there might be a better solution out there.

Now if you’ve thought of submitting a wiki article to SQLServerPedia… now is the time.  SQLServerPedia is giving away an all expenses paid trip to the PASS Summit for new writers that have never been to the Summit before…  Details Here

See you in Seattle!!

Enjoy!!

Everytime I see a database with both AutoGrow and AutoShrink enabled I cringe.  I don’t “snicker” cause I leave any and all snickering to when someone says the words ”Access Database”.  See here if you’re confused.

OK so why is having both AutoGrow and AutoShrink enabled on your database such a bad thing… I mean Microsoft wouldn’t give us these options unless there was a good reason to use them right? WRONG!!

I’ll admit that as a safety net, having AutoGrow set on your database is not a bad idea.  The rub here is that with proper capacity planning and Database File Management the “AutoGrow” process should never actually happen.  I don’t know about you but I like to know when, where, and how my database files are growing to minimize the amount of fragmentation and potential performance hit. 

Let’s talk about a world where Capacity Planning and Database File Managment doesn’t exist… a world where AutoGrow and AutoShrink are both enabled… a world where if you database needs more space you let SQL Server AutoGrow and if your database has too much space you let SQL Server AutoShrink.  This is a world I don’t ever want to live in… EVER

Why is this viscious cycle such a bad idea?

• Resources – If you’re constantly growing and shrinking your database files there is a potentially significant CPU and IO hit (especially with the shrink) and you’re going to end up nickel and diming yourself to death. 
• Fragmentation – If you’re constantly autogrowing and autoshrinking your database files will end up in bits and pieces all over your disk.  If you’re data is spread out all over the disk think of the additional performance hit in reading that data and combine that with the fact that autoshrinking can cause huge fragmentation in your indexes and that’s quite a double whammy.
• Control – If you’re relying on AutoGrow and AutoShrink to manage your database files you’re giving away control as to when these operations will execute.  These processes don’t wait or care about scheduled downtime or maintenance windows… if SQL Server determines it needs to autogrow or autoshrink it’s going to do it. 

At the end of the day when it comes to your Database File Management you want to be a control freak.  You don’t want to put your database on a yo-yo diet where it’s constantly growing and shrinking and ending up having a negative impact on it’s “metabolism” (Performance)

There is actually a Microsoft Connect Item by Aaron Bertrand to Deprecate all Shrink Database Options… join the fight and give this item the 5 star rating it deserves!!

Enjoy!!

No not THAT KISS… the Keep It Simple Stupid theory.

So when it comes to backups… if I’m not needing to “trick them out” or do anything custom I don’t have any problems with setting up a nice and easy maintenance plan and calling it a day.  I know there are some strong opinions out there that you should avoid Maintenance Plans like the plague and I agree that in some situations that they are not your best option but I don’t like to write them off altogether.

Sorry… that went longer then expected…ok back on topic…

xp_delete_file… So I needed to build a custom backup job because with the size of the database I would get greater performance by backing up to multiple backup files (an option that is not available in a maintenance plan).  Now normally I would just attach the maintenance cleanup task to another maintenance plan but I thought “Hey… let’s get crazy” I’m going to click on that View T-SQL Button and see what happens.

Well isn’t that easy?

EXECUTE master.dbo.xp_delete_file 0,N'\\ServerName\Folder',N'BAK','2009-08-13T08:14:17',1

Could it REALLY be that simple?

Hey… what’s with that “0″ at the front and the “1″ at the end?  What are those parameters for?

I BINGed the question (OK I really GOOGLEd it… sorry Microsoft I just can’t break my google ties) and found that the first arguement determines if it’s a backup file or a report file (0 and 1 respectively) and the last arguement determines whether or not to include first level subfolders (0 – no 1 – yes)

Great… it looks like I’ve found my solution….. but wait for it…… wait for it……..

ENTER PowerShell….

(Great picture BTW by Joe Brinkman – The Accidental Geek )

OK admittedly I’m one of the 42% on Brent Ozar’s Powershell Poll that don’t currently use PowerShell but want to.  So I thought here is my chance…

 I started out simple:

Built myself a nice little .ps1 file with this:

$directory = “C:\”
Get-ChildItem $directory

Ran that bad boy and low and behold… I’m a PowerShell developer

So I thought… let’s add my network share location for my backup files:

$directory = “\\servername\folder”
Get-ChildItem $directory

Ran that and I’m back to being a PowerShell failure:

Get-ChildItem : Cannot find path ‘\\servername\folder’ because it does not exist.

Turns out you need to map a drive for a network share (or at least that’s how I solved my problem… please comment if there is a more elegant solution):

$directory = “\\ServerName\Folder”
New-PSDrive -name X -psprovider FileSystem -root $directory
Get-ChildItem X:\

 Now we’re cooking with fire

Next I changed my Get-ChildItem statement to filter only the *.BAK files and I set it to recursively go through subdirectories:

$directory = “\\ServerName\Folder”
New-PSDrive -name X -psprovider FileSystem -root $directory
Get-ChildItem X:\ -include *.BAK -recurse

This is getting good now Now I wanted to only get the *.BAK files that are older then 4 hours… this took some googling to get my syntax right:

$directory = “\\ServerName\Folder”
New-PSDrive -name X -psprovider FileSystem -root $directory
Get-ChildItem X:\ -include *.BAK -recurse | where {((get-date)-$_.creationTime).hours -ge 4}

 And finally I wanted to delete these files so we “pipe in” a remove-item and we’re done:

$directory = “\\ServerName\Folder”
New-PSDrive -name X -psprovider FileSystem -root $directory
Get-ChildItem X:\ -include *.BAK -recurse | where {((get-date)-$_.creationTime).hours -ge 4} | remove-item -force -recurse

And if you wanted to add your transaction log backups as well you could copy/paste your Get-ChildItem and change accordingly

Get-ChildItem X:\ -include *.TRN -recurse | where {((get-date)-$_.creationTime).days -ge 1.5} | remove-item -force -recurse

And there you go… I’m not sure if this means I’m now a PowerShell user but I’m definitely starting to drink the KoolAid for sure.

Enjoy!!

So how do you drop databases?

Do you use a simple DROP DATABASE statement?

Do you detach the database and remove the files manually?

Do you use the GUI?

Run these two quick and dirty queries for me:

SELECT COUNT(*)
FROM   msdb.dbo.backupset

SELECT COUNT(*)
FROM   msdb.dbo.backupset
WHERE  database_name NOT IN (SELECT name FROM master.dbo.sysdatabases)

So how much of your MSDB database is filled with backup history of databases that have long since been dropped from your instance?

MSDB love and care is not something that usually gets alot of press but if you’re a stickler for wanting to keep things neat and tidy take a look at the information in your MSDB and whether or not you’re storing (and backing up) more data then you need to.

Want to rid yourself of some MSDB bloating?

SELECT DISTINCT(database_name)
FROM   msdb.dbo.backupset
WHERE  database_name NOT IN (SELECT name FROM master.dbo.sysdatabases)

So now that you have a list of those old databases that still have backup history kicking around… now what?

Don’t jump the gun and write a DELETE statement for msdb.dbo.backupset… what about the other backup tables like backupfile, etc?

Answer: msdb.dbo.sp_delete_database_backuphistory

EXEC msdb.dbo.sp_delete_database_backuphistory @db_nm = N'<<DATABASE_NAME>>'

Using this system stored procedure you’ll ensure that you’ve removed all the backup history for the specified database.

(As always, please use with caution)

Enjoy!!

If a database fails in the enviroment and no backup is there to recover from does it make a sound?

Well no… but the users are heard for miles hahaha

Below is a nice quick sanity check showing all the dbs that have NOT been backed up for X days or have never been backed up at all. 

DECLARE @num_of_days INT

SET    @num_of_days = 1 

SELECT     A.name, MAX(B.backup_finish_date) AS 'LastBackupDateTime'

FROM   master.dbo.sysdatabases A WITH(NOLOCK)  LEFT OUTER JOIN

   msdb.dbo.backupset B WITH(NOLOCK) ON A.name = B.database_name

WHERE  (B.TYPE = 'D' OR B.TYPE IS NULL)

GROUP BY A.name

HAVING (MAX(B.backup_finish_date) < GETDATE() - @num_of_days OR MAX(B.backup_finish_date) IS NULL) 

ORDER BY A.name



Enjoy!!

A request came in to upgrade a SQL Server 2005 Standard Edition Instance to a SQL Server 2005 Enterprise Edition Instance.  Funny thing is that I’ve been working with SQL Server 2005 since it’s release and haven’t had to do this before… weird… so I setup a test box with 2005 STD to play around with and mounted myself a 2005 ENT Edition ISO and ran the GUI setup.

After a bit the wizard told me that in order to upgrade my edition of SQL Server I needed to install via command line and use the SKUUPGRADE switch.

So I googled myself some Books Online goodness cause installing SQL from the command line is not something I do everyday LOL and found this:

How to: Install SQL Server 2005 from the Command Prompt
http://msdn.microsoft.com/en-us/library/ms144259(SQL.90).aspx

 [SKUUPGRADE]
Use the SKUUPGRADE parameter when upgrading from one edition of SQL Server to another edition of SQL Server.

Important: 
If you use the SKUUPGRADE parameter, Setup will remove all hotfix and service pack updates from the SQL Server instance being upgraded. Once the edition upgrade is complete, you must reapply all hotfix and service pack updates.
 

To perform an edition upgrade of a default instance of SQL Server (SQL Engine only) via unattended Setup, run the following command using either /qb or /qn:

start /wait setup.exe ADDLOCAL=SQL_Engine INSTANCENAME=MSSQLSERVER UPGRADE=SQL_Engine SKUUPGRADE=1 /qb

The /qb switch allows display of basic Setup dialog boxes. Error messages are also displayed.

The /qn switch suppresses all Setup dialog boxes and error messages. If the /qn switch is specified, all Setup messages, including error messages, are written to Setup log files.

 So before doing this I took note of the Service Pack + Hotfix level so that I could reapply them after the edition upgrade was complete.

I ran the script as indicated (after a full backup of course) and it worked like a charm.  After reapplying the service pack + hotfixes required to get me back to the same version the users tested and all went as expected.

Not sure how I avoided doing an edition upgrade in the last 5 years or so but it was fun to do something with SQL that was new to me.

and just in case you missed it…

Important: 
If you use the SKUUPGRADE parameter, Setup will remove all hotfix and service pack updates from the SQL Server instance being upgraded. Once the edition upgrade is complete, you must reapply all hotfix and service pack updates.

Enjoy!!

If you move MSDB your notifications will stop working.  Why you may ask?

It’s because the ENABLE_BROKER configuration is set to false when you move MSDB.  Now I can’t seem to find (google) any information as to WHY this is (please send comments if you have further info) but in order to get your notifications up and running again you’ll need to run the following command:

ALTER DATABASE msdb SET ENABLE_BROKER;

If this takes a while to run then chances are you are being blocked by another process (chances are the SQL Server Agent)

If you’re interested in reading more here is the Microsoft BOL Link on moving system databases.

Enjoy!!

Enough touchy feely stuff… let’s get to the guts of the post…

OK Chicken Little your database is corrupt and your sky is falling.  You want this dirty feeling gone as quickly as possible.  Before you go and do something crazy like blindly running a repair or detaching a suspect database take some time to do some investigation work to see if you can figure out what is going on.  At the end of the day you normally have 2 options… try and repair the damage or restore the database from backup. In order to determine which path is best for you you should do some things like running DBCC CHECKDB and looking in the Error Log to try and give yourself some clues as to the extent of the problem.  Check out the Windows Event Log, are there any RAID or disk errors? In the result set of your DBCC CHECKDB you might get 1 error, you might get 100 errors, or it might not even finish if your db is in a really bad state.  If DBCC CHECKDB can’t run at all then you’re pretty well looking at recovering from backup. There are some other errors in the CHECKDB output that would usually force you to go back to a backup, things like data purity errors or page header corruption are a couple examples.

An “easy” thing that can come out of your CHECKDB output is if a nonclustered index is corrupt.  If the indexid is greater than 1 you can rebuild it offline in SQL Server 2005 or recreate it in SQL Server 2008 and Bob’s your uncle.  Now this doesn’t tell you WHY this happened but it will resolve your corrupt index and (hopefully) get you back into the game. 

Another example of a “repairable” problem is a corrupt page(s).  If you have corrupt pages your first thought could be to try and do a single page restore using a full backup and then applying the subsequent transaction log backups.  What if you only do nightly full backups of your database?  An option in this case is the REPAIR_ALLOW_DATA_LOSS argument in your DBCC CHECKDB statement.  This is a well named argument because this can/will cause rows in your table(s) to be removed in order to attempt to fix the corruption.  Always be sure to do a full backup BEFORE using the REPAIR_ALLOW_DATA_LOSS option just to be sure you can get back to where you are now if something goes wrong.  This option also doesn’t care about constraints so be sure to run a DBCC CHECKCONSTRAINTS command after the REPAIR_ALLOW_DATA_LOSS.

Anyways at the end of the day corruption happens… do what you can through alerts and scheduled running of DBCC CHECKDB to try and capture/isolate the corruption as soon as possible in order to minimize damage and impact to the business. 

One final thought… even though you may think that you fixed the problem(s) ALWAYS run a final CHECKDB just to make sure that you’ve really saved the world (or at least your database) from corruption.

Here is the Microsoft Books Online for DBCC CHECKDB make sure to give this a read (twice) before acting.

Enjoy!!

“SQL server starts Auto update stats when data modification for a table is more than 20 %. ”

“Statistics will only be updated by Auto Update when SQL Server thinks it needs to be done (which is typically when 20% of the rows have changed). If you are manually updating statistics pretty frequently, AutoUpdate may not fire very often, if at all.”

“However, you might not know that approximately 20% of a table needs to change before triggering an auto_update statistics run for the given table. If the table is really big, it’s possible that you’re simply not changing enough data.”

“It will be nice if we can change the 500+20% formula so we can have the auto update stats more often / less often”

So I thought why not put that to the test and see what is happening. Let’s first make sure that auto update statistics is enabled on our working database:

SELECT DATABASEPROPERTY(DB_NAME(), 'IsAutoUpdateStatistics') AS 'IsAutoUpdateStatisticsEnabled'


Make sure this value is set to 1 otherwise this test wont really mean much to you.

Now let’s set ourselves up a table to work with:

SET NOCOUNT ON 

CREATE INDEX IX_City ON tblSQLServerPediaUsers(City)

And just for tracking purposes let’s see what our statistics look like:

DBCC SHOW_STATISTICS ('tblSQLServerPediaUsers', PK_tblSQLServerPediaUsers) WITH STAT_HEADER
DBCC SHOW_STATISTICS ('tblSQLServerPediaUsers', IX_City) WITH STAT_HEADER

OK so no statistics for our Primary Key (Clustered Index) and up to date statistics with a full scan for our non clustered index. OK so taking what seems to be general public acceptance that if you change 20% or more of the table data that auto update statistics should kick off let’s start a profiler trace and in your events selection under Performance let’s check off Auto Stats. This will tell us if/when auto stats gets executed. So let’s update more than 20% of the data in the table (while the profiler trace is running):

UPDATE tblSQLServerPediaUsers
SET    
City = 'Edmonton',
   FirstName = 'Ales',
   LastName = 'Hemsky',
   Address1 = 'Rexall Place'
WHERE ID IN( SELECT TOP 2499 ID
               FROM tblSQLServerPediaUsers
               ORDER BY NEWID())

And if we look at the profiler trace we see Auto Stats ran but if we look at the TextData we see:

Updated: tblSQLServerPediaUsers.PK_tblSQLServerPediaUsers

Hmmmm the index IX_City didn’t appear to get updated. I updated more than 20% of the table and auto stats did happen but it only happened for my Primary Key. Let’s check our statistics just to be sure:

DBCC SHOW_STATISTICS ('tblSQLServerPediaUsers', PK_tblSQLServerPediaUsers) WITH STAT_HEADER
DBCC SHOW_STATISTICS ('tblSQLServerPediaUsers', IX_City) WITH STAT_HEADER

We see the Primary Key now has statistics (as expected) and we also confirmed that the IX_City did not get updated. Let’s see how this affects querying the tblSQLServerPediaUsers table (make sure to have “show execution plan” turned on as well):

SET STATISTICS IO ON
GO
SELECT FirstName, LastName FROM tblSQLServerPediaUsers WHERE City = 'Edmonton'

Execution Plan shows that our index (IX_City) is used and the Logical Reads is at 5011. So I guess all is right in the world right? Just for fun let’s see what kind of stats we would get if we forced the same query to use the primary key (clustered index)

SET STATISTICS IO ON
GO
SELECT FirstName, LastName FROM tblSQLServerPediaUsers WITH(INDEX=PK_tblSQLServerPediaUsers) WHERE City = 'Edmonton'

Execution Plan shows that our forcing the query to use the Primary Key (PK_tblSQLServerPediaUsers) worked but wait a sec… the logical reads is at 127. In our previous run why didn’t the query engine use the primary key instead. Let’s look at what the profiler trace has in it:

Using the IX_City Index:
CPU 47
READS 5011
WRITES 0
DURATION 61

Forcing the query to use PK_tblSQLServerPediaUsers Clustered Index:
CPU 15
READS 127
WRITES 0
DURATION 43

CPU is lower, reads is lower, and duration is lower. So why didn’t the query engine use PK_tblSQLServerPediaUsers instead of IX_city? This is because the auto update stats did not update ALL the statistics. Let’s run an sp_updatestats:

EXEC sp_updatestats

Now let’s run the original query again (without forcing the query to use the primary key):

SET STATISTICS IO ON
GO
SELECT FirstName, LastName FROM tblSQLServerPediaUsers WHERE City = 'Edmonton'

Look at that… the query now uses the primary key instead of using the IX_City index.

So back to our original question of “Do I have to update my statistics if I have auto update stats enabled” My answer stays the same (yes) but now I have a working example to argue my point of view with.

Disclaimer (lol): I’m the first to admit that I don’t know absolutely everything there is to know about the query engine so please feel free to comment and provide some further information/details that I might be leaving out, missing, or TOTALLY wrong about.

Enjoy!!