It’s not often that I have the opportunity to use an SQL Server technique for the first time. Recently I was able to dramatically improve the query performance of a stored procedure by using the WITH RECOMPILE option. It started out as a real head-scratcher. Even though I just revealed The Spoiler, let me lay out how it unfolded…
Client presents a performance problem with a stored procedure that sources data for a report. The report used to run in 3-4 minutes, but now takes an unacceptable amount of time. I don’t know how long because they have cancelled it after 10 minutes thus far. Next, the usual suspects:
- Its running SQL Server 2019, so not an old code issue.
- Stored procedure execution is only long-running activity on the server. SP_WHOISACTIVE shows other queries taking a second or less to complete, so its not overall server activity.
- CPU is pinned at 100% which I interpret as the data being in memory and not I/O bound. It also shows that this query alone is pinning the needle.
- Ozar Blitz reveals nothing remarkable.
- The query has at least a 20 table join, so the query optimizer is going to have trouble creating a plan that optimally uses indexes and so forth. Big red flag for some query redesign and tuning.
- The query does have some # type temp tables as intermediate result stores, so its broken up into chunks to some degree–could be a good thing.
Leaving us with nothing remarkable in the environment. There is also some thought from the group that the problem may have started after windows/SQL Server patching the previous week. I did not pursue this, as it did not seem a fruitful line of inquiry and the task is to get things working in the current environment, not kick the can down the road by regressing some maintenance that will need to be applied later.
My immediate recommendation is to have the client engage in some query tuning, but there is a Plot Twist!—the client has been able to copy the query code from the stored procedure and when its run as a script in a SMSS query window, it takes 10 seconds to run… As Sherlock Holmes would say, this takes the mystery from the mundane to the interesting.
That is the head-scratcher… We know that in general stored procedures run faster because they have pre-compiled query plans. There is something on the edge of my memory along the lines of the query optimizer not being able to predict the cardinality of temp tables, but I can’t quite put my finger on it. At this point, it is quittin’ time and we agree to restart discussion the next morning. The query authors will look into query tuning—but we keep getting back to the part where it runs fast outside of a stored procedure. One option is to see if the reporting technology can use raw SQL as a query source rather than a stored procedure—not a terribly satisfying solution. I keep thinking “What is it about a pre-compiled query plan that is slower than a script?
So, about 10PM that night, as my head hits the pillow, I start thinking about dynamic query optimization. A few years ago at Ignite, I had the privilege of watching a presentation on new capabilities added to the query optimizer—the ability to take intermediate results of a query and change the query plan based on this information.
I start imagining that the pre-compiled query plan in a stored procedure might not have access to dynamic query optimization. Another vague and ancient memory pops into mind—the WITH RECOMPILE option. I have only read about it, never saw it in the wild.
The next morning I suggest the use of WITH RECOMPILE to the query authors. They report back that the performance of the stored procedure is extremely fast. This seems to me a bit like a brute-force approach, but it did the job and the next task presses, so I left it with that.
The only concern about this sort of solution is that once developers see it making a huge improvement in one query, they will add this to every stored procedure they touch for the rest of their lives, and in this respect, losing the normal advantage of the pre-compiled query plan in a stored procedure
To the extent my assumptions here are wrong, glad to be educated. The only thing I can say for sure is WITH RECOPILE solved the problem.