Scott on Writing

Musings on technical writing...

Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0

My two latest articles on 4Guys examine how to efficiently page through very large resultsets in ASP.NET 2.0 using SQL Server 2005's new ROW_NUMBER() keyword. While default paging is as simple as point-and-click in ASP.NET 2.0, it's horribly inefficient and naive, bringing back all records from the underlying data source for each page of data displayed. Custom paging, on the other hand, allows you, the developer, to write code/stored procedures that will intelligently bring back the right subset of records.

The performance difference between these two techniques can be profound for sufficiently large resultsets. As I detail in Custom Paging in ASP.NET 2.0 with SQL Server 2005, paging through a 50,000 record table using default paging was nearly two orders of magnitude less efficient than custom paging. One challenge with custom paging, though, is that adding the boilerplate bi-directional sorting functionality is no longer a point-and-click proposition. Fortunately, it's not terribly difficult to implement, although there are a few subtle points that, if missed or overlooked, can impact performance. In Sorting Custom Paged Results I show how to efficiently include bi-directional sorting with custom paging - all it takes is a bit of dynamic SQL in your stored procedure and indexes on the columns that can be sorted.

Implementing custom paging in ASP.NET 2.0 is a good deal more involved than implementing default paging (as was the case in 1.x), but, in my opinion, custom paging in 2.0 is much easier to implement and get working right than in 1.x. (Ditto for default paging in 2.0 vs. default paging in 1.x.)

The whole custom paging vs. default paging brings up a good question: do you use custom paging all the time? That is, whenever creating a data interface using a GridView (or DataGrid), do you always use custom paging, or do you only implement custom paging if you know the resultset will contain hundreds or thousands of records? Personally, I do default paging if there are going to be less than 100 records in the resultset because of the ease of implementing default paging vs. custom paging. This can be a dangerous tactic, though, because even when you might expect that a resultset will remain relatively small, unless the resultset is something like the months of the year or the states in the US, there's no real guarantee that the size won't later swell into something less manageable. ... food for thought!

posted on Monday, March 20, 2006 4:23 PM

Feedback

# re: Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0 3/20/2006 8:23 PM James Wilson

Nice, but a lot of us are stuck on SQL Server 2000.

So instead of being able to have nice simple SQL queries to do paging, we have to use 3 sub-select monstrosities which require that we have 3 order by clauses containing all the sortable columns, the middle order by inverted, in order to get sorting to work.

Would be helpful to see an article about how to do this cleanly here and now on SQL Server 2000. I don't see us upgrading to 2005 any time soon =)

# re: Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0 3/21/2006 6:28 AM Steve Owens

Scott,
I agree with James in that it would be nice to also see a SQL SERVER 2000 example b/c I'm in the same boat as him.

Thanks for the 2005 article though!

# re: Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0 3/21/2006 7:52 AM Scott Mitchell

James/Steve, thanks for the suggestions - I'll put it in the article queue.

The common approach with SQL Server 2000 that I've used in the past is the stored procedure approach, where you dump the PK column contents of the table to sort through into a temporary table with an IDENTITY field, and then do a select on that, joined with the table to query, using a WHERE clause to only return those records between the startRowIndex and maximumRows.

In any event, I'll add it as an article sometime in the future. Thanks again for the suggestion.

# re: Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0 3/22/2006 2:12 PM Phil Sandler

The problem I see with the Sorting article is that it relies on executing dynamic SQL. I generally try to avoid this whenever possible, because you have to take some extra steps to ensure the query still works.

I've recently used a O/R Mapping/Dynamic Query Engine tool for paging through datasets, and it's many, many times easier to use than stored procs (it's called LLBL Gen Pro).

Unfortunately, our current client doesn't buy into the O/R Mapping concept . . .

Very interesting article and performance stats.

# re: Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0 3/22/2006 2:17 PM Scott Mitchell

I'm not a big fan of dynamic SQL either, Phil, but it's either that or have hard-coded sprocs (or multiple IF/ELSE statements in one sproc) for each of the parameters that can be searched on.

This might not be such a biggie with appropriate tools, as you noted, but I used the dynamic SQL approach in the article because it's the msot generic.

# re: Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0 3/22/2006 2:27 PM Phil Sandler

"The problem I see with the Sorting article is that it relies on executing dynamic SQL."

Eh, I misstated this a little--I didn't mean to imply a flaw in the article. There is no way around this, as you've said.

# re: Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0 3/24/2006 12:43 PM Steve Owens

Scott,
Thanks for the follow up response. I do have a off-topic "wish list" item for a possible upcomming article. Everyone has been posting articles about using the new .NET wizards expecially the new Dataset generation method of a Data Access Layer. What I'd love to see is is an article on how to use the ObjectDataSource object with a true Data Access Layer object (like a class using Generics). The ObjectDataSource plays nice if you want to use a Dataset but when you try using a true Data Access Layer things begin to turn grey. For example how do you return 2 results sets from a Stored Procedure using one trip to the database? If you are inserting a record wouldn't the ObjectDataSource call the Select method after the Insert method has been called thus creating another round trip to the database? I think there is a lot of Caveats when using the ObjectDataSource with something other than a DataSet or DataTable that I'd like to see someone explore further.

Just a thought and thanks for the great acticle.

Keep them comming,
Steve

# re: Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0 3/28/2006 11:49 AM Swanand Mokashi

For SQL Server 2000, we can try doing paging on the database end.

For details checkout my article at :http://www.dotnetgenerics.com/Modules/TricksAndTips/SQLServer/DatabasePaging.aspx

# re: Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0 4/6/2006 1:20 AM Jim

Personally I end up customizing almost everything - and only under excessive pressure/deadlines do I settle for default DataGrid paging etc. I'm trying to get faster at developing custom controls because I always like the results better.

Thanks for all the great articles and books, Scott!

# re: Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0 4/10/2006 9:29 PM Scott Mitchell

For those who are interested in a stored procedure for paging through SQL Server 2000, see:

Efficiently Paging Through Large Result Sets in SQL Server 2000
http://www.4guysfromrolla.com/webtech/041206-1.shtml

# Paging Done RIGHT (in SQL Server 2000 4/25/2006 1:25 PM Scott on Writing

# Sort Direction in Custom Sorting 6/14/2006 12:16 AM Royston

I find the article(s) very useful, however, there seems to be no easy way to pass the DataGrid sorting direction to the ObjectDataSource's Select method.

Find it strange for Microsoft to provide the SortParameterName property for passing of sort expression yet not for sort direction...

# re: Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0 6/14/2006 7:54 AM Rob

Great article again, Scott. But I am having a problem with your solution. If I set the maximumRows param in the ObjectDataSource to lets say, 10, then the Pager only assumes I have 10 results, not the 500 or so that I actually have? I need to blow out the maximumRows to get the pager to show all of the results. But then of course I am getting all the rows in the result set :( which defeats the purpose! What is going on here?

The gridview page size is set to 10...

<asp:Parameter Name="maximumRows" DefaultValue="100000" Type="Int32" />

# re: Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0 6/14/2006 8:41 AM Scott Mitchell

Rob, if you read the article you'll see how to talks about how you have to provide a method that returns the TOTAL number of records paged through. (Check out the code at the end of the article.)

Title:  
Name:  
Url:
Protected by Clearscreen.SharpHIPEnter the code you see:
Comments   

Add To Your Reader

My Links

Archives

Post Categories

 

I am a Microsoft MVP for ASP.NET.
I am an ASPInsider.
<May 2008>
SMTWTFS
27282930123
45678910
11121314151617
18192021222324
25262728293031
1234567

Comment Stats

DayTotal% of Total
Sunday 1866.8%
Monday 37913.9%
Tuesday 45316.7%
Wednesday 50418.5%
Thursday 53519.7%
Friday 49418.2%
Saturday 1666.1%
Total 2717100.0%

Hour1Total% of Total
12:00 AM 652.4%
1:00 AM 682.5%
2:00 AM 622.3%
3:00 AM 742.7%
4:00 AM 572.1%
5:00 AM 1033.8%
6:00 AM 1084.0%
7:00 AM 1585.8%
8:00 AM 1716.3%
9:00 AM 1475.4%
10:00 AM 1716.3%
11:00 AM 1816.7%
12:00 PM 1886.9%
1:00 PM 1696.2%
2:00 PM 1605.9%
3:00 PM 1324.9%
4:00 PM 1073.9%
5:00 PM 923.4%
6:00 PM 913.3%
7:00 PM 963.5%
8:00 PM 833.1%
9:00 PM 782.9%
10:00 PM 792.9%
11:00 PM 772.8%
Total 2717100.0%

Comments by Blog Entry Date/Time

Day Entry MadeAvg.Total
Sunday 5.54144
Monday 5.22339
Tuesday 4.28419
Wednesday 7.67637
Thursday 6.90607
Friday 5.48411
Saturday 5.33160
Total 5.842717

Hour1 Entry MadeAvg.Total
12:00 AM 5.0035
1:00 AM 1.002
5:00 AM 0.000
7:00 AM 7.0035
8:00 AM 5.35107
9:00 AM 6.32278
10:00 AM 6.47246
11:00 AM 4.41181
12:00 PM 6.88330
1:00 PM 3.00111
2:00 PM 5.41222
3:00 PM 8.64285
4:00 PM 4.0589
5:00 PM 5.92154
6:00 PM 4.52113
7:00 PM 9.67174
8:00 PM 9.80147
9:00 PM 5.05111
10:00 PM 5.4265
11:00 PM 4.5732
Total 5.842717

Learn More About Comment Stats
1 - All times GMT -8...


Blog Stats

Favorite Web Sites

My Books

My MSDN Articles