Scott on Writing

Musings on technical writing...

Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0

My two latest articles on 4Guys examine how to efficiently page through very large resultsets in ASP.NET 2.0 using SQL Server 2005's new ROW_NUMBER() keyword. While default paging is as simple as point-and-click in ASP.NET 2.0, it's horribly inefficient and naive, bringing back all records from the underlying data source for each page of data displayed. Custom paging, on the other hand, allows you, the developer, to write code/stored procedures that will intelligently bring back the right subset of records.

The performance difference between these two techniques can be profound for sufficiently large resultsets. As I detail in Custom Paging in ASP.NET 2.0 with SQL Server 2005, paging through a 50,000 record table using default paging was nearly two orders of magnitude less efficient than custom paging. One challenge with custom paging, though, is that adding the boilerplate bi-directional sorting functionality is no longer a point-and-click proposition. Fortunately, it's not terribly difficult to implement, although there are a few subtle points that, if missed or overlooked, can impact performance. In Sorting Custom Paged Results I show how to efficiently include bi-directional sorting with custom paging - all it takes is a bit of dynamic SQL in your stored procedure and indexes on the columns that can be sorted.

Implementing custom paging in ASP.NET 2.0 is a good deal more involved than implementing default paging (as was the case in 1.x), but, in my opinion, custom paging in 2.0 is much easier to implement and get working right than in 1.x. (Ditto for default paging in 2.0 vs. default paging in 1.x.)

The whole custom paging vs. default paging brings up a good question: do you use custom paging all the time? That is, whenever creating a data interface using a GridView (or DataGrid), do you always use custom paging, or do you only implement custom paging if you know the resultset will contain hundreds or thousands of records? Personally, I do default paging if there are going to be less than 100 records in the resultset because of the ease of implementing default paging vs. custom paging. This can be a dangerous tactic, though, because even when you might expect that a resultset will remain relatively small, unless the resultset is something like the months of the year or the states in the US, there's no real guarantee that the size won't later swell into something less manageable. ... food for thought!

posted on Monday, March 20, 2006 4:23 PM

Feedback

# re: Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0 3/20/2006 8:23 PM James Wilson

Nice, but a lot of us are stuck on SQL Server 2000.

So instead of being able to have nice simple SQL queries to do paging, we have to use 3 sub-select monstrosities which require that we have 3 order by clauses containing all the sortable columns, the middle order by inverted, in order to get sorting to work.

Would be helpful to see an article about how to do this cleanly here and now on SQL Server 2000. I don't see us upgrading to 2005 any time soon =)

# re: Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0 3/21/2006 6:28 AM Steve Owens

Scott,
I agree with James in that it would be nice to also see a SQL SERVER 2000 example b/c I'm in the same boat as him.

Thanks for the 2005 article though!

# re: Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0 3/21/2006 7:52 AM Scott Mitchell

James/Steve, thanks for the suggestions - I'll put it in the article queue.

The common approach with SQL Server 2000 that I've used in the past is the stored procedure approach, where you dump the PK column contents of the table to sort through into a temporary table with an IDENTITY field, and then do a select on that, joined with the table to query, using a WHERE clause to only return those records between the startRowIndex and maximumRows.

In any event, I'll add it as an article sometime in the future. Thanks again for the suggestion.

# re: Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0 3/22/2006 2:12 PM Phil Sandler

The problem I see with the Sorting article is that it relies on executing dynamic SQL. I generally try to avoid this whenever possible, because you have to take some extra steps to ensure the query still works.

I've recently used a O/R Mapping/Dynamic Query Engine tool for paging through datasets, and it's many, many times easier to use than stored procs (it's called LLBL Gen Pro).

Unfortunately, our current client doesn't buy into the O/R Mapping concept . . .

Very interesting article and performance stats.

# re: Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0 3/22/2006 2:17 PM Scott Mitchell

I'm not a big fan of dynamic SQL either, Phil, but it's either that or have hard-coded sprocs (or multiple IF/ELSE statements in one sproc) for each of the parameters that can be searched on.

This might not be such a biggie with appropriate tools, as you noted, but I used the dynamic SQL approach in the article because it's the msot generic.

# re: Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0 3/22/2006 2:27 PM Phil Sandler

"The problem I see with the Sorting article is that it relies on executing dynamic SQL."

Eh, I misstated this a little--I didn't mean to imply a flaw in the article. There is no way around this, as you've said.

# re: Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0 3/24/2006 12:43 PM Steve Owens

Scott,
Thanks for the follow up response. I do have a off-topic "wish list" item for a possible upcomming article. Everyone has been posting articles about using the new .NET wizards expecially the new Dataset generation method of a Data Access Layer. What I'd love to see is is an article on how to use the ObjectDataSource object with a true Data Access Layer object (like a class using Generics). The ObjectDataSource plays nice if you want to use a Dataset but when you try using a true Data Access Layer things begin to turn grey. For example how do you return 2 results sets from a Stored Procedure using one trip to the database? If you are inserting a record wouldn't the ObjectDataSource call the Select method after the Insert method has been called thus creating another round trip to the database? I think there is a lot of Caveats when using the ObjectDataSource with something other than a DataSet or DataTable that I'd like to see someone explore further.

Just a thought and thanks for the great acticle.

Keep them comming,
Steve

# re: Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0 3/28/2006 11:49 AM Swanand Mokashi

For SQL Server 2000, we can try doing paging on the database end.

For details checkout my article at :http://www.dotnetgenerics.com/Modules/TricksAndTips/SQLServer/DatabasePaging.aspx

# re: Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0 4/6/2006 1:20 AM Jim

Personally I end up customizing almost everything - and only under excessive pressure/deadlines do I settle for default DataGrid paging etc. I'm trying to get faster at developing custom controls because I always like the results better.

Thanks for all the great articles and books, Scott!

# re: Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0 4/10/2006 9:29 PM Scott Mitchell

For those who are interested in a stored procedure for paging through SQL Server 2000, see:

Efficiently Paging Through Large Result Sets in SQL Server 2000
http://www.4guysfromrolla.com/webtech/041206-1.shtml

# Paging Done RIGHT (in SQL Server 2000 4/25/2006 1:25 PM Scott on Writing

# Sort Direction in Custom Sorting 6/14/2006 12:16 AM Royston

I find the article(s) very useful, however, there seems to be no easy way to pass the DataGrid sorting direction to the ObjectDataSource's Select method.

Find it strange for Microsoft to provide the SortParameterName property for passing of sort expression yet not for sort direction...

# re: Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0 6/14/2006 7:54 AM Rob

Great article again, Scott. But I am having a problem with your solution. If I set the maximumRows param in the ObjectDataSource to lets say, 10, then the Pager only assumes I have 10 results, not the 500 or so that I actually have? I need to blow out the maximumRows to get the pager to show all of the results. But then of course I am getting all the rows in the result set :( which defeats the purpose! What is going on here?

The gridview page size is set to 10...

<asp:Parameter Name="maximumRows" DefaultValue="100000" Type="Int32" />

# re: Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0 6/14/2006 8:41 AM Scott Mitchell

Rob, if you read the article you'll see how to talks about how you have to provide a method that returns the TOTAL number of records paged through. (Check out the code at the end of the article.)

# Paging Done RIGHT (in SQL Server 2000 & 2005) 1/9/2009 8:33 PM ASPInsiders

In an earlier article of mine, Custom Paging in ASP.NET 2.0 with SQL Server 2005 , I wrote about SQL

# re: Efficiently Paging Through LARGE Resultsets in ASP.NET 2.0 11/25/2009 2:12 AM lkionnh h bv

is is an article on how to use the ObjectDataSource object with a true Data Access Layer object (like a class using Generics). The ObjectDataSource plays nice if you want to use a Dataset but when you try using a true Data Access Layer things begin to turn grey. For example how do you return 2 results sets from a Stored Procedure using one trip to the database? If you are inserting a record wouldn't the ObjectDataSource call the Select method after the Insert meth

Title:  
Name:  
Url:
Protected by Clearscreen.SharpHIPEnter the code you see:
Comments   

My Links

Ads Via DevMavens

Archives

Post Categories

 

I am a Microsoft MVP for ASP.NET.
I am an ASPInsider.
<March 2010>
SMTWTFS
28123456
78910111213
14151617181920
21222324252627
28293031123
45678910

Comment Stats

DayTotal% of Total
Sunday 2056.8%
Monday 42514.1%
Tuesday 51917.2%
Wednesday 55518.4%
Thursday 58019.2%
Friday 54718.1%
Saturday 1886.2%
Total 3019100.0%

Hour1Total% of Total
12:00 AM 782.6%
1:00 AM 812.7%
2:00 AM 682.3%
3:00 AM 822.7%
4:00 AM 692.3%
5:00 AM 1264.2%
6:00 AM 1183.9%
7:00 AM 1816.0%
8:00 AM 1926.4%
9:00 AM 1585.2%
10:00 AM 1886.2%
11:00 AM 1936.4%
12:00 PM 2016.7%
1:00 PM 1846.1%
2:00 PM 1695.6%
3:00 PM 1354.5%
4:00 PM 1153.8%
5:00 PM 1073.5%
6:00 PM 1013.3%
7:00 PM 1073.5%
8:00 PM 923.0%
9:00 PM 882.9%
10:00 PM 913.0%
11:00 PM 953.1%
Total 3019100.0%

Comments by Blog Entry Date/Time

Day Entry MadeAvg.Total
Sunday 4.97159
Monday 4.80384
Tuesday 4.04477
Wednesday 7.39680
Thursday 6.26676
Friday 5.07466
Saturday 4.78177
Total 5.403019

Hour1 Entry MadeAvg.Total
12:00 AM 5.2937
1:00 AM 1.002
5:00 AM 0.000
7:00 AM 3.8550
8:00 AM 3.72134
9:00 AM 6.06297
10:00 AM 5.63276
11:00 AM 4.22194
12:00 PM 6.16351
1:00 PM 3.09133
2:00 PM 4.89230
3:00 PM 7.64321
4:00 PM 4.00108
5:00 PM 6.07170
6:00 PM 4.64116
7:00 PM 8.95188
8:00 PM 8.63164
9:00 PM 5.00115
10:00 PM 6.31101
11:00 PM 4.5732
Total 5.403019

Learn More About Comment Stats
1 - All times GMT -8...


Blog Stats

Favorite Web Sites

My Books

My MSDN Articles