When designing software that can be consumed by third-party applications, one rule is very important: public interfaces should not introduce breaking changes, ever. Never ever ever, not in a million years ever. If you release an upgrade with breaking changes to the public interface, those third-party apps that relied on your published interface will fail, and that will piss off two groups of people:
- The creators of the third-party applications, and
- The users using the third-party applications
I put the third-party application creators ahead of the users because they will be really upset because those peeved users will be peeved at the third-party application, even though the third-party application was just abiding by the documented interface.
This all seems like common sense, no? In COM development back in the days, that's all developers heard - if you must change the interface, you need to reversion. Ditto for Web services developers today. What is a bit baffling is why people don't treat website URLs are public interfaces that may be consumed by third-party applications, because that's precisely what they are. The third-party applications are the other Web pages that link to the URL.
There's never any justifiable reason for any website to have a URL that was once existing ever return a 404; a URL that does this is breaking the public interface, a contract implicitly signed by the website creator when he or she created the Web page. I don't care if you rearchitected your entire site; I don't care if it's a URL for a product you no longer sell, in that case display a page kindly explaining that the product is no longer for sale, providing links to similar products/categories you do sell, or -gasp- a link to a site that sells the product. A contract is a contract is a contract. The only excuse for a URL's death is if the company running the website goes out of business, and even then it's a weak excuse.
When you let a URL die, its death ripples to all sites around the world that link to the URL. Users visiting those sites will no experience broken links, and blame not you, Mr. “I Don't Abide By My Public Interface,” but the site that linked to you expecting you to uphold your end of the bargain. Do you know how difficult it is to fix broken links? Sure, if you only have a few dozen broken links, no biggie. But what happens when, say, you have thousands of Web pages with links to a site like, say, MSDN, and then one day MSDN decided to rearchitect the site and all those links that used to work no longer work? What do you do then when you start getting a torrent of emails saying, “These links are broken, what's wrong with you?” You start fixing them, naturally, one at a time, but that is painful, slow, prone to error, and totally unnecessary. (I use MSDN as an example because this is precisely what happened back two years ago or so. I wish I was kidding.)
Yes, I know there are technological solutions one could utilize to aid in finding broken links quickly, but the point remains: it's work/time/effort/energy that shouldn't need to be done in the first place! And, sure, there are other approaches one could take when linking to others' sites, such as by adding a layer of indirection. For example, rather than linking directly to an off-site URL, link to a redirect page on site, like /redir.aspx?ID=x, where x is some ID field in a database table that ties the link to the URL. That way, if there are multiple links to a single now-defunct URL across many pages, all pages can be updated by updating the appropriate database record. While indirection has some nice side benefits - link click tracking, for example - it runs counter to the notion of the World Wide Web, in my opinion. Plus, search engines might not pick up on the link or give it a proper context.
And you know, even if you do rearchitect your site, you can still save those old URLs. It's called URL rewriting, and can be done in a number of ways, from ISAPI filters to 404 handlers that do an automatic redirect to the new URL. Yes, this could result in a good deal of work in the rearchitecting of a large site, but that's what's to be expected when you have such a large public interface. And regardless of how long it takes, that time and effort will pale in comparison to the energy required to fixing broken links from all of those linking to the site.
Ok, now that I got all that out, I feel better. :-)