Fix - When SharePoint 2007 Search Index Crawler is in “Crawling” status indefinitely and doesn't respond to stop requests

This problem started happening in a SharePoint production environment.

The Problem:
The Crawl in SharePoint is permanently in a "Crawling State" permanently. In addition, "View Scopes" read as "not available" and "Scope Properties and Rules" just show item counts of "Total: error".



While there were some suggestions on the web to following (e.g. at http://www.tech-archive.net/Archive/SharePoint/microsoft.public.sharepoint.portalserver/2007-02/msg00284.html)
stsadm -o osearch -action
stopstsadm -o osearch -action start -role indexQuery

And also to "Reset All Crawled Content”.

The Solution:
However, none of these solutions worked for me - after running these commands, the Content crawl status stayed indefinitely in a "Stopping" state and never stopped. It would appear that it is in some kind of infinite loop while crawling. After trying about 30 different courses of action, I finally came across a solution. This was basically to blow away the Shared Service Provider (SSP) Completely and move all existing services over to the new service provider:
http://msmvps.com/blogs/obts/archive/2006/12/18/432542.aspx

Here is the solution to fix it:
1. Go to SQL Server database SharedServices1_Search_DB and delete the records from all the tables with crawlID column (ex. CrawlHistory table).
2. This will stop the crawl.
3. Create a new SharedService with new database using SharePoint Centeral Admin
4. Set new Shared Service to default (by choosing Shared Service Providers in Central Admin and clicking on the "Change Associations" link.
5. Delete the old Shared Service (NOTE that even though the admin service is still against "SharedServices1", it will not get deleted with the SSP - it will automatically migrate over to the new default SSP.)
6. Run the User Profile Crawl (if using Active Directory Integrated Profiles)
7. Setup the crawl rules and run the full crawl.

You'll get clean search crawl results. Problem Solved!

[Update 8 July 2009]
Note that you will also need to reassociate your search scopes with the correct display group "Search Dropdown" after recreating your SSP. See http://ddkonline.blogspot.com/2009/07/sharepoint-2007-search-scopes-not.html