Monday, 31 May 2010

Fix - SharePoint Very slow to start after an IISRESET or Recycle of App Pool (30-130 seconds)

I was asked by another team at my current client to look at a performance issue that they'd been having major issues with. There were no obvious errors in the Windows Event Log or SharePoint logs related to the issue. The problem was that:
  1. If the application pool is recycled, it would take around 90-120 seconds for the first page to be served. This would be unacceptable to the client in case the App pool was recycled in the middle of the day - it would mean 2 minutes of downtime for all employees.
  2. A similar issue with after an IIS reset was performed - it also happened with ALL sites, not just one or two.
To diagnose the issue, I did the following:
  1. ANY performance improvement should be measurable. So I used the Fiddler Web Debugger (http://www.fiddler2.com/fiddler2/)  to measure the total request time. Time was 84 seconds on this particular test server.
  2. Used Sysinternals Process Explorer to see what the threads were doing. This revealed little - but it was clear that the process wasn't 100% the whole time so it wasn't a problem related to intensive CPU processing.
  3. I enabled ASP.NET tracing at the application level as per http://msdn.microsoft.com/en-us/library/1y89ed7z(VS.71).aspx and viewed the trace log through http://servername/Pages/Trace.axd. However, looking at the load of the control tree - nothing was taking a particularly long time. Even when the trace.axd was loading up, it would take an inordinately long time to start up and server the first requested page. This ruled out the possibility of it being a slow control being rendered.
  4. I created a completely new web application in SharePoint and it exhibited the same problem. I began to suspect machine-level config settings.
  5. I found and fixed several errors in the Windows Event Log and Sharepoint Log but they made no difference.
  6. I began to look at the Fiddler trace while testing again and by chance noticed that requests were also being made to an external address at Microsoft for code signing certificates. I thought this was unusual - so did a bit of research and found that it was checking for a revoked certificates list on a Microsoft web server. This is done when any of the cryptography calls are performed. Some details about this can be found here - but the article is related to Exchange specifically:
    http://msexchangeteam.com/archive/2010/05/14/454877.aspx  
  7. To work around the issue, I tried the registry entries suggested by http://msexchangeteam.com/archive/2010/05/14/454877.aspx, but it didn't seem to work. What DID work was pointing the hosts file so that crl.microsoft.com would resolve to the local host (127.0.0.1).  This meant that the call would much more quickly fail when it tries to access the certificate revoke list at http://crl.microsoft.com/pki/crl/products/CSPCA.crl and http://crl.microsoft.com/pki/crl/products/CodeSignPCA2.crl, and not hold up the loading of Applications on the SharePoint server.
  8. After the HOSTs file change, recycle time (and reset time) went from 84 seconds to 20 seconds.
Hopefully this blog entry helps someone else with diagnosing this slowdown problem. Note that this fix only applies if your server doesn't have access to the internet - it is a problem specific to offline or intranet servers.

[UPDATE] - Found that someone else encountered this same issue as per
http://blogs.technet.com/b/markrussinovich/archive/2009/05/26/3244913.aspx and
http://www.muhimbi.com/blog/2009/04/new-approach-to-solve-sharepoints.html


The first article suggests the use of an XML file in each config - but I've not tested this out:

<?xml version="1.0" encoding="utf-8"?> 
<configuration> 
      <runtime> 
              <generatePublisherEvidence enabled="false"/> 
      </runtime> 
</configuration>

[UPDATE - 11 October 2010]
One of my colleagues from Oakton had a similar issue and the above fix (using the hosts file) didn't work for them.

One of the fixes that did work was to do the following:
"Disable the CRL check by modifying the registry for all user accounts that use STSADM and all service accounts used by SharePoint. Find yourself a group policy wizard or run the vbscript at the end of this posting to help you out. Alternatively you can manually modify the registry for each account:


[HKEY_USERS\\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing]
"State"=dword:00023e00 "

The following script applies the registry change to all users on a server. This will solve the spin-up time for the service accounts, interactive users and new users.


const HKEY_USERS = &H80000003
strComputer = "."
Set objReg=GetObject("winmgmts:{impersonationLevel=impersonate}!\\" _
& strComputer & "\root\default:StdRegProv")
strKeyPath = ""
objReg.EnumKey HKEY_USERS, strKeyPath, arrSubKeys
strKeyPath = "\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing"
For Each subkey In arrSubKeys
  objReg.SetDWORDValue HKEY_USERS, subkey & strKeyPath, "State", 146944
Next



DDK

Wednesday, 26 May 2010

Warning - BizTalk Server 2009 and SQL Server 2008 R2 are incompatible - wait for BizTalk 2010 (aka BizTalk 2009 R2) for "realignment" of compatibility

During installation of Biztalk 2009 tonight I found that SQL Server 2008 R2 and 2009 are in fact incompatible - I couldn't ever get the BizTalk group to install as it was giving errors in the log like so:

2010-05-26 01:30:09:0039 [WARN] AdminLib GetBTSMessage: hrErr=c0c02524; Msg=Failed to create Management database "BizTalkMgmtDb" on server "SERVER01".

You will also get a message box with just a hex code of "0xC0C02524" as below:

I tried manually creating the database - but then it started to give errors with the stored procedure creation.

The below blog matches what I experienced during BizTalk 2009 Group Configuration:

http://blogs.msdn.com/b/biztalkcpr/archive/2009/11/09/biztalk-09-and-sql-r2-not-supported-biztalk-09-and-project-references.aspx

Tuesday, 25 May 2010

Fix - When Configuring BizTalk 2009 - "Failed to connect to the SQL database SSODB on SQL Server SERVERNAME"

If you get the error "Failed to connect the the SQL database SSODB on SQL Server SERVERNAME", in the Biztalk 2009 Configuration Wizard Or the Enterprise SSO Service doesn't start, then you may not have the sso assemblies registered. To do this, do the following:
  1. Open a Visual Studio Command Prompt (so regasm.exe is in the path)
  2. Go to the directory C:\Program Files\Common Files\Enterprise Single Sign-On
  3. Register the assembles with the command regasm ssosql.dll.
  4. Reboot.
  5. DONE!
Additional Note:
If you are doing this in a 64-bit environment, you MUST use the 64-bit version of regasm to register the assemblies. Otherwise it will succeed the regasm command but will not support the Biztalk 2009 configuration wizard and keep failing. The 64 bit version of regasm is here: C:\Windows\Microsoft.NET\Framework64\v2.0.50727 
[As per
http://abdulrafaysbiztalk.wordpress.com/2010/03/30/enterprise-sso-service-start-failure-due-to-installation-of-vs2010/
]DDK

SharePoint Installation/Deployment Best Practice - Using SQL Server Aliases to avoid issues if the SQL Database Server is Renamed, a SQL Instance is added or the SQL Port is changed

One of the problems SharePoint 2007 is that the database name and server name is held in several tables within the configuration database. When you need to change the name of the database server from Central Administration, several references in the configuration tables need to change. There is the hard way and the easy way of doing this as below:
Changing SharePoint Database Server Name - the HARD/WRONG way
Here is the typical process to change the database Server that SharePoint is using:
  1. Move or attach all the sharepoint databases to the new server.
  2. Change the location of the config database with
    stsadm.exe -o setconfigdb -databaseserver ServerName -farmuser MyUserName – farmpassword MyPassword
  3. Delete the Central Administration Site in IIS
  4. Re-run the the SharePoint Products and Technologies Configuration Wizard.
  5. DONE
Changing SharePoint Database Server Name - the EASY/RIGHT way
The above process is a bit painful - and I never like deleting a core component of SharePoint like Central Admin. 
To greatly simplify migraton if you should need to change the port or the server name of your SharePoint database server, you should instead use SQL Server Aliases for your connection from SharePoint to SQL. To do this, go to a command prompt on all servers (as they all connect to the database) and enter the following to open the SQL Client Configuration Tool (note that it doesn't have an i between the n and the f):

cliconfg

Once this is open, you can add a TCP/IP alias as neccessary, which points to your physical Server name, port or instance. Use this alias name when entering the database server in SharePoint - and you wont' look back! The server name change process is then as follows:
  1. Move or attach all the sharepoint databases to the new server.
  2. Change the Alias
  3. DONE!
DDK

Friday, 21 May 2010

Demo Virtual Machine with Office 2010, SharePoint 2010, Visual Studio 2010, Project 2010, Visio 2010 - Hyper-V image now RTM

The "2010 Information Worker Demonstration and Evaluation Virtual Machine (RTM)" image is finally available on Microsoft Downloads here:
http://www.microsoft.com/downloads/details.aspx?familyid=751FA0D1-356C-4002-9C60-D539896C66CE&displaylang=en

The beta has been around for ages so I'm looking forward to getting this working. You can even convert the Hyper-V image to VMWare with the VMware vCenter ConverterTM http://www.vmware.com/products/converter/faqs.html

It includes the following:
  1. Windows Server 2008 R2 Standard Evaluation Edition x64, running as an Active Directory Domain Controller for the “CONTOSO.COM” domain with DNS and WINS
  2. Microsoft SQL Server 2008 R2 Enterprise Edition with Analysis, Notification, and Reporting Services
  3. Microsoft Office Communication Server 2007 R2
  4. Microsoft Visual Studio 2010
  5. Microsoft SharePoint Server 2010 Enterprise Edition
  6. Microsoft Office Web Applications
  7. Microsoft FAST Search for SharePoint 2010
  8. Microsoft Project Server 2010
  9. Microsoft Office Professional Plus 2010
  10. Microsoft Visio 2010
  11. Microsoft Project 2010
  12. Microsoft Office Communicator 2007 R2  
DDK

Fix for SharePoint 2007 Deployment Issue - Solutions in Permanent state of "Deploying", Application Server Administration Service Timer Job in permanent State of 'Initialized'

If you encounter the following issues when attempting to deploy a SharePoint solution (wsp):

  1. It is permanently/constantly in a status of "Deploying" (as seen in Central Administration - Operations - Solution Management)
  2. You cannot cancel the Deployment (no matter how many times you click on 'Cancel')
  3. The SharePoint Timer job definitions are permanently in a status of "Initialized" (as seen in Central Administration - Operations - Timer Job Status).
  4. Your SharePoint Log files are being flooded with the following 5uuf error and growing extremely large (one of ours got to 3GB!):

    05/21/2010 11:41:44.87 OWSTIMER.EXE (0x08E0) 0x08E8 Windows SharePoint Services Timer 5uuf Monitorable The previous instance of the timer job 'Config Refresh', id '{604B2E6E-5850-4C95-8015-D49A61449456}' for service '{681C12E2-4C2C-4BB5-9C9C-BCCF5B4FF5BE}' is still running, so the current instance will be skipped. Consider increasing the interval between jobs.
This is due to an invalid configuration cache on one or or all of the servers. What was happening in my situation was that:
  1. The SharePoint Configuration databasse went down.
  2. The XML configuration files on SERVER02 were updated by SharePoint but not on SERVER01
  3. In fact, there were 500 XML files on one server and 520 on the other whereas they should always be in sync.
However, is is not sufficient to JUST do it on the central admin server. You have to follow this blog entry (http://blogs.msdn.com/josrod/archive/2007/12/12/clear-the-sharepoint-configuration-cache-for-timer-job-and-psconfig-errors.aspx) or (http://support.microsoft.com/kb/939308)  and perform a similar process (as desribed above on each server.)

You must do the following:

  1. Stop the OWSTIMER service on the Index Server and then ALL of the MOSS servers in the farm (for me it was the 2 servers mentioned above e.g. SERVER01 and SERVER02. Just use the following at a command prompt to do this on EVERY WFE and INDEX SERVER
    net stop "Windows SharePoint Services Timer"
  2. [DO FOR ALL SERVERS BEFORE RESTARTING THE TIMER SERVICE]
    Go to C:\Documents and Settings\All Users\Application Data\Microsoft\SharePoint\Config\
    Move (not just copy) all the xml config files (don't delete the config file or the folder itself, just all the xml files) to another location (e.g. a "zz" folder") in a temporary directory as a backup.
  3. [DO FOR ALL SERVERS BEFORE RESTARTING THE TIMER SERVICE]
    Open the cache.ini with Notepad and reset the number to 1. Save and close the file.
  4. Once all XML files are removed and the cache.ini files reset to 1 for ALL SERVERS, run
    net stop "Windows SharePoint Services Timer"on the Index Server first.
  5. Once all the Xml files are generated, run
    net stop "Windows SharePoint Services Timer" on Query Servers and Web Front End Servers in turn.
DDK


Additional Blog Note - how to remove unwanted solutions that are stuck in a "Deploying"State (without fixing the underlying Config problems as detailed above)

  1. Reboot Server (in line with my "If in doubt, restart it!" motto). This in effect will restart all the services anyway.
  2. Use the following command to get a list of all solutions that are being deployed.
    stsadm -o enumdeployments
  3. Use the JobId GUID that comes back from this e.g. Deployment JobId="e99b7304-cfc0-419a-a3f2-18ca5193c838"
  4. Cancel the "stuck" deployment (in "Deploying" status" with the following command
    stsadm -o canceldeployment -id e99b7304-cfc0-419a-a3f2-18ca5193c838
  5. Delete the stuck solution once and for all with
    stsdm -o deletesolution -name mysolution.wsp -override
  6. Redeploy your solution.

Wednesday, 5 May 2010

How to Recursively Get the Group Membership of a User in Active Directory using .NET/C# and LDAP (without just 2 hits to Active Directory)

Problem
What should you do if you need to find all the indirect group memberships that a user has in Active Directory? While it is possible to recursively navigate (ie traverse) through all the group structures that your user is a member of, this can be a very intensive process and can potentially involve 100s of calls to the LDAP server (A slight performance hit to say the list. In sum, BAD!)

In the example below, how do we determine that a user is a member of a top level group without making intensive, recursive calls to Active Directory/LDAP?


Solution
A better option is to use the power of [Microsoft's Implementation] of LDAP to get the results in only 2 hits to the server.
  1. We start of with the user's login name (e.g. david.klein)
  2. We query ldap to get their Container Name (CN) e.g. CN=David Klein
  3. We use the special query syntax provided by Microsoft LDAP in the Directory Searcher Filter to recursively get a list of all groups that the user is directly AND indirectly a member of.
Details

See source code below for 2 helper methods you can use to recursively determine if the designated user is directly or indirectly a member of a particular group. Note that we use the special filter syntax using a specific member flag that will get all indirect memberships automatically for us:

"(member:1.2.840.113556.1.4.1941:=CN=My User Name,OU=Users,OU=NSW,OU=DDKONLINE,DC=DDKONLINE,DC=int)"

/// 
        /// Recursively Gets ALL nested group memberships of a user and checks the input group is there.
        /// 
        /// 
e.g. david.klein or kled123/// 
Container Name of Group e.g. "SP_DEV_HR"/// Uses following config entries
        /// 
        /// 
        /// 
        public static bool IsUserMemberOfGroup(string username, string groupname)
        {
            ///ConfigHelper.LDAPRoot is "LDAP://DC=DDKONLINE,DC=int"
            DirectoryEntry entry = new DirectoryEntry(ConfigHelper.LDAPRoot);
            // Create a DirectorySearcher object.
            DirectorySearcher mySearcher = new DirectorySearcher(entry);
            //Filter by special recursive LDAP string e.g. 
            //"(member:1.2.840.113556.1.4.1941:=CN={0},OU=Users,OU=NSW,OU=DDKONLINE,DC=DDKONLINE,DC=int)"
            mySearcher.Filter = string.Format(ConfigHelper.LDAPGroupMemberFilterRecursive, 
                GetUserContainerName(username));
            mySearcher.SearchScope = SearchScope.Subtree; //Search from base down to ALL children. 
            SearchResultCollection result = mySearcher.FindAll();
            //StringBuilder sb = new StringBuilder();

            for (int i = 0; i < result.Count - 1; i++)
            {
                if (result[i].Path.ToUpper().Contains(string.Format("CN={0}", groupname.ToUpper())))
                    return true; //Success - group found
            }
            //No match found
            return false;
        }

        /// 
        /// Gets the Container Name (CN) of the input user.
        /// 
        /// 
/// 
        public static string GetUserContainerName(string userName)
        {
            DirectoryEntry entry = new DirectoryEntry(ConfigHelper.LDAPRoot);
            // Create a DirectorySearcher object.
            DirectorySearcher mySearcher = new DirectorySearcher(entry);
            mySearcher.Filter = string.Format("(&(sAMAccountName={0}))", userName);
            mySearcher.SearchScope = SearchScope.Subtree; //Search from base down to ALL children. 
            SearchResultCollection result = mySearcher.FindAll();
            if (result.Count == 0)
                throw new ApplicationException(string.Format("User '{0}' Not Found in Active Directory.", userName));
            return result[0].GetDirectoryEntry().Name.Replace("CN=",string.Empty);  
        }

Example Unit Test Methods
/// 
        /// This Test checks that the recursive search works correctly against Active directory.
        /// ie. that it picks up indirect membership
        /// Uses following config entries
        /// 
        /// 
        /// 
        [TestMethod()]
        public void IsUserMemberOfGroup_DirectMembership_Positive_Test()
        {
            string username = "sp_dev_pdmtest1"; 
            string groupname = "SP_DEV_HR"; // TODO: Initialize to an appropriate value
            bool expected = true; // TODO: Initialize to an appropriate value
            bool actual;
            actual = ADHelper.IsUserMemberOfGroup(username, groupname);
            Assert.AreEqual(expected, actual);
        }

        /// 
        /// This Test checks that the recursive search works correctly against Active directory.
        /// ie. that it picks up indirect membership
        /// 
        [TestMethod()]
        public void IsUserMemberOfGroup_IndirectMembership_Positive_Test()
        {
            string username = "sp_dev_pdmtest1";
            //Naming Convention for Groups is Environment_AppDomain_FunctionalArea_ObjectType (e.g. Form)_Role
            string groupname = "SP_DEV_Onlineforms_Peoplemgmt_Termination_F_Contributors"; // TODO: Initialize to an appropriate value
            bool expected = true; // TODO: Initialize to an appropriate value
            bool actual;
            actual = ADHelper.IsUserMemberOfGroup(username, groupname);
                Assert.AreEqual(expected, actual);
        }

        ///This Test Checks that the container name is resolved. Container name is used by the recursive group search.
        ///
        [TestMethod()]
        public void GetUserContainerNameTest()
        {
            string username = "david.klein"; 
            string expected = "David Klein"; 
            string actual = ADHelper.GetUserContainerName(username);
            Assert.AreEqual(expected, actual);
        }


DDK

Error when creating Purchase Order using SAP BAPI_PO_CREATE1 Error - "Material purchase order" is not allowed"

If you get the following error when attempting to create a Purchase Order with Project/WBS related line items:

No instance of object type PurchaseOrder has been created. External reference: ; Purchase order still contains faulty items ; "Material purchase order" is not allowed


The issue may be that the Project (to which you are trying to allocate the cost) is either unapproved or Inactive. Check the status of the WBS or Project that you have chosen and try again.

DDK