Wednesday, 8 June 2011

SharePoint 2010 - Setting the Title for your External Content Type (esp for Display of Friendly name in BCS Associations)

A simple tip for modifying External Content Types in SharePoint Designer 2010 - if you have an external content type (e.g. talking to SAP via Duet Enterprise) - and you want to control the Display name that is shown in the Pickers, you need to change the Title of your External Content Type. You can do this by clicking on the Field you want to use as the Display name, and then click the "Set as Title" Ribbon Button under the field group. See below:

I was half expecting a context menu on right click for this functionality - but the UI designer at Microsoft has chosen the ribbon option for this functionality (unlike some other areas of SharePoint Designer).

DDK

Wednesday, 4 May 2011

Exception when Browsing Locally to Report Server/Creating an new TFS Project on Local Machine

If you have a Report Server e.g. as Part of a TFS 2010 Setup, and attempt to access your site locally (e.g. https://servername/Reports/) , you may get the following exception:

User 'DOMAIN\UserName' does not have required permissions. Verify that sufficient permissions have been granted and Windows User Account Control (UAC) restrictions have been addressed.

When creating a new project in TFS, you may also have the same problem accessing the Report server when the TFS 2010 new project wizard attempts to create a new reporting site. You may even be able to access the site remotely but just not on the local box.
To fix the problem, you can just go to Internet Explorer>Tools>Options>Trusted Sites>Sites and add the site experiencing the problems to the Trusted Sites zone.
Full details are here (Point 7 and 8):
http://support.microsoft.com/kb/934164

Note that you will need to restart Visual Studio for it to pick up the above changes in the TFS 2010 new project wizard.

DDK

Monday, 2 May 2011

Integrating TFS 2010 with SharePoint 2010 using SharePoint 2010 Claims Based Authentication Mode

I recently attempted to create a TFS 2010 portal against a remote portal in SharePoint 2010. However, if you attempt to create this TFS 2010 Remote portal against a claims-based authenticated web application, you will typically get the following exception:

TF218017: A Sharepoint Site could not be created for use as the team project portal. The following error ocurred: Server was unable to process request. --> The User does not exists or is not unique --> The User does not exists or is not unique.The problem is the same as the one experienced here:
http://social.msdn.microsoft.com/Forums/en-US/tfsgeneral/thread/bd396f15-02d7-4431-9d75-005c8b63007e/


The error related to "Unique Users" so I had a look in the SharePoint Content database to see if there were duplicate users with the following query:


select * from userinfo

Indeed there are two active users with my loginid for the one web application. Both were active.
One loging was named (in the tp_Login column of the table) as you'd expect as:

DOMAINNAME\LoginName
Whereas the other was named with an unusual prefix "i:0#.w" like so:

i:0#.w|DOMAINNAME\LoginName
 
However, this didn't present a problem when creating sites within the site collection through the normal SharePoint 2010 user interface. I created sites with both my account, the setup account and service account - there were no errors releated to uninque users at all.

While this "i:0#.w" prefix on logins is not documented, it identifies a login as using claims-based authentication. Thing is that it doesn't make sense that there would be both claims-based and classic users both active for the one site collection - unless that behaviour is by design for backwards compatability. My theory - on the TFS side, the SharePoint object model is unexpectedly (from the TFS 2010 perspective) returning 2 accounts against the one login.

I used Wireshark and Fiddler to trace the calls that TFS Team explorer is trying to make when creating project sites, and it it was making a call to https://servername/_vti_bin/TeamFoundationIntegrationService.asmx. It is making a call to the "CheckUrl" method. An internal error (500) is thrown by the webservice - as per the screenshot of the Fiddler trace:

Upon further investigation, the physical webservice is in the following location on the SharePoint server:
%Program Files%\Common Files\Microsoft Shared\Web Server Extensions\14\ISAPI\TeamFoundationIntegrationService.asmx

Looking inside the asmx, it points to the Microsoft.TeamFoundation.SharePoint.TeamFoundationIntegrationService class in the
Microsoft.TeamFoundation.SharePoint.dll assembly.

The SharePoint 2010 ULS Logs have only very basic information reiterating the error in Team Explorer - "The user does not exist or is not unique".
Looking for the CheckUrl method definition in Reflector, it runs many checks using the Office 12 object model - but nothing clearly erroneous in the code warranting further investigation. We could run something like the Visual Studio.NET plugin for reflector to step through the code - but I didn't think it was worthwhile continuing investigation into the problematic calls in the TFS object model.
I had another search and according to "Chris Co" from Microsoft (http://social.msdn.microsoft.com/Forums/en-US/tfssetup/thread/a2665904-ec35-4dff-a809-69c2c9316378) "This is something the product group is thinking about supporting in the future, but unfortunately, we currently do not support and will not be supported in SP1."

So for now, if you want your TFS Portal to be hosted in SharePoint 2010, your SharePoint 2010 web application will need to use Classic rather than claims authentication.
DDK

Wednesday, 20 April 2011

Custom Extraction Rules for Visual Studio Load Tests - Dealing with Dynamic Control Identifiers

I was recently commissioned to create and fix up several Visual Studio 2008 Load Tests for one of my clients. There were some questions raised about the validity and accuracy of the load test results - so I was brought in to do a "Load Test Audit".

I started by creating some data validation scripts which confirmed that data was being updated correctly. Unexpectedly, even though the tests were running without failure, the number of records affected in the database was completely different to the anticipated results.

Upon investigation, I found several issues which caused the Load tests to intermittently or "silently" fail (i.e. tests passed, but exceptions were logged in the backend of the custom application). There were several problems I found. I cover two of them below:

PROBLEM 1:
One of the problems is that one of the buttons in the recorded webtest would sometimes not be correctly triggered. Consequently all subsequent web test requests in that Load Test scenario would fail.

Turns out, there was a link button in one of the ASP.NET Web Parts rendered as part of a GridView - and the id of the control would actually change based on the number of records included in the grid.

Consequently, in most cases there was 1 record in the grid so the "lnkAdd" button was "lnkAdd1". This matched what was recorded in the Visual Studio Webtest. However, the load test would fail when there were a larger number of records as Visual Studio couldn't find the "lnkAddX" button.

To resolve this, the simplest way to dynamically determine the id of the dynamic control was to use a custom extraction rule that sets a context value for use in subsequent requests. Like so:
public class GetAddButtonId : ExtractionRule
    {
        /// 
        /// Add button name. TODO: Add defensive code
        /// 
        /// /// public override void Extract(object sender, ExtractionEventArgs e)
        {
            e.Success = true;
            if (!string.IsNullOrEmpty(e.Response.BodyString))
            {
                Match value = Regex.Match(e.Response.BodyString, @"ctl00\$ContentPlaceHolder1.[^>]*?grdScheduleChangeResults\$.[^>]*?\$lnkAdd");
                if (value.Success)
                {
                    this.ContextParameterName = "AddButtonId";
                    if (!e.WebTest.Context.ContainsKey("AddButtonId"))
                    {
                        e.WebTest.Context.Add("AddButtonId", value.Groups[0].Value);
                    }
                    else
                    {
                        e.WebTest.Context["AddButtonId"] = value.Groups[0].Value;
                    }
                }
            }
        }
    }

I added this Custom Extraction Rule as part of the web test, updated subsequent requests to use the Context value for the control name - and the issues were resolved.

PROBLEM 2:
There were also issues with the Validation rules in the Load Test - as the tests were running without failure, but they were actually just hitting the CustomError.aspx page (this occurs when there's an exception). I had to add a new Web Test Validation rules as below:

public override IEnumerator GetRequestEnumerator()
        {
            if ((this.Context.ValidationLevel >= Microsoft.VisualStudio.TestTools.WebTesting.ValidationLevel.High))
            {
                if (!Context.ContainsKey("IgnoreErrors") || (Context["IgnoreErrors"].ToString() != null && bool.Parse(Context["IgnoreErrors"].ToString())))
                {
                    ValidationRuleFindText validationRule2 = new ValidationRuleFindText();
                    validationRule2.FindText = "Unable to perform operation";
                    validationRule2.IgnoreCase = true;
                    validationRule2.UseRegularExpression = false;
                    validationRule2.PassIfTextFound = false;
                    this.ValidateResponse += new EventHandler(validationRule2.Validate);

                    
                    ValidationRuleFindText ErrorSummary_Validation = new ValidationRuleFindText();
                    ErrorSummary_Validation.FindText = "ErrorSummary";
                    ErrorSummary_Validation.IgnoreCase = true;
                    ErrorSummary_Validation.UseRegularExpression = false;
                    ErrorSummary_Validation.PassIfTextFound = false;
                    this.ValidateResponse += new EventHandler(ErrorSummary_Validation.Validate);

                    //'An open change request already exists

                    //Wasn't detecting custom error page as issue in test.
                    ValidationRuleFindText customError_Validation = new ValidationRuleFindText();
                    customError_Validation.FindText = "CustomError.aspx";
                    customError_Validation.IgnoreCase = true;
                    customError_Validation.UseRegularExpression = false;
                    customError_Validation.PassIfTextFound = false;
                    this.ValidateResponse += new EventHandler(customError_Validation.Validate);

                    ValidationRuleFindText callback = new ValidationRuleFindText();
                    callback.FindText = "Invalid postback or callback argument";
                    callback.IgnoreCase = true;
                    callback.UseRegularExpression = false;
                    callback.PassIfTextFound = false;
                    this.ValidateResponse += new EventHandler(callback.Validate);

                    this.StopOnError = true;
                }
            }
            return null;
        }
Hope this helps someone in the future when they are troubleshooting their Visual Studio Web Tests or Load Tests!

DDK

Friday, 8 April 2011

DDK's Guide to the Estimation of IT Projects

As an Architect and the .NET Principal at Oakton NSW, I have to do my fair share of Estimates and Proposals. I am also often asked to review and revise other peoples estimates - to "Quality Stamp" them so to speak. There are some common things that I pick up on - hence the driver behind this blog post.

Summary Diagram of the DDK Estimation Technique:



Here are some of the important things you should consider when developing estimates:

1. Estimate from the bottom up rather than from the top down. The focus and detail of this approach helps you to substantiate your estimates to others and show you've used due diligence in arriving at your estimate. A detailed function point analysis (FPA) is ideal when trying to minimise risk as much as possible (especially for fixed cost projects).
2. There are 2 critical variables which can make a project take much longer than expected and estimated. If you have these components, you need to increase your estimate to more than you expect:
  a. The larger and more complex the project, the more likely it is to take longer than expected.
  b. The more new technologies or new techniques involved, the more likely it is to take longer than expected

3. Communicate with and update the client regularly – Don’t be afraid to re-estimate your tasks and let the client know if things will take longer or shorter than expected. The earlier they know, the earlier corrective action can be taken.

4. Larger projects are harder to estimate. Only estimate small components of a project if possible – don’t estimate all releases. Deliver and estimate in increments if the client/contracts allow.

5. Make sure you consider the following components in your Estimation Checklist before giving it to the client:
  a. End User Documentation
  b. System Documentation
    i. Rollback Plans
    ii. Non Functional Requirements (NFR)
    iii. Technical Design & Specifications
    iv. Functional Design & Specifications
    v. Establishing Metrics (e.g. what performance is expected on what servers and with what data load)
    vi. Test Plans
    vii. Test Scripts
  c. Testing
  d. Training and Change Management (you can't just give someone an application and expect them to start using it effectively!)
  e. User Acceptance Testing
  f. Integration Testing (esp when integrating with Legacy Systems)
  g. Deployment Activities and Productionizing Systems
    i. Especially with complex deployments. These deployment activities are typically ongoing rather than once off.
  h. Meetings Drag Factors such as regular Meetings and Discussions. Team leaders and architects need a drag factor SCRUM meetings and requirement gathering meetings.
    i. Triage Meetings
    ii. Code Reviews
    iii. Level 2 Reviews by Testers e.g. For one of our projects, it took an average of 45 minutes per TFS work item/ticket.
  i. Focus Groups
  j. Licenses
  k. Configuration
    i. Third Party Components
    ii. Firewall Configuration
    iii. Database Configuration
  l. Data Migration
  m. Handovers (Including Warranty Periods)
  n. You need to anticipate who is developing it.
    i. Offshore Models/Engagements – have to spend 2-3 times the effort in non-development activities such as coordination, “hand-holding” and quality assurance when your team is overseas. You also need to spend much more time on specification documents to avoid communication issues.
  o. Temper your Optimism by considering different scenarios
    i. Best Case
    ii. Worst Case
    iii. Likely Case Scenario
  p. If uncertainty on a project is High, add an uncertainty multiplier to your estimate
  q. Don’t estimate more than 8 hours per day.
  r. Budget time for Performance Testing
    i. Time for Load Tests

6. If possible, do a Proof of Concept (PoC) before providing the estimates (or just estimate the PoC) – especially when using new combinations of technologies.

7. Learn from Historical Data. Try to learn from similar projects and how the estimates compared with the actuals. Use your company portal to discover estimates and ask other people in your company for similar estimates that they did.

8. Sanity Check your estimates. i.e. Have your estimates peer-reviewed to help you ensure consistency and coverage in your estimates.

9. Cross-Check your estimates. If you can convince all stakeholders that the estimate is valid and establish buy-in to that estimate, you have the basis of a good estimate.

10. Don’t underestimate Non-Programming/Infrastructure activities.

11. Don’t change estimates if possible if they are based on a solid agreement or understanding. Instead, try and change the commercial arrangements surrounding the estimate. E.g. don’t change the estimates unless they are proven unreasonable – change the rate if possible.

12. All estimates are guesses – try and reduce the uncertainty – but you cannot remove uncertainty completely.

13. Try to make your estimates from a fully informed standpoint - the same as a General shouldn't make strategic decisions in the fog of war. Request and Review as many materials (including scribbled diagrams and requirements documents) as time allows. The more you discuss your understanding of the project and talk to the end users, the more likely you are to make an informed estimate. Again, this allows you to back up your estimates to all stakeholders. If there are factors in the project that are particularly unclear or uncertain, add an "uncertainty" multiplier on the item. Encourage the client to help you clear up this uncertainty if possible.

14. Use the right tools. Microsoft Project is a good start. Learn how to use it properly - plus it can then create and update developer work items for you in Microsoft Team Foundation Server (TFS)  when you are ready to start work.

15. Don't forget that external dependencies (e.g. a 3rd party is creating web services for you, waiting on documentation) will slow the project down. Ideally start the project when work you depend on is complete - otherwise you'll need to factor downtime into your budget, estimates (an expected "downtime" item) and estimate assumptions.

Any thoughts or comments on this guide and checklist are welcome - I will update the list based on feedback.

DDK

Thursday, 7 April 2011

I'm now a SharePoint 2010 Microsoft Certified Professional Developer (MCPD)

I recently passed my 2nd SharePoint 2010 exam (70-573 and now 70-576) - so I'm now officially a SharePoint 2010 MCPD!



Next step is MCITP (covering the Infrastructure side of the SharePoint equation) and then onto the coveted Microsoft Certified Master (MCM) certification.

The MCM is apparently a challenge to get (involving panel interviews by other MCMs and 3 weeks of training) - and there is only one so far in Australia.

DDK

Friday, 18 March 2011

Using the Fiddler Tool Proxy to Debug Visual Studio Web Tests and Load Tests

The Fiddler Tool (http://www.fiddlertool.com/) only listens to WinInet traffic by default - so it doesn't normally pick up any traffic which comes from your Visual Studio Web Tests or Load Tests - even when capturing is on. To help debug your Visual Studio tests, you can set the proxy manually within your code so it forces the traffic through the Fiddler proxy:

//For Fiddler Debugger
this.Proxy = "http://localhost:8888";
WebProxy webProxy = (WebProxy)this.WebProxy;
webProxy.BypassProxyOnLocal = false;

Note that you should have the System.Net namespace in your usings/Imports statements for the WebProxy class. This also assumes Fiddler is running and is set up to use the default proxy port of 8888.

DDK

Thursday, 24 February 2011

TFS 2008 Build Service will not start, with exception: "The underlying connection was closed: An unexpected error occurred on a receive. (type WebException)"

Yesterday, the build agent on the TFS 2008 of our client suddently stopped working after a reboot. I was assured that No settings had changed on the server, so it was a bewildering problem. Several errors started to occur in the event logs such as "Detailed Message: TF224002: An unexpected error has occurred. Exception Message: The underlying connection was closed: An unexpected error occurred on a receive. (type WebException)".
As a symptom of the problem, the TFS 2008 Build Agents would never initialize and would constantly go to "Unreachable" status on the server.

The full exception in the event log was as follows:

TF53010: The following error has occurred in a Team Foundation component or extension:

Date (UTC): 23/02/2011 11:06:06 PM
Machine: TFS1-MYSERVER
Application Domain: TFSBuildService.exe
Assembly: TfsBuildService, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v2.0.50727
Process Details:
  Process Name: TFSBuildService
  Process Id: 6256
  Thread Id: 5132
  Account name: MYDOMAIN\TFSService

Detailed Message: TF224002: An unexpected error has occurred.
Exception Message: The underlying connection was closed: An unexpected error occurred on a receive. (type WebException)

Exception Stack Trace:    at System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest request)
   at System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(WebRequest request)
   at Microsoft.TeamFoundation.Client.TeamFoundationSoapProxy.GetWebResponse(WebRequest request)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
   at Microsoft.TeamFoundation.Proxy.BisRegistrationServiceProxyWsdl.GetRegistrationEntries(String toolId)
   at Microsoft.TeamFoundation.Proxy.BisRegistrationProxy.GetRegistrationEntries(String toolId)
   at Microsoft.TeamFoundation.Proxy.BisRegistrationService.RefreshMemoryCache()
   at Microsoft.TeamFoundation.Proxy.BisRegistrationService.RefreshCachesIfNeeded(Boolean direct)
   at Microsoft.TeamFoundation.Proxy.BisRegistrationService.GetRegistrationEntries(String toolId)
   at Microsoft.TeamFoundation.Build.Client.BuildServer.Microsoft.TeamFoundation.Client.ITeamFoundationServerObject.Initialize(TeamFoundationServer tfs)
   at Microsoft.TeamFoundation.Client.TeamFoundationServer.CreateITFSObjectInstance(Assembly assembly, String fullName)
   at Microsoft.TeamFoundation.Client.TeamFoundationServer.GetService(Type serviceType)
   at Microsoft.TeamFoundation.Build.Agent.AgentService.InitializeTeamFoundationServer(String callingAT)
   at Microsoft.TeamFoundation.Build.Agent.AgentService.GetBuildInProgress(String tfsUrl, String teamProject)
   at SyncInvokeGetBuildInProgress(Object , Object[] , Object[] )
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage3(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage2(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage1(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)

Inner Exception Details:

Exception Message: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. (type IOException)

Exception Stack Trace:    at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.PooledStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetrievedStream, Boolean probeRead)
Inner Exception Details:
Exception Message: An existing connection was forcibly closed by the remote host (type SocketException)
Exception Stack Trace:    at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)

The problem in our case was that we had (apparently) new problems with our proxy configuration in Internet Explorer (IE). Fiddler had been installed on this machine several months prior to me arriving at the client and it stopped working. I removed all references to Fiddler (port 8888 by default) in the proxy configuration and the build started to work correctly.

So this is an important reminder that - the Build Agent in TFS 2008 requires local HTTP access to resources. It uses IE proxy settings during every build run. If these are not correct, TFS will cause you absolute grief and fall over left,right and centre.

Now to the Fiddler problem...

DDK