How to force a SharePoint 2010 Business Connectivity Services (BCS) Client Cache to rebuild

WARNING! This tip involves potential loss of data.
The Business Connectivity Services (BCS) Client Cache is a combination of PST files and a SQL Server Compact Edition database - that allows for you to operate with Business data while in disconnected or limited-connection scenarios.

It contains all the subscription information and data for any items that you have made available offline via Business Connectivity Services. However, there is no official way of flushing this cache - your BusinessDataCache.sdf file will just keep growing and growing ad infinitum. You can query this data by connecting to it via Visual Studio 2010 via the Server Explorer/Data Connections task pane.(For reference, the Schema of the BCS Client Business Data Cache is shown below)

At the risk of data loss (if you have any unsynched items), there is a way of recreating your cache without deleting your whole Windows Logon profile. This is as follows:

1. Uninstall any Office Add-ins you have e.g. Outlook Addins that use the BCS cache or create subscriptions in the BCS cache.
2. Kill the BCSSync.exe process in memory using Windows task manager.
3. Go to the following folder on the machine:
   %userprofile%\AppData\Local\Microsoft\BCS
4. Rename BusinessDataCache.sdf to zzBusinessDataCache.sdf
5. Run "%ProgramFiles%\Microsoft Office\Office14\BCSSync.exe" /Restart /Activation /RestartApps to restart the BCS Synchronization service.
6. Your BusinessDataCache.sdf  file will be regenerated.

DDK

How to Troubleshoot Business Connectivity Services (BCS) in SharePoint 2010 using Performance Monitor Data Collector Sets

When dealing with Business Connectivity Services (BCS) errors (especially if they are occurring on Offline clients such as Outlook), the Event Log and SharePoint ULS Logs don't give you the whole picture.

To get a full verbose breakdown as to why your configuration or code is generating errors, you need to use Windows Performance Monitor (perfmon) to generate a verbose trace to get the full details and stack traces that are often needed to diagnose issues.

To set up an Event Trace on BCS you can do the following:

1. Start Up Performance Monitor by typing "perfmon" at the Windows Start Prompt.
   
   
2. In Windows Performance Monitor, Expand the "User Defined Node" and then Right-Click and Choose "New Collector Set"
   
   
3. In the First Screen of the Wizard, Name the Collector Set (e.g. BCSCollectorSet). Choose the "Create Manually (Advanced)" radio button.
4. When asked "What type of data do you want to include?", choose "Event Trace Data"
   
   
5. When asked "Which event trace providers would you like to enable?", select the "Microsoft-Office-Business Connectivity Services" and the "Microsoft-SharePoint-Products-Business Connectivity Services" providers. If you are using Duet Enterprise, you may also want to add the "Duet Enterprise" Event Provider.
   
   
6. Click "Finish" to Leave the other settings at the defaults.
7. Start your trace by clicking on your collector set and clicking the play icon.
   
8. Once the trace is recorded, you can then open up the logs and view them in Windows Event Viewer by clicking on "Open Saved Log" and pointing to your newly created BCS Event trace file.
   
   

If found this particularly useful when troubleshooting BCS Offline Caching issues on Outlook clients.

The event trace indicated that an old version of my BCS models and connections was being used to the Duet/SAP system.

To resolve my problem, I had to remove all instances of the BCS lists in Outlook by uninstalling Data Lists and Solutions that refer to that one SharePoint 2010 External Content Type. This resolved my caching problems and it stopped giving me "Access Denied" Exceptions when Synching with Outlook 2010.

 

DDK

Fix for "The service was unable to start because the version of the database does not match the version of the product." in ForeFront Identity Manager Sychronization Service for SharePoint 2010

I had an issue today with the ForeFront Identity Manager Synchronization Service when Synching to Active Directory and SAP in the User Profile Store. It was generating the following exception when I attempted to run a full or incremental User Profile synchronization:

  

"The service was unable to start because the version of the database does not match the version of the product."

 

If you get this problem, you will need to reprovision the User Profile Synchronization Service Application by stopping and restarting it (and reentering the password for your syncrhonization service user). This resolved the issues I was having - and the ForeFront Identity Manager service could then start correctly. The steps to do this are as follows:

 

1.Central Admin>System Settings>Manage Services on Server (select the server where the User Profile Service is running)

2.Stop the User Profile Service

3.Stop the User Profile Synchronization Service (you will be prompted that this will deprovision the service)

4.Once the services have stopped, Start the User Profile Service again

5.Start the User Profile Synchronization Service again (you will be prompted to enter the password for the User Profile Service's svc account) -note1: this service can take a little while to restart; note2: if it does not restart successfully, restart the server and try again (this has worked for me)

6.Once complete, the User Profile Service and User Profile Synchronization service should show as Started, and the 2 corresponding FIM services on the server should be running again.

For Reference:
I've mentioned this in the blog previously, but you can start up the ForeFront Identity Manager UI from the following location:

"%ProgramFiles%\Microsoft Office Servers\14.0\Synchronization Service\UIShell\miisclient.exe"

DDK

SharePoint 2010 Business Connectivity Services (BCS) Local Cache Data

When you take a SharePoint 2010 list offline (e.g. into SharePoint Workspace or Outlook), BCS entities are created in your local drive. These are often supported by ClickOnce Office Solutions deployed via Visual Studio Tools for Office 4.0 (VSTO). For reference, the following locations are used for BCS data and ClickOnce installations:

BCS Client Cache Data files are located in a hidden folder at:
%USERPROFILE%\AppData\Local\Microsoft\BCS

Supporting ClickOnce Applications are located at:
%USERPROFILE%\AppData\Local\Apps

DDK

Fix for VSTO ClickOnce application - "Unable to install this application because an application with the same identity is already installed. To install this application, either modify the manifest version for this application or uninstall the preexisting application"

If you receive the following exception even though your VSTO ClickOnce application is already uninstalled:

"Unable to install this application because an application with the same identity is already installed. To install this application, either modify the manifest version for this application or uninstall the preexisting application."


Then you may have issues with your ClickOnce application Cache. You have 2 options for clearing it:

Running the following at a Visual Studio Command Prompt:
mage -cc

OR

Running the following at a command line on any machine with the problem:
rundll32 dfshim CleanOnlineAppCache


See http://vijayvepa.wordpress.com/2011/01/11/unable-to-install-this-application-because-an-application-with-the-same-identity-is-already-installed-to-install-this-application-either-modify-the-manifest-version-for-this-application-or-uninstall/ for more details.

DDK

Business Benefits of Duet Enterprise and Comparison to SAP integration via Biztalk/Custom Solutions - Bringing SAP Business Data and SharePoint Collaboration Features Together and Ensuring a ROI

I've been involved in several SAP-SharePoint integration projects recently. Such projects typically relied heavily on an Enterprise Service Bus (ESBs - such as Biztalk or SAP PI), customizations and code. When you integrate SAP with SharePoint, there are many benefits that are provided when users don't have to use SAP GUI directly:

• Hard benefits such as reduced licensing costs
• Usability benefits such as allowing for a much simplified User Interface (people familiar with SAP GUI and SAP Portal will chuckle at this)
• Supportability Benefits (as SharePoint and .NET skills are more common that SAP Portal skills - at least in the Australian market).

However, since the release of Duet Enterprise 1.0 (http://ddkonline.blogspot.com/2011/01/duet-enterprise-released-and-available.html) at the end of January, another integration option has been thrown into the mix.

When my clients have been considering Duet Enterprise, one of the most often asked questions is

"What does Duet Enterprise offer me that I can't do already with SharePoint, SAP, custom BAPIs, SAP Enterprise Services and SharePoint 2010 Business Connectivity Services?".

Duet Enterprise brings a lot to the table - both on the SAP side and on the SharePoint side - however it's not the perfect solution for all problems. While consistency between systems is a vital principle in enterprise architecture, you really do have to choose the right combination of tools to fit a given problem. Indeed, as with any architectural approach - you have to weigh up the pros and cons.

I see the Duet Enterprise product offering (and the development framework it provides) as having the following advantages over just a custom integration build:

1. Security and Security Best Practices. Through the Duet Enterprise Claims-based Role Provider, it allows you to leverage the SAP security model and SAP Roles in SharePoint. This avoids the duplication of a potentially massive security hierarchy into the SharePoint space. By design, communication channels between systems are encrypted. This is one of the biggest value-adds. This also allows you to keep all your unstructured data (e.g. pdfs/documents) in SharePoint and still leverage the security model in SAP to limit access to these resources.
2. SAP UWL Functionality - It exposes your SAP Universal Worklist (UWL) into the native SharePoint task list - you don't have to do a mashup or WSRP or some other IFrame hack.
3. Maintainability and Reduced Development Efforts -  It provides some out of the box functionality (site creation, business centric collaboration) as a guideline and template for further development.
4. Diagnostic Tool Support - Duet Provides Health Checks that allow you to check your setup, and provides mechanisms (via Correlation Identifiters) to help you diagnose problems if requests are failing. 
5. Official Supportability - It is a product supported by Microsoft and SAP that incorporates their recommended approach for integration. There is also official documentation for this integration approach.
6. It has a Roadmap - It is a platform that has a roadmap (as opposed to a completely custom system) which is something you build and have to take a lot more responsibility for. As Duet Enterprise grows in capabilities, you will regret  going down the 100% custom path.
7. Reporting Functionality - it brings Reporting Functionality out of SAP via the Reporting Request mechanisms and alows you and your colleagues to collaboration around SAP Reports via standard SharePoint functionality (e.g. Social tagging)  
8. The BCS Solution Design Gallery - It provides an enhanced model for the deployment of Office Based Solutions through SharePoint e.g. Administration screens for arranging your solution and for generating your solution from within SharePoint. See http://msdn.microsoft.com/en-us/library/ff963717.aspx for more details.
9. Entity Collaboration Functionality - Supports creation of collaboration sites surrounding SAP business entities. This however has to be checked against your SharePoint governance strategy and capacity plan.
10. SAP HR Information imported into SharePoint User Profiles - ForeFront Identity Manager is used synchronize SAP User information (HR infotypes) from SAP via a BCS user profile connection, along with role synchronization via the Duet Enterprise Profile Synchronization timer job.

I have also been approached around concerns that it Duet Enterprise doesn't use a full ESB. However that's not true. Duet is a mechanism for providing services on the SAP side (via an ESB on the SAP side e.g through SAP's middleware solution "SAP PI") - and consuming these services on the SharePoint side. Duet Enterprise can live comfortably in an ESB-enabled an ecosystem. 

If you have SharePoint and SAP in your environment, I suggest that you consider Duet Enterprise as part of your enterprise strategy for improving the accessibility and collaboration around your most important business data.

In most circumstances, it will give you a large amount of momentum (and provides all the plumbing) to allow you to provide a solid and supported solution for SAP to SharePoint integration - with an upgrade path.

DDK

SharePoint 2010 Business Connectivity Services (BCS) - Exposing External Lists through Outlook and the InfoPath External Data Part - Fix for "Catastrophic Failure"

Business Connectivity Services (BCS) along with BCS solutions allow you to expose your line of business data (e.g. SAP via Duet Enterprise) through Office client applications such as Outlook 2010.

There are several walkthoughs of generating these solutions available online. Some examples of these are:

• "Creating Intermediate Declarative Outlook Solutions Using Business Connectivity Services" (http://msdn.microsoft.com/en-us/library/ee819909.aspx)
• "How To: Create an Outlook Solution Using Duet Enterprise Outlook Application Designer"
   - http://msdn.microsoft.com/en-us/library/ff963717.aspx (Though this is specific to the Duet Enterprise Deployment mechanisms, many of the same steps apply).

I recently had issues with deployment of a Business Connectivity Services (BCS) solution in conjunction with Duet Enterprise 1.0. When my Duet Enterprise solution was deployed , I was being greeted with an exception as per the screenshot below, with a "Catastrophic failure" error message in place of where my custom task pane should show.

The main reason this exception was occuring was because my layout file in my solution (the xsd and schema explanation for BCS Solution layout files can be found here: http://msdn.microsoft.com/en-us/library/ff394500.aspx) had a "Width" custom property.

Outlook couldn't handle this as a parameter to the web part - so it crashed out in a "Catastrophic" manner.

Also note that for your task pane to have any datasource at all whatsoever, you still need to have a "DataSourceName" property set on your OBA part. The description on MSDN here (http://msdn.microsoft.com/en-us/library/ff394500.aspx) is somewhat misleading as it indicates that you only need a datasource for your External Data Part if it relies on data from one of the other parts in your layout. If you remove the datasource property, the attributes weren't populated - so the statement on MSDN isn't correct.

Fix for SQL 2008 Exception - An exception occurred while enqueueing a message in the target queue. Error: 15404, State: 19. Could not obtain information about Windows NT group/user 'DOMAIN\username', error code 0x5

After installing a recent series of Windows update, I began to get hundreds of SQL 2008 errors in the Windows Event log (20 or so per second) on one of our demonstration servers. These were as follows:

An exception occurred while enqueueing a message in the target queue. Error: 15404, State: 19. Could not obtain information about Windows NT group/user 'DOMAIN\username', error code 0x5.

or

An exception occurred while enqueueing a message in the target queue. Error: 15404, State: 19. Could not obtain information about Windows NT group/user 'DOMAIN\username', error code 0x2.

If you do have these exceptions, then you most likely have connectivity issues to your Active Directory server - and the identity of the owner of your databases  (dbo) is on that Active directory server. The SQL Service Broker is causing this exception. If this is not a resolvable issue condition, then you can change the owner to a SQL login (e.g. sa) and this will resolve the exception.

You can do this with the following command on each database that has the wrong owner:

sp_changedbowner 'sa'

DDK

SharePoint 2010 - Business Connectivity Services (BCS) / BDC model export - Default vs Client Option

When exporting BCS entities, you may have noticed 2 different "Settings" options available in the export dialog. If you're wondering what the differences are in the output - there don't appear to be any at all.

To confirm this, I just did an export of 2 entities to a model file using the "Default" option and the other export using the "Client" option. Using the Notepad++ text compare tool, there was no physical difference between the output of the different export settings - nor does there seem to be any documentation regarding this dropdown at all.

However, if you look at the BCS connection information, you'll also notice 2 tabs that match the different settings or "profiles" that you can choose. Consequently, this is expected behaviour - as you'll only have differences when your settings are different between default and client.

DDK

Not all Reflection Tools (or Obfuscators) Are Created Equal

I had a client last week in Melbourne who wanted to salvage some code from an existing SharePoint 2010 implementation. As long as it wasn't obfuscated, then I thought there would be no problems at all.

Red Gate's (previously Lutz Roeder's) Reflector is designed for just a situation - and I'd recently purchased the awesome VS PRO version (which is phenomenal lets you step and debug through other people's applications!).

However, when I tried to open up the assemblies in Reflector or on ILDASM, it appeared to indeed be obfuscated - by the Smart Assembly tool from Red Gate.

Typically, the obfuscated code will be shown with an error or garbled characters. e.g. "This item is obfuscated and can not be translated." - as below:

If you try and open it in ILDASM, it throws an exception as below:

This is because the Assembly has the CompilerServices "SuppressIldasmAttribute" applied to it.

However, if you open up the assembly with the new tool JetBrains dotPeek (the makers of Resharper), then you will be able to see the source code - even of those allegedly obfuscated methods and properties.

I'm not sure whether Red Gate deliberately set a flag inside Reflector when they purchased it from Lutz Roeder - but it seems like a few shortcuts were taken with the obfuscation engine.

So be warned - not all Reflectors and not all Obfuscation methods are created equal.


DDK