Duet Enterprise Integration - Fix for "System.ArgumentException: An item with the same key has already been added due to Extra Content Type Headers" in SharePoint 2010 ULS Logs

If you're not aware, Duet Enterprise (http://www.duet.com/) is an integration framework and infrastructure for providing SAP to SharePoint integration - of task lists, SAP workflows, SAP reports, SAP security and SAP data (through SharePoint BCS).

I recently had an issue with a Duet Enterprise deployment for one of my clients. When retrieving data, the out of the box web parts were failing and dropping the following Exception in the SharePoint 2010 ULS Logs:

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ArgumentException: An item with the same key has already been added. Server stack trace:
at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
at System.Xml.ContentTypeHeader.ParseValue()
at System.Xml.ContentTypeHeader.get_MediaType()
at System.Xml.XmlMtomReader.ReadRootContentTypeHeader(ContentTypeHeader header, Encoding[] expectedEncodings, String expectedType)
at System.Xml.XmlMtomReader.Initialize(Stream stream, String contentType, XmlDictionaryReaderQuotas quotas, Int32 maxBufferSize)
at System.Xml.XmlMtomReader.SetInput(Stream stream, Encoding[] encodings, String contentType, XmlDictionaryReaderQuotas quotas, Int32 maxBufferSize, OnXmlDictionaryReaderClose onClose)
at System.Xml.XmlDictionaryReader.CreateMtomReader(Byte[] buffer, Int32 offset, Int32 count, Encoding[] encodings, String contentType, XmlDictionaryReaderQuotas quotas, Int32 maxBufferSize, OnXmlDictionaryReaderClose onClose)
at System.ServiceModel.Channels.MtomMessageEncoder.MtomBufferedMessageData.TakeXmlReader()
at System.ServiceModel.Channels.BufferedMessageData.GetMessageReader()
at System.ServiceModel.Channels.BufferedMessage..ctor(IBufferedMessageData messageData, RecycledMessageState recycledMessageState, Boolean[] understoodHeaders)
at System.ServiceModel.Channels.MtomMessageEncoder.ReadMessage(ArraySegment`1 buffer, BufferManager bufferManager, String contentType)
at System.ServiceModel.Channels.HttpInput.DecodeBufferedMessage(ArraySegment`1 buffer, Stream inputStream)
at System.ServiceModel.Channels.HttpInput.ReadBufferedMessage(Stream inputStream)
at System.ServiceModel.Channels.HttpInput.ParseIncomingMessage(Exception& requestException)
at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown
at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at BCSServiceProxy.IWXManageCustomerIn.FindCustomerByElements(FindCustomerByElementsRequest request)
at BCSServiceProxy.WXManageCustomerInClient.BCSServiceProxy.IWXManageCustomerIn.FindCustomerByElements(FindCustomerByElementsRequest request)
at BCSServiceProxy.WXManageCustomerInClient.FindCustomerByElements(BPCCustGetAll BPCCustGetAll) -


-- End of inner exception stack trace ---
at System.RuntimeMethodHandle._InvokeMethodFast(Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
at Microsoft.SharePoint.BusinessData.SystemSpecific.Wcf.WcfSystemUtility.Execute(Object[] args)

I immediately thought that it was something wrong with the data. Because Duet uses HTTPS & SSL for all communications between SAP Netweaver 7.02+ and SharePoint 2010, I had to decrypt the traffic to work out what was going on. Running the calls on the SAP site returned blank results. To get WireShark (http://www.wireshark.org/) decrypting the traffic we had to generate a certificate in SAP with a private key. We had to get the key out in PKCS#12 format and then convet it to the .PEM format that wireshark accepts - similar to what is desribed here:

http://htluo.blogspot.com/2009/01/decrypt-https-traffic-with-wireshark.html
We used "sapgenpse" in SAP to generate the certificate with the private key - as per the screenshot below:

Once the key was set in WireShark preferences, you will see many of the packets being processed being marked as green. You can then right click on them and choose "Follow SSL Stream" on your web request - and you will see the decrypted version of your HTTPS traffic.

Turns out, the issue was with SAP retuning duplicate Content headers in the response to SharePoint BCS (as per the screenshot).

There is a SAP Note to fix this very issue - "SAP Note 1539888 - Composite Note -Duet Enterprise Installation Wizard". You will need a SAP Service Market place to log on to get this note.

DDK

Duet Enterprise - [ERROR] - "Cannot Merge Contents of LobSystem (External System) with Name 'Account' as it appears to be different from preexisting LobSystem in the current load context." - Fix

When using Duet Enterprise (for SAP and SharePoint 2010 Integration) you may receive the following exception when importing Duet Business Connectivity Service Models (as per http://technet.microsoft.com/en-us/library/gg394164.aspx):

"Duet Enterprise - [ERROR] - Cannot Merge Contents of LobSystem (External System) with Name 'Account' as it appears to be different from preexisting LobSystem in the current load context.".

This may occur even after you've used Duet to remove all of the models with the commandline DuetConfig.exe /RemoveBDC.

You can fix this issue by going to:

This may occur even after you've used Duet to remove all of the models with the commandline DuetConfig.exe /RemoveBDC.

SharePoint Central Administration - Your Business Connectivity Services (BCS) Service Application - Manage
Then:

1) Choose "Models" in the "View" dropdown and remove any Duet models if they still exist

2) Choose "External Systems" and Remove any Duet External systems (this is the external system referred to in the error).

You should now successfully be able to import the Duet Enterprise 1.0 out of the box Models.

DDK

SharePoint 2010 Business Connectivity Services (BCS) - Exception when adding new Entities - "Null was returned to FindSpecific (Read Item operation)" - Fix

Scenario: You've set up SharePoint 2010 External Content Types and you receive the following exception when adding a new item to your BCS list:

Form submission failed. (User: DOMAIN\UserName, Form Name: Template, IP: , Request: http://servername/sites/Duet/Lists/Leave Request/Item/newifs.aspx?RootFolder=/sites/Duet/Lists/Leave Request&IsDlg=1#, Form ID: urn:schemas-microsoft-com:office:infopath:entity:-AutoGen-2011-06-08T05:13:33:61Z, Type: DataAdapterException, Exception Message: The server responded with the following error: Null was returned to FindSpecific (Read Item operation).. Null was returned to FindSpecific (Read Item operation).)

Then your Specific Finder (e.g. "GetLeaveRequestById") most likely is not handling calls when an object is newly created (e.g. the Primary Key of the BCS Entity is either blank or 0). Basically, when a new object is created, it will be re-queried by the BCS engine - causing the exception.

To fix, you just have to change your BCS Read Item (e.g. GetLeaveRequestById) to handle a call for newly created objects when the Identifier is Blank or zero - and return a placeholder object with initialized fields.

This will appease BCS and allow it to continue to re-render your external entity list (from SAP services or otherwise)

DDK

SharePoint 2010 - Setting the Title for your External Content Type (esp for Display of Friendly name in BCS Associations)

A simple tip for modifying External Content Types in SharePoint Designer 2010 - if you have an external content type (e.g. talking to SAP via Duet Enterprise) - and you want to control the Display name that is shown in the Pickers, you need to change the Title of your External Content Type. You can do this by clicking on the Field you want to use as the Display name, and then click the "Set as Title" Ribbon Button under the field group. See below:

I was half expecting a context menu on right click for this functionality - but the UI designer at Microsoft has chosen the ribbon option for this functionality (unlike some other areas of SharePoint Designer).

DDK