Fix - Duet Enterprise/SharePoint 2010 Exception - "System.ServiceModel.QuotaExceededException: The size necessary to buffer the XML content exceeded the buffer quota."

When attempting to establish communications between SAP and SharePoint (using the Duet Enterprise Claims provider) today at a new client, I encountered the following exception in SharePoint (as per the ULS logs):

InnerException 1: System.ServiceModel.QuotaExceededException: The size necessary to buffer the XML content exceeded the buffer quota. Server stack trace:

at System.ServiceModel.Channels.BufferedOutputStream.WriteCore(Byte[] buffer, Int32 offset, Int32 size)
at System.Xml.XmlStreamNodeWriter.FlushBuffer()
at System.Xml.XmlBinaryNodeWriter.FlushBuffer()
at System.Xml.XmlStreamNodeWriter.GetBuffer(Int32 count, Int32& offset)
at System.Xml.XmlBinaryNodeWriter.UnsafeWriteText(Char* chars, Int32 charCount)
at System.Xml.XmlBinaryNodeWriter.WriteText(Char[] chars, Int32 offset, Int32 count)
at System.Xml.XmlBaseWriter.WriteChars(Char[] chars, Int32 offset, Int32 count)
at System.Xml.XmlBinaryWriter.WriteTextNode(XmlDictionaryReader reader, Boolean attribute)
at System.Xml.XmlDictionaryWriter.WriteNode(XmlDictionaryReader reader, Boolean defattr)
at System.ServiceModel.Channels.ReceivedFault.CreateFault12Driver(XmlDictionaryReader reader, Int32 maxBufferSize, EnvelopeVersion version)
at System.ServiceModel.Channels.MessageFault.CreateFault(Message message, Int32 maxBufferSize)
at System.ServiceModel.Channels.SecurityChannelFactory`1.ClientSecurityChannel`1.TryGetSecurityFaultException(Message faultMessage, Exception& faultException)
at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.ProcessReply(Message reply, SecurityProtocolCorrelationState correlationState, TimeSpan timeout)
at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown
at [0]:
at Microsoft.SharePoint.BusinessData.SystemSpecific.Wcf.WcfSystemUtility.Execute(Object[] args)
at Microsoft.SharePoint.BusinessData.SystemSpecific.Wcf.WcfSystemUtility.ExecuteStatic(IMethodInstance methodInstance, ILobSystemInstance lobSystemInstance, Object[] args, IExecutionContext context)
at Microsoft.SharePoint.BusinessData.Runtime.DataClassRuntime.ExecuteInternalWithAuthNFailureRetry(ISystemUtility systemUtility, IMethodInstance methodInstanceToExecute, IMethod methodToExecute, ILobSystemInstance lobSystemInstance, ILobSystem lobSystem, IParameterCollection nonReturnParameters, Object[] overrideArgs)
at Microsoft.SharePoint.BusinessData.Runtime.DataClassRuntime.ExecuteInternal(IDataClass thisDataClass, ILobSystemInstance lobSystemInstance, ILobSystem lobSystem, IMethodInstance methodInstanceToExecute, IMethod methodToExecute, IParameterCollection nonReturnParameters, Object[]& overrideArgs)

I also received the following exception immediately after the above exception:
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ServiceModel.QuotaExceededException: The size necessary to buffer the XML content exceeded the buffer quota. Server stack trace:

at System.ServiceModel.Channels.BufferedOutputStream.WriteCore(Byte[] buffer, Int32 offset, Int32 size)
at System.Xml.XmlStreamNodeWriter.FlushBuffer()
at System.Xml.XmlBinaryNodeWriter.FlushBuffer()
at System.Xml.XmlStreamNodeWriter.GetBuffer(Int32 count, Int32& offset)
at System.Xml.XmlBinaryNodeWriter.UnsafeWriteText(Char* chars, Int32 charCount)
at System.Xml.XmlBinaryNodeWriter.WriteText(Char[] chars, Int32 offset, Int32 count)
at System.Xml.XmlBaseWriter.WriteChars(Char[] chars, Int32 offset, Int32 count)
at System.Xml.XmlBinaryWriter.WriteTextNode(XmlDictionaryReader reader, Boolean attribute)
at System.Xml.XmlDictionaryWriter.WriteNode(XmlDictionaryReader reader, Boolean defattr)
at System.ServiceModel.Channels.ReceivedFault.CreateFault12Driver(XmlDictionaryReader reader, Int32 maxBufferSize, EnvelopeVersion version)
at System.ServiceModel.Channels.MessageFault.CreateFault(Message message, Int32 maxBufferSize)
at System.ServiceModel.Channels.SecurityChannelFactory`1.ClientSecurityChannel`1.TryGetSecurityFaultException(Message faultMessage, Exception& faultException)
at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.ProcessReply(Message reply, SecurityProtocolCorrelationState correlationState, TimeSpan timeout)
at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown
at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at BCSServiceProxy.IWXManageCustomerIn.FindCustomerByElements(FindCustomerByElementsRequest request)
at BCSServiceProxy.WXManageCustomerInClient.BCSServiceProxy.IWXManageCustomerIn.FindCustomerByElements(FindCustomerByElementsRequest request)
at BCSServiceProxy.WXManageCustomerInClient.FindCustomerByElements(BPCCustGetAll BPCCustGetAll) -
-- End of inner exception stack trace ---
at System.RuntimeMethodHandle._InvokeMethodFast(Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
at Microsoft.SharePoint.BusinessData.SystemSpecific.Wcf.WcfSystemUtility.Execute(Object[] args)

With my experiences with SAP, buffer overflows normally indicate that a large Java stack trace is coming back from SAP and it overloads the WCF client buffer which is expecting a normal SOAP response.

To see the real exception, I tried to use Wireshark (http://www.wireshark.org/)- but our setup had (not by choice) Netweaver and SharePoint on the same box - and Wireshark cannot listen to localhost traffic.

As usual, the Fiddler Tool HTTP proxy (http://www.fiddler2.com/fiddler2/) came to the rescue- as it can actually listen to traffic occurring between applications on the same machine. When running Fiddler and listening to the https traffic, it popped up with an exception regarding certificate errors:

Session #24: The remote server (ausyd-a-sh1) presented a certificate that did not validate, due to RemoteCertificateChainErrors.

SUBJECT: CN=ausyd-a-sh1, OU=I0020310622, OU=SAP Web AS, O=SAP Trust Community, C=DE
ISSUER: CN=ausyd-a-sh1, OU=I0020310622, OU=SAP Web AS, O=SAP Trust Community, C=DE
EXPIRES: 1/01/2038 11:00:01 AM
I went to the server url in Internet Explorer and sure enough there was a certificate error occurring when going to the URL of the SAP server - at https://ausyd-a-sh1:8001/

Basically this exception was occurring because the SAP SSL certificate was not in the Trusted Root Certification Authorities store in Windows. To import it, I just opened mmc at a command prompt and added the certificates Snap-In (via File - Add/Remove Snap In... - for the Local Computer.

In Certificates (Local Computer), I went to Trusted Root Certification Authorities - Certificates and Imported the SAP Certificate in.

However, I kept getting the same exception. On closer inspection, the thumbprint of the SharePoint Security Token Service (STS) certificate was not the same as what had been imported into SAP - this was becuase SharePoint had been reinstalled - which changes the SharePoint STS certificate (i.e. SharePoint STS certificates are install specific).

I then received the following Exception - which was related to user mappings in SAP:

An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail

To resolve this last exception:

1) We made sure SharePoint:: was used to prefix the name in SAP (in the VUSEREXITID table) - this prefix would normally be added through the Duet Active Directory user import job (specific to Duet Enterprise) - but we don't have AD in our environment.
2) We then updated the STS certificate in the SAP Certificate store (using SAP transaction /nstrust)

3) Used the /nsaml2 transaction to update the certificate used there as well.

This resolved all our SAP to SharePoint communication issues.

DDK