Business Benefits of Duet Enterprise and Comparison to SAP integration via Biztalk/Custom Solutions - Bringing SAP Business Data and SharePoint Collaboration Features Together and Ensuring a ROI

I've been involved in several SAP-SharePoint integration projects recently. Such projects typically relied heavily on an Enterprise Service Bus (ESBs - such as Biztalk or SAP PI), customizations and code. When you integrate SAP with SharePoint, there are many benefits that are provided when users don't have to use SAP GUI directly:

• Hard benefits such as reduced licensing costs
• Usability benefits such as allowing for a much simplified User Interface (people familiar with SAP GUI and SAP Portal will chuckle at this)
• Supportability Benefits (as SharePoint and .NET skills are more common that SAP Portal skills - at least in the Australian market).

However, since the release of Duet Enterprise 1.0 (http://ddkonline.blogspot.com/2011/01/duet-enterprise-released-and-available.html) at the end of January, another integration option has been thrown into the mix.

When my clients have been considering Duet Enterprise, one of the most often asked questions is

"What does Duet Enterprise offer me that I can't do already with SharePoint, SAP, custom BAPIs, SAP Enterprise Services and SharePoint 2010 Business Connectivity Services?".

Duet Enterprise brings a lot to the table - both on the SAP side and on the SharePoint side - however it's not the perfect solution for all problems. While consistency between systems is a vital principle in enterprise architecture, you really do have to choose the right combination of tools to fit a given problem. Indeed, as with any architectural approach - you have to weigh up the pros and cons.

I see the Duet Enterprise product offering (and the development framework it provides) as having the following advantages over just a custom integration build:

1. Security and Security Best Practices. Through the Duet Enterprise Claims-based Role Provider, it allows you to leverage the SAP security model and SAP Roles in SharePoint. This avoids the duplication of a potentially massive security hierarchy into the SharePoint space. By design, communication channels between systems are encrypted. This is one of the biggest value-adds. This also allows you to keep all your unstructured data (e.g. pdfs/documents) in SharePoint and still leverage the security model in SAP to limit access to these resources.
2. SAP UWL Functionality - It exposes your SAP Universal Worklist (UWL) into the native SharePoint task list - you don't have to do a mashup or WSRP or some other IFrame hack.
3. Maintainability and Reduced Development Efforts -  It provides some out of the box functionality (site creation, business centric collaboration) as a guideline and template for further development.
4. Diagnostic Tool Support - Duet Provides Health Checks that allow you to check your setup, and provides mechanisms (via Correlation Identifiters) to help you diagnose problems if requests are failing. 
5. Official Supportability - It is a product supported by Microsoft and SAP that incorporates their recommended approach for integration. There is also official documentation for this integration approach.
6. It has a Roadmap - It is a platform that has a roadmap (as opposed to a completely custom system) which is something you build and have to take a lot more responsibility for. As Duet Enterprise grows in capabilities, you will regret  going down the 100% custom path.
7. Reporting Functionality - it brings Reporting Functionality out of SAP via the Reporting Request mechanisms and alows you and your colleagues to collaboration around SAP Reports via standard SharePoint functionality (e.g. Social tagging)  
8. The BCS Solution Design Gallery - It provides an enhanced model for the deployment of Office Based Solutions through SharePoint e.g. Administration screens for arranging your solution and for generating your solution from within SharePoint. See http://msdn.microsoft.com/en-us/library/ff963717.aspx for more details.
9. Entity Collaboration Functionality - Supports creation of collaboration sites surrounding SAP business entities. This however has to be checked against your SharePoint governance strategy and capacity plan.
10. SAP HR Information imported into SharePoint User Profiles - ForeFront Identity Manager is used synchronize SAP User information (HR infotypes) from SAP via a BCS user profile connection, along with role synchronization via the Duet Enterprise Profile Synchronization timer job.

I have also been approached around concerns that it Duet Enterprise doesn't use a full ESB. However that's not true. Duet is a mechanism for providing services on the SAP side (via an ESB on the SAP side e.g through SAP's middleware solution "SAP PI") - and consuming these services on the SharePoint side. Duet Enterprise can live comfortably in an ESB-enabled an ecosystem. 

If you have SharePoint and SAP in your environment, I suggest that you consider Duet Enterprise as part of your enterprise strategy for improving the accessibility and collaboration around your most important business data.

In most circumstances, it will give you a large amount of momentum (and provides all the plumbing) to allow you to provide a solid and supported solution for SAP to SharePoint integration - with an upgrade path.

DDK

SharePoint 2010 Business Connectivity Services (BCS) - Exposing External Lists through Outlook and the InfoPath External Data Part - Fix for "Catastrophic Failure"

Business Connectivity Services (BCS) along with BCS solutions allow you to expose your line of business data (e.g. SAP via Duet Enterprise) through Office client applications such as Outlook 2010.

There are several walkthoughs of generating these solutions available online. Some examples of these are:

• "Creating Intermediate Declarative Outlook Solutions Using Business Connectivity Services" (http://msdn.microsoft.com/en-us/library/ee819909.aspx)
• "How To: Create an Outlook Solution Using Duet Enterprise Outlook Application Designer"
   - http://msdn.microsoft.com/en-us/library/ff963717.aspx (Though this is specific to the Duet Enterprise Deployment mechanisms, many of the same steps apply).

I recently had issues with deployment of a Business Connectivity Services (BCS) solution in conjunction with Duet Enterprise 1.0. When my Duet Enterprise solution was deployed , I was being greeted with an exception as per the screenshot below, with a "Catastrophic failure" error message in place of where my custom task pane should show.

The main reason this exception was occuring was because my layout file in my solution (the xsd and schema explanation for BCS Solution layout files can be found here: http://msdn.microsoft.com/en-us/library/ff394500.aspx) had a "Width" custom property.

Outlook couldn't handle this as a parameter to the web part - so it crashed out in a "Catastrophic" manner.

Also note that for your task pane to have any datasource at all whatsoever, you still need to have a "DataSourceName" property set on your OBA part. The description on MSDN here (http://msdn.microsoft.com/en-us/library/ff394500.aspx) is somewhat misleading as it indicates that you only need a datasource for your External Data Part if it relies on data from one of the other parts in your layout. If you remove the datasource property, the attributes weren't populated - so the statement on MSDN isn't correct.

Fix for SQL 2008 Exception - An exception occurred while enqueueing a message in the target queue. Error: 15404, State: 19. Could not obtain information about Windows NT group/user 'DOMAIN\username', error code 0x5

After installing a recent series of Windows update, I began to get hundreds of SQL 2008 errors in the Windows Event log (20 or so per second) on one of our demonstration servers. These were as follows:

An exception occurred while enqueueing a message in the target queue. Error: 15404, State: 19. Could not obtain information about Windows NT group/user 'DOMAIN\username', error code 0x5.

or

An exception occurred while enqueueing a message in the target queue. Error: 15404, State: 19. Could not obtain information about Windows NT group/user 'DOMAIN\username', error code 0x2.

If you do have these exceptions, then you most likely have connectivity issues to your Active Directory server - and the identity of the owner of your databases  (dbo) is on that Active directory server. The SQL Service Broker is causing this exception. If this is not a resolvable issue condition, then you can change the owner to a SQL login (e.g. sa) and this will resolve the exception.

You can do this with the following command on each database that has the wrong owner:

sp_changedbowner 'sa'

DDK

SharePoint 2010 - Business Connectivity Services (BCS) / BDC model export - Default vs Client Option

When exporting BCS entities, you may have noticed 2 different "Settings" options available in the export dialog. If you're wondering what the differences are in the output - there don't appear to be any at all.

To confirm this, I just did an export of 2 entities to a model file using the "Default" option and the other export using the "Client" option. Using the Notepad++ text compare tool, there was no physical difference between the output of the different export settings - nor does there seem to be any documentation regarding this dropdown at all.

However, if you look at the BCS connection information, you'll also notice 2 tabs that match the different settings or "profiles" that you can choose. Consequently, this is expected behaviour - as you'll only have differences when your settings are different between default and client.

DDK

Not all Reflection Tools (or Obfuscators) Are Created Equal

I had a client last week in Melbourne who wanted to salvage some code from an existing SharePoint 2010 implementation. As long as it wasn't obfuscated, then I thought there would be no problems at all.

Red Gate's (previously Lutz Roeder's) Reflector is designed for just a situation - and I'd recently purchased the awesome VS PRO version (which is phenomenal lets you step and debug through other people's applications!).

However, when I tried to open up the assemblies in Reflector or on ILDASM, it appeared to indeed be obfuscated - by the Smart Assembly tool from Red Gate.

Typically, the obfuscated code will be shown with an error or garbled characters. e.g. "This item is obfuscated and can not be translated." - as below:

If you try and open it in ILDASM, it throws an exception as below:

This is because the Assembly has the CompilerServices "SuppressIldasmAttribute" applied to it.

However, if you open up the assembly with the new tool JetBrains dotPeek (the makers of Resharper), then you will be able to see the source code - even of those allegedly obfuscated methods and properties.

I'm not sure whether Red Gate deliberately set a flag inside Reflector when they purchased it from Lutz Roeder - but it seems like a few shortcuts were taken with the obfuscation engine.

So be warned - not all Reflectors and not all Obfuscation methods are created equal.


DDK

Duet Enterprise Integration - Fix for "System.ArgumentException: An item with the same key has already been added due to Extra Content Type Headers" in SharePoint 2010 ULS Logs

If you're not aware, Duet Enterprise (http://www.duet.com/) is an integration framework and infrastructure for providing SAP to SharePoint integration - of task lists, SAP workflows, SAP reports, SAP security and SAP data (through SharePoint BCS).

I recently had an issue with a Duet Enterprise deployment for one of my clients. When retrieving data, the out of the box web parts were failing and dropping the following Exception in the SharePoint 2010 ULS Logs:

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ArgumentException: An item with the same key has already been added. Server stack trace:
at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
at System.Xml.ContentTypeHeader.ParseValue()
at System.Xml.ContentTypeHeader.get_MediaType()
at System.Xml.XmlMtomReader.ReadRootContentTypeHeader(ContentTypeHeader header, Encoding[] expectedEncodings, String expectedType)
at System.Xml.XmlMtomReader.Initialize(Stream stream, String contentType, XmlDictionaryReaderQuotas quotas, Int32 maxBufferSize)
at System.Xml.XmlMtomReader.SetInput(Stream stream, Encoding[] encodings, String contentType, XmlDictionaryReaderQuotas quotas, Int32 maxBufferSize, OnXmlDictionaryReaderClose onClose)
at System.Xml.XmlDictionaryReader.CreateMtomReader(Byte[] buffer, Int32 offset, Int32 count, Encoding[] encodings, String contentType, XmlDictionaryReaderQuotas quotas, Int32 maxBufferSize, OnXmlDictionaryReaderClose onClose)
at System.ServiceModel.Channels.MtomMessageEncoder.MtomBufferedMessageData.TakeXmlReader()
at System.ServiceModel.Channels.BufferedMessageData.GetMessageReader()
at System.ServiceModel.Channels.BufferedMessage..ctor(IBufferedMessageData messageData, RecycledMessageState recycledMessageState, Boolean[] understoodHeaders)
at System.ServiceModel.Channels.MtomMessageEncoder.ReadMessage(ArraySegment`1 buffer, BufferManager bufferManager, String contentType)
at System.ServiceModel.Channels.HttpInput.DecodeBufferedMessage(ArraySegment`1 buffer, Stream inputStream)
at System.ServiceModel.Channels.HttpInput.ReadBufferedMessage(Stream inputStream)
at System.ServiceModel.Channels.HttpInput.ParseIncomingMessage(Exception& requestException)
at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown
at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at BCSServiceProxy.IWXManageCustomerIn.FindCustomerByElements(FindCustomerByElementsRequest request)
at BCSServiceProxy.WXManageCustomerInClient.BCSServiceProxy.IWXManageCustomerIn.FindCustomerByElements(FindCustomerByElementsRequest request)
at BCSServiceProxy.WXManageCustomerInClient.FindCustomerByElements(BPCCustGetAll BPCCustGetAll) -


-- End of inner exception stack trace ---
at System.RuntimeMethodHandle._InvokeMethodFast(Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
at Microsoft.SharePoint.BusinessData.SystemSpecific.Wcf.WcfSystemUtility.Execute(Object[] args)

I immediately thought that it was something wrong with the data. Because Duet uses HTTPS & SSL for all communications between SAP Netweaver 7.02+ and SharePoint 2010, I had to decrypt the traffic to work out what was going on. Running the calls on the SAP site returned blank results. To get WireShark (http://www.wireshark.org/) decrypting the traffic we had to generate a certificate in SAP with a private key. We had to get the key out in PKCS#12 format and then convet it to the .PEM format that wireshark accepts - similar to what is desribed here:

http://htluo.blogspot.com/2009/01/decrypt-https-traffic-with-wireshark.html
We used "sapgenpse" in SAP to generate the certificate with the private key - as per the screenshot below:

Once the key was set in WireShark preferences, you will see many of the packets being processed being marked as green. You can then right click on them and choose "Follow SSL Stream" on your web request - and you will see the decrypted version of your HTTPS traffic.

Turns out, the issue was with SAP retuning duplicate Content headers in the response to SharePoint BCS (as per the screenshot).

There is a SAP Note to fix this very issue - "SAP Note 1539888 - Composite Note -Duet Enterprise Installation Wizard". You will need a SAP Service Market place to log on to get this note.

DDK

Duet Enterprise - [ERROR] - "Cannot Merge Contents of LobSystem (External System) with Name 'Account' as it appears to be different from preexisting LobSystem in the current load context." - Fix

When using Duet Enterprise (for SAP and SharePoint 2010 Integration) you may receive the following exception when importing Duet Business Connectivity Service Models (as per http://technet.microsoft.com/en-us/library/gg394164.aspx):

"Duet Enterprise - [ERROR] - Cannot Merge Contents of LobSystem (External System) with Name 'Account' as it appears to be different from preexisting LobSystem in the current load context.".

This may occur even after you've used Duet to remove all of the models with the commandline DuetConfig.exe /RemoveBDC.

You can fix this issue by going to:

This may occur even after you've used Duet to remove all of the models with the commandline DuetConfig.exe /RemoveBDC.

SharePoint Central Administration - Your Business Connectivity Services (BCS) Service Application - Manage
Then:

1) Choose "Models" in the "View" dropdown and remove any Duet models if they still exist

2) Choose "External Systems" and Remove any Duet External systems (this is the external system referred to in the error).

You should now successfully be able to import the Duet Enterprise 1.0 out of the box Models.

DDK

SharePoint 2010 Business Connectivity Services (BCS) - Exception when adding new Entities - "Null was returned to FindSpecific (Read Item operation)" - Fix

Scenario: You've set up SharePoint 2010 External Content Types and you receive the following exception when adding a new item to your BCS list:

Form submission failed. (User: DOMAIN\UserName, Form Name: Template, IP: , Request: http://servername/sites/Duet/Lists/Leave Request/Item/newifs.aspx?RootFolder=/sites/Duet/Lists/Leave Request&IsDlg=1#, Form ID: urn:schemas-microsoft-com:office:infopath:entity:-AutoGen-2011-06-08T05:13:33:61Z, Type: DataAdapterException, Exception Message: The server responded with the following error: Null was returned to FindSpecific (Read Item operation).. Null was returned to FindSpecific (Read Item operation).)

Then your Specific Finder (e.g. "GetLeaveRequestById") most likely is not handling calls when an object is newly created (e.g. the Primary Key of the BCS Entity is either blank or 0). Basically, when a new object is created, it will be re-queried by the BCS engine - causing the exception.

To fix, you just have to change your BCS Read Item (e.g. GetLeaveRequestById) to handle a call for newly created objects when the Identifier is Blank or zero - and return a placeholder object with initialized fields.

This will appease BCS and allow it to continue to re-render your external entity list (from SAP services or otherwise)

DDK