Tuesday, 3 February 2009

Linq To Sql - Using System.Linq.Dynamic and Databinding Syntax to Allow Sorting without a Separate Projection Entity

Sometimes you will need to sort entities displayed in your UI by fields that are not direct properties of the entity themselves. E.g. the Territory Name for your Customer is defined on the lookup “Territories” entity, not the Customer entity itself. Now there are a couple of Linq and non-Linq ways to allow sorting and filtering on these fields related by joins:

  1. Create a SQL Server view that has the field in it e.g. adding the Territory Name to the Customer Table by pre-joining with the Territory lookup table. This view can then be consumed by Linq to Sql


  2. Forcing an enumeration of all elements in your query (ie bringing back ALL the data), and then appending the resolved value client side (NOT RECOMMENDED) e.g.
    a) By Adding a new property to the Linq entity partial class like so and forcing client side enumeration to do the sort/filter:



    ///

    /// Territory name - incorporates property from related entity Territory (name) to allow server-side queries

    ///


    public string TerritoryName

    {

    get

    {

    return Sales_SalesTerritory == null ? string.Empty : Sales_SalesTerritory.Name;

    }

    }


    b) By going through all your objects, adding the properties client-side via a Lookup process
    c) Relying on a client side datasource to do the paging for you - simpler but not scalable when there are 1000s of records.

  3. Creating your own Sql command strings using LINQ, a datareader and context.Translate() and doing the sorting – see http://www.west-wind.com/Weblog/posts/143814.aspx (NOT RECOMMENDED)

  4. Using Dynamic Expressions – Also see http://www.west-wind.com/Weblog/posts/143814.aspx (NOT RECOMMENDED)

  5. By Projecting all the properties you need into a new Projected Entity central object via your normal Linq joins, filtering and sorting as needed. However, if you have a complex entity that has many fields, it can be a bit of a waste of time (and an extra maintenance hassle) essentially rewriting all your Linq properties to Projection entities. Instead, you can use the .Select() Extension method and databinding to get your original object out of the projection. This means you DON'T need a separate projection entity to map these extra fields.

Here is a sample of how to do option number 5 using this shorthand projection technique. NOTE: This sample uses System.Linq.Dynamic from 101 LINQ Code Samples which (if you have Visual Studio 2008) can be found in %ProgramFiles%\Microsoft Visual Studio 9.0\Samples\1033. System.Linq.Dynamic allows you to pass non-typed values (ie just strings) for your sort expressions.

See when using Server-side sorting with System.Linq.Dynamic, you can also put the correct full-qualified sort expressions in your templated control (e.g. GridView) for nested entities in your projection. Note the sort expression in the code below which is consumed by System.Linq.Dynamic to generate the correct Sql:







/// <summary>
/// Project -> Sort -> Extract to avoid the need for a separate projection entity.
/// </summary>
/// </pre><param name="criteria">
/// <param name="totalCount">
/// <returns></returns>
public List<sales_customer> SearchCustomer(CustomerSearchCriteriaDto criteria ,out int totalCount)
{
var query =
from customers in _context.Sales_Customer
join territories in _context.Sales_SalesTerritory on customers.Sales_SalesTerritory equals territories
join customerType in _context.LookupValue on customers.LookupValue equals customerType
select new {Customer = customers, TerritoryName = territories.Name, CustomerTypeName = customerType.Value};

if (criteria.AccountNumber.Length > 0)
{
// Account Number
query = from customer in query
where customer.Customer.AccountNumber.Contains(criteria.AccountNumber)
select customer;
}

if (criteria.TerritoryId.HasValue)
{
// Territory
query = from customer in query
where customer.Customer.TerritoryId == criteria.TerritoryId
select customer;
}

if (criteria.CustomerTypeLookupId.HasValue)
{
// Customer Type
query = from customer in query
where customer.Customer.CustomerTypeLookupId == criteria.CustomerTypeLookupId
select customer;
}

totalCount = query.ToList().Count();

//Run server side query to get correct 10 records to display to user based on paging and sorting
//criteria.SortExpression is just a string.
return query.OrderBy(criteria.SortExpression).Skip(criteria.StartRowIndex)
.Take(criteria.MaximumRows).Select(newCustomer => newCustomer.Customer).ToList();
}

Sunday, 11 January 2009

Just Passed 70-541 - Technical Specialist: Microsoft Windows SharePoint Services 3.0 – Application Development

After the success of last week's exam (see http://ddkonline.blogspot.com/2009/01/just-passed-microsoft-exam-70-630-ts.html), I followed up with the Windows SharePoint services application development (WSS 3.0 development) exam. One of the main differences between the format of the 70-541 and 70-630 exam is that there is first a survey about (basically) how good you think you are at different areas of WSS development. I don't know if these affect the questions you are asked in the exam proper - but it is possible they use it as a way to choose the set of questions you recieve. Your guess is as good as mine, unless the folks at Microsoft/Prometric want to let us know how the application works...

The exam itself was slightly longer than the MOSS 2007 one (59 questions vs 51 questions) and as is typical for development exams - the difficulty level was a lot higher. You have to know some of your configuration basics PLUS the coding side of things. I came out of the exam after I got my score (900/1000) and was a little miffed that I answered the way I did for some of the questions. Many of the questions try to lead you up the garden path - and I probably was trying to anticipate what the exam creators were thinking a little too much.

I am considering Performance Point (70-556 - Technical Specialist: Microsoft Office PerformancePoint Server 2007, Applications) next as Oakton has some clients who are interested in using it - and skills in the area seem to be in short supply. One problem is that there are no recommended readings for the PeformancePoint exam, so it will largely be a Technet and MSDN study effort. I was in a similar situation for the ASP.NET 2.0 beta exams as well - so I should be OK.

Thursday, 8 January 2009

ABA bank payment file format (Australian Bankers Association) - Field Definitions and Sizes

I'm currently working on a payroll application for a large Australian engineering firm. As part of this, I need to export files for processing in the Australian defacto standard for Electronic Funds Transfer (EFT) files - the ABA format. If the file doesn't go through, hordes of angry engineers and mine workers won't get paid - so it is important that the file format of these files is in order.

So where does this fixed width .ABA file format come from? The Australian Bankers' Association is the national organisation of licensed banks in Australia, ranging from traditional retail, trading bank-style organisations to regional banks, foreign and wholesale banks.

These Banks (including the "Big Four", namely National Australia Bank (NAB), The Commonwealth Bank of Australia (CBA), Australia and New Zealand Banking Group (ANZ) and Westpac Banking Corporation (WBC)) reached agreement on a file format for Electronic Funds Transfers (EFT).

I presume for legacy reasons that the chief solutions architects at the banks chose a fixed width file format rather something more advanced like XML with XSD schema definitions (which would allow a simpler process for file format validation and improved readability). I suppose that readability isn't one of the primary goals for these "system generated" files.

Seeing as we are stuck with the format for now, I often have a hard time finding format definition for these files. e.g. the main ABA website doesn't seem to have it documented in any part of their site - http://www.bankers.asn.au/. Instead, for future reference, I've detailed the (.ABA) file format below with a list of the fields and dimensions of these fields:

1. Definitions


Commonly used terms associated with file formatting and their definitions are as follows:

  1. Left justified - start input in the first character position of that field
  2. Right justified - end input in the last character position of that field
  3. Blank filled - fills the unused portion of that field with blank spaces
  4. Zero filled - fills the unused portion of that field with zeros
  5. Unsigned - used in amount fields. Amounts will not be specified as debit or credit.




2. Header Record Definition ('0' record) (just the first line):
































































Character PositionField sizeField descriptionSpecification
1 1 Record Type 0 Must be '0'
2-18 17 Blank Must be blank filled.
19-20 2 Reel Sequence Number Must be numeric
commencing at 01. Right justified. Zero filled
21-23 3 Name of User's Financial
Institution
Must be approved
Financial Institution abbreviation. Westpac's abbreviation is "WBC".
24-30 7 Blank Must be blank filled.
31-56 26 Name of User supplying file
Must be User Preferred
Specification as advised in Application. Left justified, blank filled. All
coded character set valid. Must not be all blanks.
57-62
6Number of User supplying
file

Must be User
Identification Number which is allocated by APCA. Must be numeric, right
justified, zero filled.
63-74 12 Description of entries on
file e.g. "PAYROLL"

All coded character set
valid. Must not be all blanks. Left justified, blank filled.
75-80 6Date to be processed
(i.e. the date transactions are released to all Financial Institutions)


Must be numeric in the
format of DDMMYY. Must be a valid date. Zero filled.
81-120 40Blank
Must be blank filled.





3. Detail Record ('1' record)






























































































Character PositionField sizeField description Specification
1 1 Record Type 1 Must be '1'
2-8 7 Bank/State/Branch Number Must be numeric with a
hyphen in character position 5. Character positions 2 and 3 must equal
valid Financial Institution number. Character position 4 must equal a
valid State number (0-9).
9-17 9 Account number to be
credited/debited
Numeric, hyphens and blanks
only are valid. Must not contain all blanks or zeros. Leading zeros which
are part of a valid account number must be shown, e.g. 00-1234. Westpac
recommends that (except in the above example), ALL hyphens are edited out.
Where account number exceeds nine characters, edit out hyphens. Right
justified, blank filled.
18 1 Indicator "N" -for new or varied
Bank(FI)/State/Branch number or name details, otherwise blank filled.
Withholding Tax Indicators: "W" -dividend paid to a resident of a country
where a double tax agreement is in force. "X" -dividend paid to a resident
of any other country. "Y" -interest paid to all non-residents The amount
of withholding tax is to appear in character positions
113-120. Note: Where withholding tax has been deducted the appropriate
Indicator as shown above is to be used and will override the normal
indicator.
19-20 2 Transaction Code Must only be valid
industry standard trancodes (see list). Only numeric valid.
21-30 10 Amount Only numeric valid. Must
be greater than zero. Shown in cents without punctuations. Right
justified, zero filled. Unsigned.
31-62 32 Title of Account to be
All coded character set
valid. Must not be all blanks.
credited/debited Left justified, blank
filled. Desirable format: - surname (period) blank
- given names with blank
between each name
63-80 18 Lodgement Reference All coded character set
valid. Reference as submitted by the User indicating details of the origin
of the entry e.g. Payroll number, invoice, contract number.
Left justified, blank
filled. Must not contain all blanks.
81-96 (81-87) 16Trace Record (-BSB Number
in format XXX-XXX)
Bank(FI)/State/Branch and
account number of User to enable retracing of the entry to its source if
necessary. Only numeric and hyphens valid. Character positions 81 & 82
must equal a valid Financial Institution number. Character position 83
must equal a valid State number (0-9). Character position 84 must be a
hyphen.
(88-96) 9 (-Account Number) Right justified, blank
filled.
97-112 16 Name of Remitter Name of originator of the
entry. This may vary from Name of the User. All coded character set valid.
Must not contain all blanks. Left justified, blank filled.
113-8 Amount of Numeric only valid. Show
in cents without punctuation.
120 Withholding Tax Right justified, zero
filled. Unsigned.



4. File Total Record ‘7’ (Trailer)


























































Character
Position
Field sizeField description Specification
1 1 Record Type 7 Must be '7'.
2-8 7 BSB Format Filler Must be '999-999'.
9-20 12 Blank Must be blank filled.
21-30 10 File (User) Net Total Amount
Numeric only valid. Must
equal the difference between File Credit & File Debit Total Amounts.
Show in cents without punctuation. Right justified, zero filled. Unsigned.
31-40 10 File (User) Credit Total
Amount
Numeric only valid. Must
equal the accumulated total of credit Detail Record amounts. Show in cents
without punctuation. Right justified, zero filled. Unsigned.
41-50 10 File (User) Debit Total
Amount
Numeric only valid. Must
equal the accumulated total of debit Detail Record amounts. Show in cents
without punctuation. Right justified, zero filled. Unsigned.
51-74 24 Blank Must be blank filled.
75-80 6 File (User) count of
Records Type 1
Numeric only valid. Must
equal accumulated number of Record Type 1 items on the file. Right
justified, zero filled.
81-120 40 Blank Must be blank filled.


5. Direct Entry Transaction Codes











































13Externally initiated
debit items
50Externally initiated
credit items with the exception of those bearing Transaction Codes
51-57 inclusive
51Australian Government
Security Interest
52Family Allowance
53Pay
54Pension
55Allotment
56Dividend
57Debenture/Note Interest


Saturday, 3 January 2009

Just passed Microsoft Exam 70-630 Technical Specialist: Microsoft Office SharePoint Server 2007, Configuring with Full Marks

Yesterday, I just passed my first SharePoint 2007 exam (70-630) with a score of 1000/1000. I've done many Microsoft exams but this is the first that I received full marks for. It was a bit of a shock!

Funny thing was, I almost didn't get to do the exam because the Prometric testing site @ North Ryde (in Sydney) didn't have my name registered (and the name of 3 other guys who arrived at the testing centre). This was because I rescheduled the exam online (at http://www.register.prometric.com/Index.asp) the day Prometric closed over the holiday period. I had to wait around for an hour before the Prometric help desk opened up and did a data synchronisation. Why the process isn't just a scheduled SQL job is beyond me.

Without violating the Non-Disclosure Agreement (these are all in the preparation guide at http://www.microsoft.com/learning/en/us/exams/70-630.mspx), here are some of the things to look for:
  1. Content types, Content types, Content types
  2. Security, Audiences and how to enable/disable Personalisation functionality
  3. Business Data Catalogs
  4. Know all the stsadm command line parameters inside out - http://technet.microsoft.com/en-us/library/cc263384.aspx
  5. Know the innards of the logging and diagnostic functionality of MOSS.

My primary sources of information were the 1200 page corker named "The Microsoft Office SharePoint Server 2007 Administrator's Companion", SharePoint 2007 Central Administration online help and, of course, TechNet for SharePoint - see http://technet.microsoft.com/en-au/office/sharepointserver/default.aspx

Thursday, 18 December 2008

"Command line error." when installing Web Part into WSS 3.0/MOSS 2007 with stsadm.exe

This is a bizarre problem - but if you get the stsadm.exe generic "command line error." whilst trying to install SharePoint web parts and your paths look fine, it may just be an encoding issue when you copy the commandline arguments between different apps. You do NOT need to have your wsp file in the same directory as stsadm.exe when installing parts to your site. You see, different apps interpret hyphens differently. If you copy a hyphen from a web site, it may just be a unicode representation of a hyphen and not a "real" hyphen. For more detail, see:

http://weblogs.asp.net/soever/archive/2007/12/22/sharepoint-stsadm-exe-and-the-infamous-quot-command-line-error-quot.aspx
and
http://www.celestialsoftware.net/support/forums?ubb=get_topic%3bf=1%3bt=000048

A simple solution is just to make sure you type all your stsadm.exe command parameters in manually and not copy and paste them into your DOS prompt.

Friday, 12 December 2008

Telerik Releases "Open Access", a Database Agnostic ORM Product that Works with LINQ

My favourite 3rd party WebUI control provider Telrik just released its new ORM product called "Open Access" - http://www.telerik.com/products/orm.aspx. I didn't realise they were developing such a thing - but it turns out they just acquired German company Vanatec that specializes in ORM products. They have performed a few updates to the original Vanatec software since they acquired it (such as removing a dependency on J#) - so they have grabbed this product and are running with it full steam. If the quality of their controls is anything to go by, this could be a valuable asset in any .NET developer's toolbelt.

I'm going to try it out and evaluate it against some of the custom LINQ, LINQ to SQL and Nhibernate-based efforts that I've created and worked with on previous projects. It also supports non-SQL Server databases such as Oracle. Now there is also a fledgling Codeplex project called LINQ to Oracle http://www.codeplex.com/LinqToOracle) but this ORM product could shoot it out of the water. Telerik Open Access also supports direct SQL. I'll do a a review before the end of the year.

Wednesday, 3 December 2008

How can I set the Modified By, Created By, Date Modified, Date Updated fields via the MOSS object model? (without making an new version)

There are a few problems with the MOSS object model when adding new files using the SPFileCollection.Add() method. In particular, there is no overload that accepts both the "bool overwrite" parameter AND the details of the user who did the update at the same time.



Consequently, the upload of a file to a versioned list in SharePoint requires that you separately add the file with overwrite on and then update the User and Time stamps at a later stage.

Unfortuntately, the "Author" and "Editor" fields accessible via the SPFile Object are read-only. You can however take advantage of the UpdateOverwriteVersion(); available on list items to update these stamps manually. See the code below:



//The authenticating user needs to be service account as it uses database access,
// so we must pass in current user as parameter when adding file.
SPUser updatingUser = EnsureUser(HttpContext.Current.User.Identity.Name, web);
currentFile = fileCollection.Add(newDocument.Name, contents, fileProperties, addAsNewVersionToExistingFiles);

//Get list item from SPFile object
SPListItem listItem = currentFile.Item;

//Overwrite with correct values as the object model doesn't allow us
// to both specify overwrite=true and the specific user names.
listItem["Author"] = updatingUser;
listItem["Editor"] = updatingUser;
listItem.UpdateOverwriteVersion();


How do I handle or abort Function Key events (e.g. F1, F2,etc) in both IE and Firefox?

Run this page and you will be shown the keycode for the Function Key you pressed. In addition, any standard browser handlers (such as help prompts when F1 is pressed or Searches when F3 is pressed) will be aborted - so you can pass them to your app instead. See below:


<script type="text/javascript" language="javascript">
/////////////////////////////////////////////////////////////////////
///Demo Script to display the function key that was pressed
///and abort any browser event e.g. F3 for IE find,
///F1 for IE help, F1 for Firefox Help
/////////////////////////////////////////////////////////////////////
///Version Author Date Comment
///1.0 DDK 03 Dec 2008 Original Version for
/// Application Tender
/// Proof of concept
/// when users wanted 'green
/// screen' functionality
/////////////////////////////////////////////////////////////////////
//debugger;
document.onkeydown = showDownAndAbortEvent;
//Stop F1 opening Help in IE
document.onhelp=function() {return false};
window.onhelp=function() {return false};

// decipher key down codes
function showDownAndAbortEvent(evt)
{
//clearCells();
evt = (evt) ? evt : ((event) ? event : null);
if (evt) {
alert('keyCode:' + evt.keyCode + ';charcode' + evt.charCode + ';' ) ;
try
{
evt.preventDefault(); // disable default help in Firefox
evt.stopPropagation();
}
catch (ex) {}
try
{
//Kill any intercepts for ie
window.event.cancelBubble = true;
window.event.returnValue = false;
window.event.keyCode = 0;
}
catch (ex) {}
return false;
}
}
</script>