InfoPath Forms "Dropdown of Repeating Self" Bug Stomp


I was fooling around with InfoPath this weekend, trying to create a generic form and data definition for a flexible ad hoc workflow.

One thing I wanted in this form was the ability to specify dynamic routing between workflow steps. So, I created a repeating table for the steps and I also bound a repeating section to the same repeating group in the data. This let me summarize the steps in the table and then let the user specify the properties for each step in the details section below.

In the details section, I have dropdowns for Approve To, Reject To, and Overdue To that are meant to tell the workflow which step it should switch to when these events take place. Each of these dropdowns is meant to be populated by the list of steps.

Here is a shot of the designer:

Screenshot of the Designer View in InfoPath

So, in the data the XML looks like this:

Workflow Data Source Schema

Basically, I have the dropdowns bound to RouteTo in each case, and I want them to let the user pick from ID and TItle for the entire list of WorkflowTask items.

This is where InfoPath barfed, showing a glaring weakness. When you bind the dropdown list's items to the form data, and specify the repeating group as //WorkflowTasks/WorkflowTask it *should* encode that as relative to the parent element, giving you the full list, but instead it encodes it relative to the current element (which turns out to be just '.') giving you only the current WorkflowTask node.

I should also note that if you point the list to a repeating node that is outside the parent oath of the node of your dropdown list, then it will behave itself normally and give you all the nodes under that group.

Though it may be okay in some cases, for a dropdown list this behavior is less than useless. The field browser should give you enough control to let you specify a differnt xpath query at least, which it does not, leaving you stuck with what the wizard lets you pick. I will completely ignore the obvious usefulness of an xpath expression complete with functions, etc. that could've easily allowed for dynamic filtering of the list items.

So, my workaround was to save the IP template as source files, and then go into the source for view1.xsl and find the specific xsl that does the transform for the list. In my case, this looks like this.

I search for "Route To", which is my display name for the dropdown. Below, look for code like:

<xsl:for-each select="."><-- This is the offending xpath
<option>
<xsl:attribute name="value">
<xsl:value-of select="@my:ID"/>
</xsl:attribute>
<xsl:if test="$val=@my:ID">
<xsl:attribute name="selected">selected</xsl:attribute>
</xsl:if>
<xsl:value-of select="@my:DisplayName"/>
</option>
</xsl:for-each>

So, to get the complete list of nodes when you are already inside of it, just swing up to the parent node and select all the desired sub-nodes, like so:

<xsl:for-each select="../my:WorkflowTask">

Once I am done, I open my manifest.xsf and save as back to the original xsn form template. It isn't clear to me how long this hack will hold, but I was able to preview the modified template, and even make some changes to other parts of the form without reverting to the undesirable '.'.

Here's the preview of the form:

Screenshot of InfoPath Form in Preview Mode

It's not the most elegant solution in terms of maintainability, but it was a lot easier than writing custom code behind (and in some cases that isn't necessarily an option anyway), so I am glad to have found it.

Come to think of it, though I haven't tried it yet, you could hack the select with an xpath expression to filter the dropdown. If anyone gives that a shot, shoot me an email or comment and let me know. (Send it to anything at thomascarpe.com until I get my blog comments working.)

Update:
I did eventually see some weird looking XSL in the dropdowns after a couple of successive load-saves. This caused the value that was selected to be displayed as a choice within the list, and was completely undesirable behavior. I simply used the same hack as above to rid myself of the annoying extra option tag. :-)

Custom Fields and Content Types

In a word: BE CAREFUL.
 
Okay, that is actually two words. :-)
 
Just be careful about the naming conventions that you use and how you change them later. If you need to uninstall / reinstall a custom field and control, you will likely have problems with the lists that were built on top of it. You will also have problems with the SharePoint Site Column and Content Type management pages. Needless to say, this is NOT good.
 
Fortunately, it looks like the fields can be manually deleted using the SharePoint web services. I imagine it is also possible to do this from stsadm, but I never much liked command line tools, so I am trying now to create a little Windows Forms application that will do the job for me.
 
I will post my code back here when I get it stable. For now, just be careful about changing or removing custom field names and namespaces, etc.
 
Update:
 
Okay, there is only a very small amount of useful stuff online at this point, so I will try to be as specific as I can - for a Friday evening.
 
My Issue and Its Root Cause
 
Create a custom field control and field type and install them to SharePoint either by hand, with an MSI, or a WSP. In particular, I want to focus on the FLDTYPES_xyz.xml file (or whatever you decide to call yours). This is the file that tells SharePoint how to make use of your custom field control.
 
Now, suppose you decide later that you don't like the name of one of your fields. Maybe you mis-spelled it, or maybe (like me) you just decided that you didn't want there to be any naming ambiguity between your custom fields and those that other developers might provide. So, you rename your field in the definition. You think it will be okay because you are going to uninstall and reinstall your solution anyway.
 
Here is the gotcha. If you have used your custom field in any list, then a reference to it will have been added to the SPWebs.Fields collection, which is a master list of all the field definitions that are used for your entire web site. Additionally, if you have added it to any of your content types, then a reference will exist in the form of a SPFieldLink in the SPWeb.ContentTypes[x].FieldLinks collection.
 
If you remove the FLDTYPES.xml file that these references are relying upon, or change the names in it, then you'll see some pretty interesting behavior out of SharePoint! Hopefully what I am posting here will save somebody the entire workday that this issue cost me.
 
What manifests is that your lists, the content types admin web page, and the site columns admin web page will all be broken by this change. In particular, the site columns page will give an error:
The given key was not present in the dictionary.
Very helpful indeed - as helpful approaches zero. The other pages will have other cryptic messages as well, but I did not capture them here. Maybe some time I will go back and repro this error so I can take some screen shots. Sadly, nothing I did to the CustomErrors settings on my server allowed me to see a stack trace.
 
What you can do to fix this behavior is change the TypeName property back to what it was in your XML file - assuming you can remember what it was.
 
<Field Name="TypeName">MyOldTypeName</Field>
 
If you can't remember the name you used that is causing you the problem, you can use the SharePoint API to find it.
 
Simply open your web site with an SPWeb object, then attempt to iterate through the SPWeb.Fields collection. You will immediately receive an exception to the effect of something like "FieldType named 'x' is not properly installed. Go to the Site Collection page in Site Settings to correct the problem." Of course this is bollocks, because the web page is broken by this problem.
 
   string customColumnName = "My_x0020_Custom_x0020_Column";
   string contentTypeName = "MyContentType";
   using (SPSite site = new SPSite( "http://spdev" )) {
    using (SPWeb web = site.OpenWeb( "/site1" )) {
     // just some useful code for getting the lsit of names
     List<string> contentTypeNames = new List<string>();
     SPContentType ct = web.ContentTypes[contentTypeName];
     foreach (SPFieldLink link in ct.FieldLinks) {
      contentTypeNames.Add( link.Name );
     }
     List<string> fieldNames = new List<string>();
     foreach (SPField field in web.Fields) { // <!-- this is where exception gets thrown!
      fieldNames.Add( field.Title );
     }
     // delete the column from all content types
     ct.FieldLinks.Delete( customColumnName );
     ct.Update( true );
     // delete the site column / field
     web.Fields.Delete( customColumnName );
     web.Update();
    } // using SPWeb
   } // using SPSIte

 
In fact, as it turns out, my first approach was totally wrong, because I was trying to use the web service or API to simply remove the offending site column since I didn't want to keep it anyway. I was able to get rid of the reference to the column in the content type, but SPWeb.Fields.Delete("FieldName") throws the same error I describe above, so you can't actaully delete the column as long as it is still in a broken state. Nice!
 
But, now you will have the name of the field type name that can't be found - and hopefully some ability to stitch togehter a FLDTYPES.xml field to match it, either using your own control or one of the base field controls from SharePoint. Having corrected the issue you should be able to use the pages on Site Settings to delete the offending column and/or remove it from any content types.
 
However, bear in mind that if it is in use in any lsits, then it will be removed and any data housed there will be lost. If you need to keep you data you will either have to find a way to move it into another field or else leave the naming conventions the way they were.
 
What I Learned About SharePoint from This Mess
About the most useful thing I learned today is that it's about 1,000 times easier to make programmatic changes to columns and content types if you use the APU than it is using the web services. Sure, the web services are there and they have their uses I suppose. But, you'll have to learn to write some CAML and get used to parsing your results using XPath queries, because the results don't deserialize (at least not easily).
 
I wrote a nifty applet that can connect to a web site using the Webs.asmx server and let you browse site columns and even select one for deletion. Getting the UI populated was a chore, because the XML that the web services return needed to be massaged to strip the namespaces from it so that SelectNodes would actually work. (It was either that or implemenet a NamespaceManager, which let's just say wasn't going well.) In the end, though I got all the code written to pick a column from a column group and then delete it, it didn't work anyway! This was  because of the issue that caused the exception I describe above, I guess. But, crossing the web service the only error information you will see is "800040005 Operation Failed". Sadly, that's not very useful.
 
So, in the end using the web service to do this type of administrative work is probably not worth it. The only real benefit you'd get from a lot of extra effort is that your admin tool would be able to connect to services across the internet even if they haven't installed anything on the server side. If you have control over the server environemnt it would probably be much better to write your own easy to use web service wrapped around the API functions you want to implement.
 
Another thing that I learned is that the SharePoint web services still largely use CAML passed as either strings or untyped XmlNode objects. These are somewhat difficult to work with, and I will have to do some research into finding out how they can either be serialized/deserialized into useful objects in >NET or otherwise made easier to handle. On the birght side, I wrote a useful serialization tool for SPQuery CAML code some time ago and it looks liekt hat will probably still be relevant and I should go dust it off. :-)
 
Finally, I leanred a lot about the naming conventions for columns and content types as they realte to the API, the web service, and the terminology on the web site itself. This can be pretty confusing, and there's not a lot of good documentation out there. But, it's a little out of scope for today and I'll just have to put together another article for that topic some other time.
 
Now, it's off to New York to see my in laws and hopefully spend a nice weekend with my head buried in my laptop and some comic books.
 
More Information:
Careful When Working with Sealed Site Columns
MSDN: SPFieldLinkCollection.Delete Method
MSDN: Webs.UpdateContentType Method
MOSS 2007: WSS 3.0: How to add/delete/update site columns by using SharePoint WebService
Tags: Architecture, studies, Tips, coding, sharepoint

In a word: BE CAREFUL.
 
Okay, that is actually two words. :-)
 
Just be careful about the naming conventions that you use and how you change them later. If you need to uninstall / reinstall a custom field and control, you will likely have problems with the lists that were built on top of it. You will also have problems with the SharePoint Site Column and Content Type management pages. Needless to say, this is NOT good.
 
Fortunately, it looks like the fields can be manually deleted using the SharePoint web services. I imagine it is also possible to do this from stsadm, but I never much liked command line tools, so I am trying now to create a little Windows Forms application that will do the job for me.
 
I will post my code back here when I get it stable. For now, just be careful about changing or removing custom field names and namespaces, etc.
 
Update:
 
Okay, there is only a very small amount of useful stuff online at this point, so I will try to be as specific as I can - for a Friday evening.
 
My Issue and Its Root Cause
 
Create a custom field control and field type and install them to SharePoint either by hand, with an MSI, or a WSP. In particular, I want to focus on the FLDTYPES_xyz.xml file (or whatever you decide to call yours). This is the file that tells SharePoint how to make use of your custom field control.
 
Now, suppose you decide later that you don't like the name of one of your fields. Maybe you mis-spelled it, or maybe (like me) you just decided that you didn't want there to be any naming ambiguity between your custom fields and those that other developers might provide. So, you rename your field in the definition. You think it will be okay because you are going to uninstall and reinstall your solution anyway.
 
Here is the gotcha. If you have used your custom field in any list, then a reference to it will have been added to the SPWebs.Fields collection, which is a master list of all the field definitions that are used for your entire web site. Additionally, if you have added it to any of your content types, then a reference will exist in the form of a SPFieldLink in the SPWeb.ContentTypes[x].FieldLinks collection.
 
If you remove the FLDTYPES.xml file that these references are relying upon, or change the names in it, then you'll see some pretty interesting behavior out of SharePoint! Hopefully what I am posting here will save somebody the entire workday that this issue cost me.
 
What manifests is that your lists, the content types admin web page, and the site columns admin web page will all be broken by this change. In particular, the site columns page will give an error:
The given key was not present in the dictionary.
Very helpful indeed - as helpful approaches zero. The other pages will have other cryptic messages as well, but I did not capture them here. Maybe some time I will go back and repro this error so I can take some screen shots. Sadly, nothing I did to the CustomErrors settings on my server allowed me to see a stack trace.
 
What you can do to fix this behavior is change the TypeName property back to what it was in your XML file - assuming you can remember what it was.
 
<Field Name="TypeName">MyOldTypeName</Field>
 
If you can't remember the name you used that is causing you the problem, you can use the SharePoint API to find it.
 
Simply open your web site with an SPWeb object, then attempt to iterate through the SPWeb.Fields collection. You will immediately receive an exception to the effect of something like "FieldType named 'x' is not properly installed. Go to the Site Collection page in Site Settings to correct the problem." Of course this is bollocks, because the web page is broken by this problem.
 
   string customColumnName = "My_x0020_Custom_x0020_Column";
   string contentTypeName = "MyContentType";
   using (SPSite site = new SPSite( "http://spdev" )) {
    using (SPWeb web = site.OpenWeb( "/site1" )) {
     // just some useful code for getting the lsit of names
     List<string> contentTypeNames = new List<string>();
     SPContentType ct = web.ContentTypes[contentTypeName];
     foreach (SPFieldLink link in ct.FieldLinks) {
      contentTypeNames.Add( link.Name );
     }
     List<string> fieldNames = new List<string>();
     foreach (SPField field in web.Fields) { // <!-- this is where exception gets thrown!
      fieldNames.Add( field.Title );
     }
     // delete the column from all content types
     ct.FieldLinks.Delete( customColumnName );
     ct.Update( true );
     // delete the site column / field
     web.Fields.Delete( customColumnName );
     web.Update();
    } // using SPWeb
   } // using SPSIte

 
In fact, as it turns out, my first approach was totally wrong, because I was trying to use the web service or API to simply remove the offending site column since I didn't want to keep it anyway. I was able to get rid of the reference to the column in the content type, but SPWeb.Fields.Delete("FieldName") throws the same error I describe above, so you can't actaully delete the column as long as it is still in a broken state. Nice!
 
But, now you will have the name of the field type name that can't be found - and hopefully some ability to stitch togehter a FLDTYPES.xml field to match it, either using your own control or one of the base field controls from SharePoint. Having corrected the issue you should be able to use the pages on Site Settings to delete the offending column and/or remove it from any content types.
 
However, bear in mind that if it is in use in any lsits, then it will be removed and any data housed there will be lost. If you need to keep you data you will either have to find a way to move it into another field or else leave the naming conventions the way they were.
 
What I Learned About SharePoint from This Mess
About the most useful thing I learned today is that it's about 1,000 times easier to make programmatic changes to columns and content types if you use the APU than it is using the web services. Sure, the web services are there and they have their uses I suppose. But, you'll have to learn to write some CAML and get used to parsing your results using XPath queries, because the results don't deserialize (at least not easily).
 
I wrote a nifty applet that can connect to a web site using the Webs.asmx server and let you browse site columns and even select one for deletion. Getting the UI populated was a chore, because the XML that the web services return needed to be massaged to strip the namespaces from it so that SelectNodes would actually work. (It was either that or implemenet a NamespaceManager, which let's just say wasn't going well.) In the end, though I got all the code written to pick a column from a column group and then delete it, it didn't work anyway! This was  because of the issue that caused the exception I describe above, I guess. But, crossing the web service the only error information you will see is "800040005 Operation Failed". Sadly, that's not very useful.
 
So, in the end using the web service to do this type of administrative work is probably not worth it. The only real benefit you'd get from a lot of extra effort is that your admin tool would be able to connect to services across the internet even if they haven't installed anything on the server side. If you have control over the server environemnt it would probably be much better to write your own easy to use web service wrapped around the API functions you want to implement.
 
Another thing that I learned is that the SharePoint web services still largely use CAML passed as either strings or untyped XmlNode objects. These are somewhat difficult to work with, and I will have to do some research into finding out how they can either be serialized/deserialized into useful objects in >NET or otherwise made easier to handle. On the birght side, I wrote a useful serialization tool for SPQuery CAML code some time ago and it looks liekt hat will probably still be relevant and I should go dust it off. :-)
 
Finally, I leanred a lot about the naming conventions for columns and content types as they realte to the API, the web service, and the terminology on the web site itself. This can be pretty confusing, and there's not a lot of good documentation out there. But, it's a little out of scope for today and I'll just have to put together another article for that topic some other time.
 
Now, it's off to New York to see my in laws and hopefully spend a nice weekend with my head buried in my laptop and some comic books.
 
More Information:
Careful When Working with Sealed Site Columns
MSDN: SPFieldLinkCollection.Delete Method
MSDN: Webs.UpdateContentType Method
MOSS 2007: WSS 3.0: How to add/delete/update site columns by using SharePoint WebService
Tags: Architecture, studies, Tips, coding, sharepoint