Appboard/2.4/builder/data sources/web service

Revision as of 16:44, 4 October 2013 by imported>David.moore (→‎Connecting with XML XSLT)

There are several web service data adapters available to choose from in AppBoard: CSV WebService, JSON WebService, and XML XSLT. As each type of data adapter will require different information to connect to AppBoard, this page will assist you in understanding some of the specific options associated with web service data adapters.


CSV WebService

This is a web services adapter that uses CSV to communicate with an application server.

Connecting with CSV WebService

The Data Source Wizard begins with the "Connect" step. Here you must fill in the necessary information to connect AppBoard to your CSV WebService data source. These are the possible elements you will need to provide:


Name Description
URL Base URL for the web service from which to fetch data. This can also be the complete URL, if you want to include any needed parameters here instead of as separate settings. Sample CSV files have been included with this distribution and are located within various folders under ${application.home}/data/. On a Windows, these can be accessed using a URL like file:///${application.home}/data/path_to_CSV (on UNIX, the URL would start as file://). Alternatively, the sample CSV files can be copied to a custom folder in the enportal directory and accessed via http://host:port/enportal/custom_folder/path_to_CSV.
HTTP Auth Username Username to use for HTTP Auth (Basic, Digest, etc.). Ignored if HTTP Auth is not needed.
HTTP Auth Password Password to use for HTTP Auth (Basic, Digest, etc.). Ignored if HTTP Auth is not needed.
Cache Timeout Time limit before reloading results from the database.
HTTP Headers Optional HTTP headers to set on request.
HTTP Post Body Optional POST body to send with request. If set, the request will be made via a POST; if not, the HTTP request will be a GET request.
Contains Header Line Whether the data includes a header to define the columns/field names.
Header Meta Delimiter Whether the header includes a delimiter.
Data Delimiter Whether the data includes a delimiter to define the columns/field names.
String Boundaries Whether the data fields should be written within quotes.
Proxy URL Optional URL for web proxy.
Proxy Username Optional Proxy Username.
Proxy Password Optional Proxy Password.
Use NTLM Authentication Optional Use NTLM Auth for Proxy.
Proxy Domain Optional Proxy NT Domain.


JSON Web Service

This data source adapter is for web services that provide JavaScript Object Notation (JSON) formatted responses.

Connecting with JSON Web Service

The Data Source Wizard begins with the "Connect" step. Here you must fill in the necessary information to connect AppBoard to your JSON data source. These are the possible elements you will need to provide:

Name Description
URL Base URL for the web service from which to fetch data. This can also be the complete URL, if you want to include any needed parameters here instead of as separate settings. Although the distribution itself does not include any sample JSON files, the following section in the documentation includes links to sample JSON files that demonstrate valid formatting. Any JSON files placed on the AppBoard server can go either within ${application.home}/data/ or within a custom folder in the enportal directory. Just like with the CSV files, these can be accessed using URLs like file:///${application.home}/data/path_to_JSON or http://host:port/enportal/custom/path_to_JSON.
HTTP Auth Username Username to use for HTTP Auth (Basic, Digest, etc.). Ignored if HTTP Auth is not needed.
HTTP Auth Password Password to use for HTTP Auth (Basic, Digest, etc.). Ignored if HTTP Auth is not needed.
Cache Timeout Time limit before reloading results from the database.
HTTP Headers Optional HTTP headers to set on request.
HTTP Post Body Optional POST body to send with request. If set, the request will be made via a POST, if not, the HTTP request will be a GET request.
Proxy URL Optional URL for web proxy.
Proxy Username Optional Proxy Username.
Proxy Password Optional Proxy Password.
Use NTLM Authentication Optional Use NTLM Auth for Proxy.
Proxy Domain Optional Proxy NT Domain.


Valid JSON Formats

JSON can be used to represent complex data structures using a combination of arrays, objects, and key/value pairs. While this flexibility is good in general, for AppBoard the data source adapter needs to understand and convert this into one or more entities with a set number of columns. The following formats deliver the best results:

  • An array of objects (see the example data.json)
  • An object that contains an array of objects (see services.json)
  • An object containing a variety of "key":value fields, with the fields consisting of strings, numbers, boolean values, arrays, or objects (see glossary.json, person.json, and widget.json).

Some additional notes:

  • root level key/value pairs may cause parsing issues.
  • named objects within objects will be created as a separate entity, it's important that each of these named objects have a consistent set of keys.

XML XSLT

This adapter lets you get data from an XML web service and transform it into records that are in a format applicable to AppBoard.


Connecting with XML XSLT

The Data Source Wizard begins with the "Connect" step. Here you must fill in the necessary information to connect AppBoard with your XML XSLT data source. These are the possible elements you will need to provide:


Name Description
URL This is the location of the information file used for your data source. Included in the distribution are some sample XML files which can be accessed by using file:/// (on Windows) or file:// (on UNIX) url. For example, on Windows it would look like file:///${application.home}/data/pkg/sample/atlas/atlas.xml and on UNIX it would look like file://${application.home}/data/pkg/sample/atlas/atlas.xml. Please see http://en.wikipedia.org/wiki/File_URI_scheme for more details on using file:/// (or file://) urls. Normally you will be using http:// to reference the RESTful WebService.
Style Sheet Path The path from ${application.home}/stylesheets/ of the XSL stylesheet to be used to convert the data from the web service into the AppBoard internal XML representation. Note that the stylesheet must be on the AppBoard server as importing stylesheets from foreign servers is not currently supported.
HTTP Auth Username Username to use for HTTP Auth (Basic, Digest, etc.). Ignored if HTTP Auth is not required.
HTTP Auth Password Password to use for HTTP Auth (Basic, Digest, etc.). Ignored if HTTP Auth is not required.
Cache Timeout Time limit before re-loading results from the RESTful web service. Note: if a timeout is not specified, it will calculate the timeout based on the timeout specfied by the response header. If the request should be run on every request, then a value of '0' should be specified.
HTTP Headers Optional HTTP headers to set on request.
HTTP Post Body Optional POST body to send with request. If set, the request will be made via a POST; if not, the HTTP request will be a GET request.
Proxy URL Optional URL for web proxy.
Proxy Username Optional Proxy Username.
Proxy Password Optional Proxy Password.
Use NTLM Authentication Optional Use NTLM Auth for Proxy.
Proxy Domain Optional Proxy NT Domain.

XSLT Tutorial

Using ${application.home}/data/pkg/sample/hierarchy/hierarchy.xml as an example, this tutorial builds the XSL Transformation in a step-by-step manner. It will be helpful in following the tutorial to have the source XML and the target XML in separate windows or printed out.

Note: Included with the distribution under ${application.home}/stylesheets/ are some example XSL files that correspond to a variety of sample XML files included under ${application.home}/data/pkg/sample/. You can also use these as guides when you create your own XSL files. To see the result of a transformation (and to help debug custom XSL files), you can use the 'Transform' option when running portal.bat or portal.sh. To do this, in the command line navigate to <Path_to_AppBoard_Server_Parent_Directory>/server/bin/ and run (without quotes) 'portal Transform' to display the possible arguments you can use when doing an XML/XSL transform. An example of a standard command is 'portal Transform -in example.xml -xsl example.xsl -out output.xml', which will take the input XML file and the indicated XSL file, do a transformation, and output the resulting XML to output.xml

1. Review the basic structure of your source XML.

This example consists of a single root node ("rspec") with some descriptive child elements one of which ("computeResource") contains nested information on compute nodes which contain further information including network interfaces. In outline form:
  • rspec
    • aggregate
    • description
    • lifetime
    • computeResource
      • node
        • networkInterface
      • node
        • networkInterface
      • node
        • networkInterface

2. Identify the data elements you want to transform into AppBoard Entities.

In this case, we are interested in nodes and networkInterfaces which we will call Nodes and NICs.

3. Begin the XSLT with the xsl:stylesheet element and an xsl:output element:

[xml,N] <?xml version="1.0" encoding="UTF-8"?> <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns="http://www.w3.org/1999/xhtml" version="1.0"> 

   <xsl:output method="xml" indent="yes" encoding="UTF-8"/>

</xsl:stylesheet> 4. Identify an element that has only a single instance and is the parent of all of the data elements you wish to capture as Entities. In many cases, that would be the root element ("rspec" in this example), but because there is only one "computeResource" instance we can reference it directly as "/rspec/computeResource".

5. Add a match template for that data element that emits the "result" element and definitions for the entities we will capture:

[xml,N] <?xml version="1.0" encoding="UTF-8"?> <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns="http://www.w3.org/1999/xhtml" version="1.0"> 

   <xsl:output method="xml" indent="yes" encoding="UTF-8"/>
   <xsl:template match="/rspec/computeResource">
       <result xmlns="http://www.edgeti.com/xml-data">
           <entity name="Nodes">
               <attributes>
                   <attribute primaryKey="true">NodeId</attribute>
                   <attribute>Address</attribute>
               </attributes>
           </entity>
           <entity name="NICs">
               <attributes>
                   <attribute primaryKey="true">NICId</attribute>
                   <attribute>Address</attribute>
               </attributes>
           </entity>
       </result>            
   </xsl:template>

</xsl:stylesheet>

This only works for single instance elements. If there were multiple "computeResource" elements we would start with "/rspec" even if we weren't capturing an Entity corresponding to "computeResource" instances.

6. Add do-nothing templates for the peers of "computeResource" that we plan to ignore:

[xml,N] <xsl:template match="/rspec/aggregate"/> <xsl:template match="/rspec/description"/> <xsl:template match="/rspec/lifetime"/> 7. Add a match template for the "node" elements to construct "record" elements for each instance:

[xml,N] <template match="node"> 

   <record>
       <value name="NodeId"><value-of select="@id"/></value>
       <value name="Address"><value-of select="address"/></value>
   </record>

</template>

Note the use of "@id" to extract an attribute of the "node" element and the use of "address" to extract the value in a child element.

8. Add a match template for the "networkInterface" elements. Note that we use "node/networkInterface" since that is the relative "path" from our "computeResource" element.

[xml,N] <xsl:template match="node/networkInterface"> 

   <record>
       <value name="NICId"><xsl:value-of select="@id"/></value>
       <value name="Address"><xsl:value-of select="ipAddress"/></value>
       <value name="NodeId"><xsl:value-of select="../@id"/></value>
   </record>

</xsl:template> 9. Place these after the template for "computeResource" and then add xsl:apply-templates elements to reference them to construct children of the "entity" elements in the output:

[xml,N] <?xml version="1.0" encoding="UTF-8"?> <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns="http://www.w3.org/1999/xhtml" version="1.0"> 

   <xsl:output method="xml" indent="yes" encoding="UTF-8"/>
   <xsl:template match="/rspec/computeResource">
       <result xmlns="http://www.edgeti.com/xml-data">
           <entity name="Nodes">
               <attributes>
                   <attribute primaryKey="true">NodeId</attribute>
                   <attribute>Address</attribute>
               </attributes>
               <xsl:apply-templates select="node"/>
           </entity>
           <entity name="NICs">
               <attributes>
                   <attribute primaryKey="true">NICId</attribute>
                   <attribute>Address</attribute>
               </attributes>
               <xsl:apply-templates select="node/networkInterface"/>
           </entity>
       </result>            
   </xsl:template>
   <xsl:template match="/rspec/aggregate"/>
   <xsl:template match="/rspec/description"/>
   <xsl:template match="/rspec/lifetime"/>
   <template match="node">
       <record>
           <value name="NodeId"><value-of select="@id"/></value>
           <value name="Address"><value-of select="address"/></value>
       </record>
   </template>
   <xsl:template match="node/networkInterface">
       <record>
           <value name="NICId"><xsl:value-of select="@id"/></value>
           <value name="Address"><xsl:value-of select="ipAddress"/></value>
       </record>
   </xsl:template>

</xsl:stylesheet>

Now we have a working XSL Transform that defines our two entities and constructs all of the records. But, the hierarchy of the data gives us an association that we aren't capturing.

10. Define the association in the "Nodes" entity:

[xml,N] <associations> 

   <association fromKey="NodeId" toEntity="NICs" toKey="NodeId"/>

</associations> 11. Add an attribute to "NICs" to store the key:

[xml,N] <entity name="NICs"> 

   <attributes>
       <attribute primaryKey="true">NICId</attribute>
       <attribute>Address</attribute>
       <attribute>NodeId</attribute>
   </attributes>

</entity> 12. Use "../@id" in an xsl:value-of to add a "NodeId" value to the instances of "NICs".

[xml,N] <xsl:template match="node/networkInterface"> 

   <record>
       <value name="NICId"><xsl:value-of select="@id"/></value>
       <value name="Address"><xsl:value-of select="ipAddress"/></value>
       <value name="NodeId"><xsl:value-of select="../@id"/></value>
   </record>

</xsl:template> Resulting XSL:

[xml,N] <?xml version="1.0" encoding="UTF-8"?> <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns="http://www.w3.org/1999/xhtml" version="1.0"> 

   <xsl:output method="xml" indent="yes" encoding="UTF-8"/>
   <xsl:template match="/rspec/computeResource">
       <result xmlns="http://www.edgeti.com/xml-data">
           <entity name="Nodes">
               <attributes>
                   <attribute primaryKey="true">NodeId</attribute>
                   <attribute>Address</attribute>
               </attributes>
               <associations>
                   <association fromKey="NodeId" toEntity="NICs" toKey="NodeId"/>
               </associations>
               <xsl:apply-templates select="node"/>
           </entity>
           <entity name="NICs">
               <attributes>
                   <attribute primaryKey="true">NICId</attribute>
                   <attribute>Address</attribute>
                   <attribute>NodeId</attribute>
               </attributes>
               <xsl:apply-templates select="node/networkInterface"/>
           </entity>
       </result>            
   </xsl:template>
   <xsl:template match="/rspec/aggregate"/>
   <xsl:template match="/rspec/description"/>
   <xsl:template match="/rspec/lifetime"/>
   <template match="node">
       <record>
           <value name="NodeId"><value-of select="@id"/></value>
           <value name="Address"><value-of select="address"/></value>
       </record>
   </template>
   <xsl:template match="node/networkInterface">
       <record>
           <value name="NICId"><xsl:value-of select="@id"/></value>
           <value name="Address"><xsl:value-of select="ipAddress"/></value>
           <value name="NodeId"><xsl:value-of select="../@id"/></value>
       </record>
   </xsl:template>

</xsl:stylesheet>

Retrieved from "http://ab.edge-technologies.com/docs/index.php?title=Appboard/2.4/builder/data_sources/web_service&oldid=4074"