Appboard/2.4/builder/data sources/web service: Difference between revisions

imported>Jason.nicholls
imported>Mike.berman
No edit summary
 
(38 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{DISPLAYTITLE:Web Service Data Adapters}}
{{DISPLAYTITLE:Web Service Data Adapters}}
There are several web service data adapters available to choose from in AppBoard: CSV WebService, JSON WebService, and XML XSLT. As each type of data adapter will require different information to connect to AppBoard, this page will assist you in understanding some of the specific options associated with web service data adapters.  
[[Category:AppBoard 2.4]]
There are several web service data adapters available to choose from in AppBoard: CSV Web, JSON Web, and XML XSLT Web. As each type of data adapter will require different information to connect to AppBoard, this page will assist you in understanding some of the specific options associated with web service data adapters.  




==CSV WebService==
==CSV Web==




This is a web services adapter that uses CSV to communicate with an application server.  
This data source adapter is for web services that provide Comma Separated Value (CSV) formatted responses.  


===Connecting with CSV WebService===
===Connecting with CSV Web===




The Data Source Wizard begins with the "Connect" step. Here you must fill in the necessary information to connect AppBoard to your CSV WebService data source. These are the possible elements you will need to provide:
The Data Source Wizard begins with the "Connect" step. Here you must fill in the necessary information to connect AppBoard to your CSV Web data source. These are the possible elements you will need to provide:






{|class="wikitable"
{|class="wikitable" border="1" style="background-color:#eeeeee;"
!Name
!Name
!Description
!Description
|-
|-
|'''url'''
|'''URL'''
|Base URL for the web service from which to fetch data. This can also be the complete URL, if you want to include any needed parameters here instead of as separate settings.  
|Base URL for the web service from which to fetch data. This can also be the complete URL, if you want to include any needed parameters here instead of as separate settings. Sample CSV files have been included with this distribution and are located within various folders under <tt>${application.home}/data/</tt>. On a Windows server, these can be accessed using a URL like file:///${application.home}/data/path_to_CSV (on UNIX, the URL would start as file://).  Please see http://en.wikipedia.org/wiki/File_URI_scheme for more details on using file:/// (or file://) URLs. Alternatively, the sample CSV files can be copied to a custom folder in the enportal directory and accessed via http://host:port/enportal/custom_folder/path_to_CSV.<br><br>Example URL: <tt>file:///${application.home}/data/pkg/appboard/test/timeline_sample_data.csv</tt>
|-
|-
|'''httpAuthUsername'''
|'''HTTP Auth Username'''
|Username to use for HTTP Auth (Basic, Digest, etc.). Ignored if HTTP Auth is not needed.  
|Username to use for HTTP Auth (Basic, Digest, etc.). Ignored if HTTP Auth is not needed.  
|-
|-
|'''httpAuthPassword'''
|'''HTTP Auth Password'''
|Password to use for HTTP Auth (Basic, Digest, etc.). Ignored if HTTP Auth is not needed.  
|Password to use for HTTP Auth (Basic, Digest, etc.). Ignored if HTTP Auth is not needed.  
|-
|-
|'''cacheTimeout'''
|'''HTTP Headers'''
|Time limit before reloading results from the database.
|-
|'''httpHeaders'''
|Optional HTTP headers to set on request.
|Optional HTTP headers to set on request.
|-
|-
|'''httpPostBody'''
|'''HTTP Post Body'''
|Optional POST body to send with request. If set, the request will be made via a POST; if not, the HTTP request will be a GET request.  
|Optional POST body to send with request. If set, the request will be made via a POST; if not, the HTTP request will be a GET request.  
|-
|-
|'''headerLine'''
|'''Cache Timeout'''
|Whether the data includes a header to define the columns/field names.  
|Time limit before reloading results from the database.  
|-
|-
|'''delimiter.data'''
|'''Data Delimiter'''
|Whether the data includes a delimiter to define the columns/field names.  
|Whether the data includes a delimiter to define the columns/field names.  
|-
|-
|'''useQuoteChar'''
|'''Contains Header Line'''
|Defines whether the response contains a header line, which should be the first row. If no header is present the first row returned in discovery will be used to define the initial field names. Note: in order for query SHIMs to work then a header line is required.
|-
|'''Header Meta Delimiter'''
|Whether the header includes a delimiter.
|-
|'''String Boundaries'''
|Whether the data fields should be written within quotes.  
|Whether the data fields should be written within quotes.  
|-
|-
|'''proxyurl'''
|'''Proxy URL'''
|Optional URL for web proxy.
|Optional URL for web proxy.
|-
|-
|'''proxyuser'''
|'''Use NTLM Authentication'''
|Optional Proxy Username.
|Optional Use NTLM Auth for Proxy.
|-
|'''proxypass'''
|Optional Proxy Password.
|-
|-
|'''proxyntlm'''
|'''Proxy Domain'''
|Optional Proxy NT Domain.
|Optional Proxy NT Domain.
|-
|-
|'''proxydomain'''
|'''Proxy Username'''
|Optional Use NTLM Auth for Proxy.
|Optional Proxy Username.
|-
|-
|'''Proxy Password'''
|Optional Proxy Password.
|}
|}


==JSON Web==


==JSON Web Service==
This data source adapter is for web services that provide JavaScript Object Notation (JSON) formatted responses.
 
This data source adapter is for web services that provideJavaScript Object Notation (JSON) formatted responses.


===Connecting with JSON Web Service===
===Connecting with JSON Web===


The Data Source Wizard begins with the "Connect" step. Here you must fill in the necessary information to connect AppBoard to your JSON data source. These are the possible elements you will need to provide:
The Data Source Wizard begins with the "Connect" step. Here you must fill in the necessary information to connect AppBoard to your JSON data source. These are the possible elements you will need to provide:


{|class="wikitable"
{|class="wikitable" border="1" style="background-color:#eeeeee;"
!Name
!Name
!Description
!Description
|-
|-
|'''url'''
|'''URL'''
|Base URL for the web service from which to fetch data. This can also be the complete URL, if you want to include any needed parameters here instead of as separate settings.  
|Base URL for the web service from which to fetch data. This can also be the complete URL, if you want to include any needed parameters here instead of as separate settings. The distribution contains sample JSON files located within <tt>${application.home}/data/JSON/</tt>. The following section in the documentation also includes links to copies of these sample JSON files which demonstrate valid formatting.  Any JSON files placed on the AppBoard server can go either within <tt>${application.home}/data/</tt> or within a custom folder in the enportal directory.  Just like with the CSV files, these can be accessed using URLs like file:///${application.home}/data/path_to_JSON or http://host:port/enportal/custom/path_to_JSON.  <br><br>Example URL: <tt>file:///${application.home}/data/JSON/data.json</tt>
|-
|-
|'''httpAuthUsername'''
|'''HTTP Auth Username'''
|Username to use for HTTP Auth (Basic, Digest, etc.). Ignored if HTTP Auth is not needed.  
|Username to use for HTTP Auth (Basic, Digest, etc.). Ignored if HTTP Auth is not needed.  
|-
|-
|'''httpAuthPassword'''
|'''HTTP Auth Password'''
|Password to use for HTTP Auth (Basic, Digest, etc.). Ignored if HTTP Auth is not needed.  
|Password to use for HTTP Auth (Basic, Digest, etc.). Ignored if HTTP Auth is not needed.  
|-
|-
|'''cacheTimeout'''
|'''HTTP Headers'''
|Time limit before reloading results from the database.
|-
|'''httpHeaders'''
|Optional HTTP headers to set on request.
|Optional HTTP headers to set on request.
|-
|-
|'''httpPostBody'''
|'''HTTP Post Body'''
|Optional POST body to send with request. If set, the request will be made via a POST, if not, the HTTP request will be a GET request.  
|Optional POST body to send with request. If set, the request will be made via a POST, if not, the HTTP request will be a GET request.  
|-
|-
|'''proxyurl'''
|'''Cache Timeout'''
|Time limit before reloading results from the database.
|-
|'''Proxy URL'''
|Optional URL for web proxy.
|Optional URL for web proxy.
|-
|-
|'''proxyuser'''
|'''Use NTLM Authentication'''
|Optional Proxy Username.
|Optional Use NTLM Auth for Proxy.
|-
|'''proxypass'''
|Optional Proxy Password.
|-
|-
|'''proxyntlm'''
|'''Proxy Domain'''
|Optional Proxy NT Domain.
|Optional Proxy NT Domain.
|-
|-
|'''proxydomain'''
|'''Proxy Username'''
|Optional Use NTLM Auth for Proxy.
|Optional Proxy Username.
|-
|-
|'''Proxy Password'''
|Optional Proxy Password.
|}
|}


===Valid JSON Formats===


===Proper JSON File Formatting===
JSON can be used to represent complex data structures using a combination of arrays, objects, and key/value pairs. While this flexibility is good in general, for AppBoard the data source adapter needs to understand and convert this into one or more entities with a set number of columns. The following formats deliver the best results:
 
 
In JSON files, square brackets "[]" indicate arrays, curly braces "{}" indicate objects, and fields are defined by "key":value, where value can be a string, number, boolean, array, or object.
 
 
The following formats are accepted:


* An array of objects (see the example [[/data_json|data.json]])
* An array of objects (see the example [[/data_json|data.json]]).
* An object that contains an array of objects (see [[/services_json|services.json]])
* An object that contains an array of objects (see [[/services_json|services.json]]).
* An object containing a variety of "key":value fields, with the fields consisting of strings, numbers, boolean values, arrays, or objects (see [[/glossary_json|glossary.json]], [[/person_json|person.json]], and [[/widget_json|widget.json]]).
* An object containing a variety of "key":value fields, with the fields consisting of strings, numbers, boolean values, arrays, or objects (see [[/glossary_json|glossary.json]], [[/person_json|person.json]], and [[/widget_json|widget.json]]).


Some additional notes:


If the file is an array of objects (e.g. [[/data_json|data.json]]) or an object that contains an array of objects (e.g. [[/services_json|services.json]]), it is recommended that each object within that array have a consistent set of "keys".  Changing the number of keys or using different keys from one object to the next within the array may adversely affect the parsing of the data.
* Root level key/value pairs may cause parsing issues.
 
* Named objects within objects will be created as a separate entity, so it's important that each of these named objects have a consistent set of keys.
 
For example, do not do:
 
 
<nowiki>
{ "people" : [
{ "name" : "John Doe",
"height" : "5ft 11in",
"weight" : "150 lbs",
"age" : 22,
"Job" : "Police Officer" }
{ "name" : "Jane Doe",
"height" : "5ft 6in",
"age" : 19,
"Occupation" : "Student" }
]
}</nowiki>
 
 
Notice that "Jane Doe" does not have a "weight" key and has an "Occupation" key instead of "Job".  If certain information cannot be provided for a particular object but it can for another, then it is recommended that for the object without said information, include the key but give it an empty value.  Remember to keep the keys consistent.
 
 
i.e., do the following:
 
 
<nowiki>
{ "people" : [
{ "name" : "John Doe",
"height" : "5ft 11in",
"weight" : "150 lbs",
"age" : 22,
"Job" : "Police Officer" }
{ "name" : "Jane Doe",
"height" : "5ft 6in",
"weight" : "",
"age" : 19,
"Job" : "Student" }
]
}</nowiki>
 
 
Lastly, the following format has been known to cause parsing issues:
 
 
<nowiki>
{ "key1" : value, (string/number/boolean)
"key2" : value, (string/number/boolean)
...
"keyN" : value, (string/number/boolean)
"arrayOrObjectKey" : [ (or "arrayOrObjectKey" : {)
...
] (or })
...
}</nowiki>
 
 
Notice the "key" : value pairs (that are string, number, or boolean) preceding the array or object.  The parser is known to have issues with this style of formatting and it is therefore recommended that the "key" : value pairs are encapsulated within a named object.
 
 
i.e.:
 
 
<nowiki>
{ "namedObject" : {
"key1" : value, (string/number/boolean)
"key2" : value, (string/number/boolean)
...
"keyN" : value, (string/number/boolean)
"arrayOrObjectKey" : [ (or "arrayOrObjectKey" : {)
...
] (or })
...
}
}</nowiki>
 
==XML XSLT==


==XML XSLT Web==


This adapter lets you get data from an XML web service and transform it into records that are in a format applicable to AppBoard.
The XML Web Service Adapter supports data retrieval from HTTP REST services. XML data returned from the REST service is transformed to the AppBoard internal data structure using XSLT. Hierarchical relationships between XML data elements can be represented within AppBoard through associations within the AppBoard data model.


This adapter can also be used to access some SOAP web service calls where there is a single call and response.




===Connecting with XML XSLT===
===Connecting with XML XSLT===


The Data Source Wizard begins with the "Connect" step. Here you must fill in the necessary information to connect AppBoard with your XML XSLT data source. These are the possible elements you will need to provide:


The Data Source Wizard begins with the "Connect" step.  Here you must fill in the necessary information to connect AppBoard with your XML XSLT data source.  These are the possible elements you will need to provide:
{|class="wikitable" border="1" style="background-color:#eeeeee;"
 
 
 
{|class="wikitable"
!Name
!Name
!Description
!Description
|-
|-
|'''url'''
|'''URL'''
|This is the location of the information file used for your data source. Included in the distribution are some sample xml files which can be accessed by using the 'file://' url. For example, on windows it would look like 'file://localhost/${application.home}/data/pkg/sample/atlas/atlas.xml' and on UNIX it would look like: 'file://${application.home}/data/pkg/sample/atlas/atlas.xml'. Please see http://en.wikipedia.org/wiki/File_URI_scheme for more details on using file:// urls; although you will normally be using http:// to reference the RESTful WebService.
|This is the location of the information file used for your data source. Included in the distribution are some sample XML files which can be accessed by using file:/// (on Windows) or file:// (on UNIX) URL. For example, on Windows it would look like file:///${application.home}/data/pkg/sample/atlas/atlas.xml and on UNIX it would look like file://${application.home}/data/pkg/sample/atlas/atlas.xml. As with the CSV and JSON adapters, XML files can also be placed within a custom folder in the enportal directory and accessed via http://host:port/enportal/custom_folder/path_to_XML. Normally you will be using http:// to reference the RESTful Web service.<br><br>Example URL: <tt>file:///${application.home}/data/pkg/sample/atlas/atlas.xml</tt><br>Example Stylesheet: <tt>${application.home}/stylesheets/sample/atlas-to-adapter.xsl</tt>
|-
|'''stylesheetPath'''
|The path from '${application.home}/stylesheets/' of the XSL stylesheet to be used to convert the data from the web service into the AppBoard internal XML representation.
|-
|-
|'''httpAuthUsername'''
|'''Style Sheet Path'''
|Username to use for HTTP Auth (Basic, Digest, etc.). Ignored if HTTP Auth is not required.
|The path from <tt>${application.home}/stylesheets/</tt> of the XSL stylesheet to be used to convert the data from the web service into the AppBoard internal XML representation. Note that the stylesheet must be on the AppBoard server as importing stylesheets from foreign servers is not currently supported. AppBoard includes a number of samples in the <tt>${application.home}/stylesheets/samples</tt> directory, such as for transforming RSS feeds.
|-
|-
|'''httpAuthPassword'''
|'''HTTP Auth Username'''
|Password to use for HTTP Auth (Basic, Digest, etc.). Ignored if HTTP Auth is not required.
|Username to use for HTTP Auth (Basic, Digest, etc.). Ignored if HTTP Auth is not required.  
|-
|-
|'''cacheTimeout'''
|'''HTTP Auth Password'''
|Time limit before re-loading results from the RESTful web service. Note: if a timeout is not specified, it will calculate the timeout based on the timeout specfied by the response header. If the request should be run on every request, then a value of '0' should be specified.
|Password to use for HTTP Auth (Basic, Digest, etc.). Ignored if HTTP Auth is not required.  
|-
|-
|'''httpHeaders'''
|'''HTTP Headers'''
|Optional HTTP headers to set on request.  
|Optional HTTP headers to set on request.  
|-
|-
|'''httpPostBody'''
|'''HTTP Post Body'''
|Optional POST body to send with request. If set, the request will be made via a POST; if not, the HTTP request will be a GET request.
|Optional POST body to send with request. If set, the request will be made via a POST; if not, the HTTP request will be a GET request.
|-
|-
|'''proxyurl'''
|'''Cache Timeout'''
|Time limit before re-loading results from the RESTful web service. Note: if a timeout is not specified, it will calculate the timeout based on the timeout specfied by the response header. If the request should be run on every request, then a value of '0' should be specified.
|-
|'''Proxy URL'''
|Optional URL for web proxy.
|Optional URL for web proxy.
|-
|-
|'''proxyuser'''
|'''Use NTLM Authentication'''
|Optional Use NTLM Auth for Proxy.
|-
|'''Proxy Domain'''
|Optional Proxy NT Domain.
|-
|'''Proxy Username'''
|Optional Proxy Username.
|Optional Proxy Username.
|-
|-
|'''proxypass'''
|'''Proxy Password'''
|Optional Proxy Password.
|Optional Proxy Password.
|-
|'''proxyntlm'''
|Optional Proxy NT Domain.
|-
|'''proxydomain'''
|Optional Use NTLM Auth for Proxy.
|-
|-
|}
|}
=== Additional Resources ===
* [[appboard/2.4/builder/data_sources/web_service/xml-xslt-tutorial|XSLT Tutorial]]
* [[appboard/2.4/builder/data_sources/web_service/xml-xslt-soap|Accessing SOAP web services]]

Latest revision as of 23:22, 30 January 2014

There are several web service data adapters available to choose from in AppBoard: CSV Web, JSON Web, and XML XSLT Web. As each type of data adapter will require different information to connect to AppBoard, this page will assist you in understanding some of the specific options associated with web service data adapters.


CSV Web

This data source adapter is for web services that provide Comma Separated Value (CSV) formatted responses.

Connecting with CSV Web

The Data Source Wizard begins with the "Connect" step. Here you must fill in the necessary information to connect AppBoard to your CSV Web data source. These are the possible elements you will need to provide:


Name Description
URL Base URL for the web service from which to fetch data. This can also be the complete URL, if you want to include any needed parameters here instead of as separate settings. Sample CSV files have been included with this distribution and are located within various folders under ${application.home}/data/. On a Windows server, these can be accessed using a URL like file:///${application.home}/data/path_to_CSV (on UNIX, the URL would start as file://). Please see http://en.wikipedia.org/wiki/File_URI_scheme for more details on using file:/// (or file://) URLs. Alternatively, the sample CSV files can be copied to a custom folder in the enportal directory and accessed via http://host:port/enportal/custom_folder/path_to_CSV.

Example URL: file:///${application.home}/data/pkg/appboard/test/timeline_sample_data.csv
HTTP Auth Username Username to use for HTTP Auth (Basic, Digest, etc.). Ignored if HTTP Auth is not needed.
HTTP Auth Password Password to use for HTTP Auth (Basic, Digest, etc.). Ignored if HTTP Auth is not needed.
HTTP Headers Optional HTTP headers to set on request.
HTTP Post Body Optional POST body to send with request. If set, the request will be made via a POST; if not, the HTTP request will be a GET request.
Cache Timeout Time limit before reloading results from the database.
Data Delimiter Whether the data includes a delimiter to define the columns/field names.
Contains Header Line Defines whether the response contains a header line, which should be the first row. If no header is present the first row returned in discovery will be used to define the initial field names. Note: in order for query SHIMs to work then a header line is required.
Header Meta Delimiter Whether the header includes a delimiter.
String Boundaries Whether the data fields should be written within quotes.
Proxy URL Optional URL for web proxy.
Use NTLM Authentication Optional Use NTLM Auth for Proxy.
Proxy Domain Optional Proxy NT Domain.
Proxy Username Optional Proxy Username.
Proxy Password Optional Proxy Password.

JSON Web

This data source adapter is for web services that provide JavaScript Object Notation (JSON) formatted responses.

Connecting with JSON Web

The Data Source Wizard begins with the "Connect" step. Here you must fill in the necessary information to connect AppBoard to your JSON data source. These are the possible elements you will need to provide:

Name Description
URL Base URL for the web service from which to fetch data. This can also be the complete URL, if you want to include any needed parameters here instead of as separate settings. The distribution contains sample JSON files located within ${application.home}/data/JSON/. The following section in the documentation also includes links to copies of these sample JSON files which demonstrate valid formatting. Any JSON files placed on the AppBoard server can go either within ${application.home}/data/ or within a custom folder in the enportal directory. Just like with the CSV files, these can be accessed using URLs like file:///${application.home}/data/path_to_JSON or http://host:port/enportal/custom/path_to_JSON.

Example URL: file:///${application.home}/data/JSON/data.json
HTTP Auth Username Username to use for HTTP Auth (Basic, Digest, etc.). Ignored if HTTP Auth is not needed.
HTTP Auth Password Password to use for HTTP Auth (Basic, Digest, etc.). Ignored if HTTP Auth is not needed.
HTTP Headers Optional HTTP headers to set on request.
HTTP Post Body Optional POST body to send with request. If set, the request will be made via a POST, if not, the HTTP request will be a GET request.
Cache Timeout Time limit before reloading results from the database.
Proxy URL Optional URL for web proxy.
Use NTLM Authentication Optional Use NTLM Auth for Proxy.
Proxy Domain Optional Proxy NT Domain.
Proxy Username Optional Proxy Username.
Proxy Password Optional Proxy Password.

Valid JSON Formats

JSON can be used to represent complex data structures using a combination of arrays, objects, and key/value pairs. While this flexibility is good in general, for AppBoard the data source adapter needs to understand and convert this into one or more entities with a set number of columns. The following formats deliver the best results:

  • An array of objects (see the example data.json).
  • An object that contains an array of objects (see services.json).
  • An object containing a variety of "key":value fields, with the fields consisting of strings, numbers, boolean values, arrays, or objects (see glossary.json, person.json, and widget.json).

Some additional notes:

  • Root level key/value pairs may cause parsing issues.
  • Named objects within objects will be created as a separate entity, so it's important that each of these named objects have a consistent set of keys.

XML XSLT Web

The XML Web Service Adapter supports data retrieval from HTTP REST services. XML data returned from the REST service is transformed to the AppBoard internal data structure using XSLT. Hierarchical relationships between XML data elements can be represented within AppBoard through associations within the AppBoard data model.

This adapter can also be used to access some SOAP web service calls where there is a single call and response.


Connecting with XML XSLT

The Data Source Wizard begins with the "Connect" step. Here you must fill in the necessary information to connect AppBoard with your XML XSLT data source. These are the possible elements you will need to provide:

Name Description
URL This is the location of the information file used for your data source. Included in the distribution are some sample XML files which can be accessed by using file:/// (on Windows) or file:// (on UNIX) URL. For example, on Windows it would look like file:///${application.home}/data/pkg/sample/atlas/atlas.xml and on UNIX it would look like file://${application.home}/data/pkg/sample/atlas/atlas.xml. As with the CSV and JSON adapters, XML files can also be placed within a custom folder in the enportal directory and accessed via http://host:port/enportal/custom_folder/path_to_XML. Normally you will be using http:// to reference the RESTful Web service.

Example URL: file:///${application.home}/data/pkg/sample/atlas/atlas.xml
Example Stylesheet: ${application.home}/stylesheets/sample/atlas-to-adapter.xsl
Style Sheet Path The path from ${application.home}/stylesheets/ of the XSL stylesheet to be used to convert the data from the web service into the AppBoard internal XML representation. Note that the stylesheet must be on the AppBoard server as importing stylesheets from foreign servers is not currently supported. AppBoard includes a number of samples in the ${application.home}/stylesheets/samples directory, such as for transforming RSS feeds.
HTTP Auth Username Username to use for HTTP Auth (Basic, Digest, etc.). Ignored if HTTP Auth is not required.
HTTP Auth Password Password to use for HTTP Auth (Basic, Digest, etc.). Ignored if HTTP Auth is not required.
HTTP Headers Optional HTTP headers to set on request.
HTTP Post Body Optional POST body to send with request. If set, the request will be made via a POST; if not, the HTTP request will be a GET request.
Cache Timeout Time limit before re-loading results from the RESTful web service. Note: if a timeout is not specified, it will calculate the timeout based on the timeout specfied by the response header. If the request should be run on every request, then a value of '0' should be specified.
Proxy URL Optional URL for web proxy.
Use NTLM Authentication Optional Use NTLM Auth for Proxy.
Proxy Domain Optional Proxy NT Domain.
Proxy Username Optional Proxy Username.
Proxy Password Optional Proxy Password.

Additional Resources

Retrieved from "http://ab.edge-technologies.com/docs/index.php?title=Appboard/2.4/builder/data_sources/web_service&oldid=4098"