Appboard/old/advanced data adapter

Prerequisites

These examples use the same system configuration as the Creating An AppBoard Data Adapter examples. Review the required environment and verify it using the steps on that page before continuing.

Advanced Data Adapter Concepts

The previous examples demonstrate how to develop the necessary callback methods for the AppBoard system to interrogate an adapter for its Entities and related settings. It also demonstrated the use of fetchRecords() to retrieve all the content for a given Entity. While this approach is sufficient for many adapters, the advanced topics in this walkthrough will illustrate how to be more efficient in memory and network bandwidth usage through the use of other methods to retrieve the data.

These examples will demonstrate the following concepts:

1. Partial record retrieval -- Instead of retrieving all the attributes for each record, the adapter can return only some of the attributes, and when needed, the client will request the remainder
2. Custom query implementation -- Default query mechanism relies on the server having all records for an entity. If the adapter's data source permits filtered retrieval, the adapter can intercept incoming queries and handle them more efficiently.
3. (Future) More discrete cache control

Refactoring fetchRecords()

To make the comparison between the previous fetchRecords() implementation and the new methods that will return the data records, the original adapter class was refactored. The new method is:

[java,N] public void fetchRecords(ServiceRequestFacade request, ServiceResponseFacade response) throws Exception { logger.debug("fetchRecords('"+request.getEntity().getNamespace().getFQNamespace()+"') starting");

List<GenericRecord> recordList = new ArrayList<GenericRecord>();

int duration = getSettingAsInt(SETTING_DURATION, 600); int dataInterval = getSettingAsInt(SETTING_DATA_INTERVAL, 10);

long currentMs = System.currentTimeMillis(); long currentSec = currentMs/1000;

long startSec = ((currentSec - duration) / dataInterval) * dataInterval; // align to interval

for (long timeSec = startSec; timeSec <= currentSec; timeSec += dataInterval) { EntityDef entity = request.getEntity(); fillSineRecords(recordList, timeSec, entity, true, "Set in fetchRecords()"); }

response.addGenericRecords(recordList);

logger.debug("fetchRecords('"+request.getEntity().getNamespace().getFQNamespace()+"') completed"); }

The main change is that the GenericRecords are no longer constructed in fetchRecords(), but instead in a method fillSineRecords(). That method, and those it calls, are:

[java,N] private void fillSineRecords(List<GenericRecord> recordList, long timeSec, EntityDef entity, boolean sineComplete, String sineDescription) { int value = getSineValue(timeSec);

Date timestamp = new Date(timeSec * 1000);

boolean isWaveEntity = entity.getNamespace().getEntityName().equals(ENTITY_SINE_WAVE); if (isWaveEntity) { GenericRecord record = getSineRecord(timeSec, entity, value, timestamp, sineComplete, sineDescription); recordList.add(record); } else { int assocOffset = getSettingAsInt(SETTING_ASSOCIATED_DATA_POINTS, 3);

for (int i = -assocOffset; i <= assocOffset; i++) { GenericRecord record = new GenericRecord(entity); record.setAttributeValue(ATTR_TIMESTAMP_OFFSET, Long.toString(timeSec) + ':' + Integer.toString(i)); record.setAttributeValue(ATTR_WAVE_KEY, timeSec); record.setAttributeValue(ATTR_TIMESTAMP, timestamp); record.setAttributeValue(ATTR_OFFSET, i); record.setAttributeValue(ATTR_DESCRIPTION, "Offset " + i + " from value " + value + " is " + (value + i));

recordList.add(record); } } }

private int getSineValue(long timeSec) {

int wavePeriod = getSettingAsInt(SETTING_WAVE_PERIOD, 60); int waveAmp = getSettingAsInt(SETTING_WAVE_AMPLITUDE, 50); int waveOffset = getSettingAsInt(SETTING_WAVE_OFFSET, 50);

double phase = ((double) (timeSec % wavePeriod)) / ((double) wavePeriod) * 2.0d * Math.PI; int value = (int) (Math.sin(phase) * waveAmp + waveOffset); return value; }

private GenericRecord getSineRecord(long timeSec, EntityDef entity, int value, Date timestamp, boolean setComplete, String description) { GenericRecord record = new GenericRecord(entity); record.setAttributeValue(ATTR_KEY, timeSec); record.setAttributeValue(ATTR_TIMESTAMP, timestamp); record.setAttributeValue(ATTR_VALUE, value); if (setComplete) { record.setAttributeValue(ATTR_DESCRIPTION, description); } record.setComplete(setComplete); return record; }

Note the only new method exposed, which is GenericRecord.setComplete(boolean isComplete). This method marks the record as either complete or not, with the default being complete. If the record is complete, the client will not request any missing attribute values, which is the behavior seen so far. The next step will explore using incomplete records.

To verify the refactoring works, preview the Sine Wave Data Source. The preview should look similar to:

Note that the description field shows "Set in fetchRecords()". This will change after the next step.

Incomplete Records / findById()

Incomplete records are any GenericRecords that have been marked with record.setComplete(false), which informs the client that when accessing the details of that record, the client will need to first request the rest of the record. This is done automatically by the client through the findById() method.

The typical scenario that would indicate an incomplete record response is one where one or more attributes are very large (long strings) or costly to calculate (complicated status calculation). In these instances, returning incomplete records allows the adapter to delay the costly data retrieval until required by the client.

If records are marked as incomplete in the fetchRecords() method, findById() must be implemented to get the missing data to the client and to prevent the client from asking the server for an update every time any attribute of the record is access. This is because the retrieval is triggered by a check of the record on the client on every access to any attribute value, and if the record is incomplete, findById() is called with the intention of getting a complete record.

findById() is similar to fetchRecords(), but instead of returning all the GenericRecords for the Entity, only a single record needs to be returned. As a side note, you may fill the response object with additional records other than the one requested by ID (may be useful if the target record has associations that are being filled). findById() must first determine the appropriate key for the record requested, and then return a completed GenericRecord. The code for this is: [java,N] public GenericRecord findByID(ServiceRequestFacade request, ServiceResponseFacade response) { logger.debug("findById('"+ request.getEntity().getNamespace().getFQNamespace()+ "') starting");

Map<String, Object> paramMap = request.getParams(); if (paramMap == null || paramMap.size() < 0) { response.setStatusCode(-5); response.setMessage("No Query Parameters defined"); return null; } Number id = null; Object idObj = paramMap.get(ID); if (idObj instanceof Number) { id = (Number) idObj; } else { id = Double.parseDouble(idObj.toString()); }

long timeSec = id.longValue(); Date timestamp = new Date(timeSec * 1000); EntityDef entity = request.getEntity();

int value = getSineValue(timeSec);

GenericRecord record = getSineRecord(timeSec, entity, value, timestamp, true, "Set in findById()");

logger.debug("findById('"+ request.getEntity().getNamespace().getFQNamespace()+ "','"+id+"') completed");

return record; }

The first half of the method extracts the ID parameter from the request and performs some error checking (if no ID exists, this is the error response), then converts the ID object to a Number (since the primary key for the Sine Wave Entity is the numeric field Key -- recall new AttributeDef(ATTR_KEY, true, AttributeType.INT)).

Once the key is retrieved, it is converted to a meaningful value for this Entity, namely the time in seconds, then that is used, along with the helper methods for sine wave calculations, to create and return a GenericRecord. Not that this record will have setComplete(true) due to the parameter for the getSineRecord() call.

The last step is to modify the fetchRecords() method to have it set the record as incomplete. This will force the client to make a secondary call to retrieve the complete records.

[java,N] public void fetchRecords(ServiceRequestFacade request, ServiceResponseFacade response) throws Exception { ... fillSineRecords(recordList, timeSec, entity, false, "Set in fetchRecords()"); ... }

The new behavior can be verified through a couple changes. First, a preview of the Sine Wave Entity will show "Set in findById()" for the description, and careful observation will show that the descriptions fill in after the other attributes, since that is the field left incomplete in the getSineRecord() helper method.

The other way is through the logger. If the following code is added to the adapter class: [java,n] static { logger.addAppender(new ConsoleAppender(new PatternLayout(PatternLayout.TTCC_CONVERSION_PATTERN), "system.out")); logger.setLevel(Level.DEBUG); } The logger statements will output to the console as the adapter is exercised. When the preview of Sine Wave is displayed and then scrolled down, the description fields will need filled, resulting in blocks of calls to findById(), as the following except from the Tomcat console illustrates:

11335 [http-8081-exec-3] DEBUG com.demo.DemoDataAdapter  - findById('dev.demo.Sine Wave') starting
11335 [http-8081-exec-2] DEBUG com.demo.DemoDataAdapter  - findById('dev.demo.Sine Wave') starting
11335 [http-8081-exec-5] DEBUG com.demo.DemoDataAdapter  - findById('dev.demo.Sine Wave') starting
11335 [http-8081-exec-7] DEBUG com.demo.DemoDataAdapter  - findById('dev.demo.Sine Wave') starting
11335 [http-8081-exec-6] DEBUG com.demo.DemoDataAdapter  - findById('dev.demo.Sine Wave') starting
11335 [http-8081-exec-4] DEBUG com.demo.DemoDataAdapter  - findById('dev.demo.Sine Wave') starting
11340 [http-8081-exec-3] DEBUG com.demo.DemoDataAdapter  - findById('dev.demo.Sine Wave','1.314922665E9') completed
11340 [http-8081-exec-6] DEBUG com.demo.DemoDataAdapter  - findById('dev.demo.Sine Wave','1.31492265E9') completed
11340 [http-8081-exec-5] DEBUG com.demo.DemoDataAdapter  - findById('dev.demo.Sine Wave','1.31492266E9') completed
11340 [http-8081-exec-7] DEBUG com.demo.DemoDataAdapter  - findById('dev.demo.Sine Wave','1.314922645E9') completed
11340 [http-8081-exec-2] DEBUG com.demo.DemoDataAdapter  - findById('dev.demo.Sine Wave','1.31492267E9') completed
11340 [http-8081-exec-4] DEBUG com.demo.DemoDataAdapter  - findById('dev.demo.Sine Wave','1.314922655E9') completed
11482 [http-8081-exec-7] DEBUG com.demo.DemoDataAdapter  - findByQuery('dev.demo.Associated Data') starting
11482 [http-8081-exec-3] DEBUG com.demo.DemoDataAdapter  - findByQuery('dev.demo.Associated Data') starting
11482 [http-8081-exec-2] DEBUG com.demo.DemoDataAdapter  - findByQuery('dev.demo.Associated Data') starting
11482 [http-8081-exec-4] DEBUG com.demo.DemoDataAdapter  - findByQuery('dev.demo.Associated Data') starting
11482 [http-8081-exec-5] DEBUG com.demo.DemoDataAdapter  - findByQuery('dev.demo.Associated Data') starting
11482 [http-8081-exec-6] DEBUG com.demo.DemoDataAdapter  - findByQuery('dev.demo.Associated Data') starting
11488 [http-8081-exec-5] DEBUG com.demo.DemoDataAdapter  - findByQuery(): queryString = SELECT * WHERE `Wave Key`=1314922660
11486 [http-8081-exec-4] DEBUG com.demo.DemoDataAdapter  - findByQuery(): queryString = SELECT * WHERE `Wave Key`=1314922655
11485 [http-8081-exec-2] DEBUG com.demo.DemoDataAdapter  - findByQuery(): queryString = SELECT * WHERE `Wave Key`=1314922670
11483 [http-8081-exec-3] DEBUG com.demo.DemoDataAdapter  - findByQuery(): queryString = SELECT * WHERE `Wave Key`=1314922665
11483 [http-8081-exec-7] DEBUG com.demo.DemoDataAdapter  - findByQuery(): queryString = SELECT * WHERE `Wave Key`=1314922645
11493 [http-8081-exec-3] DEBUG com.demo.DemoDataAdapter  - fetchRecords('dev.demo.Associated Data') starting
11492 [http-8081-exec-4] DEBUG com.demo.DemoDataAdapter  - fetchRecords('dev.demo.Associated Data') starting
11492 [http-8081-exec-5] DEBUG com.demo.DemoDataAdapter  - fetchRecords('dev.demo.Associated Data') starting
11492 [http-8081-exec-2] DEBUG com.demo.DemoDataAdapter  - fetchRecords('dev.demo.Associated Data') starting
11489 [http-8081-exec-6] DEBUG com.demo.DemoDataAdapter  - findByQuery(): queryString = SELECT * WHERE `Wave Key`=1314922650
11494 [http-8081-exec-7] DEBUG com.demo.DemoDataAdapter  - fetchRecords('dev.demo.Associated Data') starting
11511 [http-8081-exec-5] DEBUG com.demo.DemoDataAdapter  - fetchRecords('dev.demo.Associated Data') completed
11517 [http-8081-exec-3] DEBUG com.demo.DemoDataAdapter  - fetchRecords('dev.demo.Associated Data') completed
11502 [http-8081-exec-6] DEBUG com.demo.DemoDataAdapter  - fetchRecords('dev.demo.Associated Data') starting
11519 [http-8081-exec-4] DEBUG com.demo.DemoDataAdapter  - fetchRecords('dev.demo.Associated Data') completed
11513 [http-8081-exec-7] DEBUG com.demo.DemoDataAdapter  - fetchRecords('dev.demo.Associated Data') completed
11512 [http-8081-exec-2] DEBUG com.demo.DemoDataAdapter  - fetchRecords('dev.demo.Associated Data') completed
11522 [http-8081-exec-6] DEBUG com.demo.DemoDataAdapter  - fetchRecords('dev.demo.Associated Data') completed
11778 [http-8081-exec-6] DEBUG com.demo.DemoDataAdapter  - findByQuery('dev.demo.Associated Data') completed
11779 [http-8081-exec-3] DEBUG com.demo.DemoDataAdapter  - findByQuery('dev.demo.Associated Data') completed
11779 [http-8081-exec-4] DEBUG com.demo.DemoDataAdapter  - findByQuery('dev.demo.Associated Data') completed
11779 [http-8081-exec-7] DEBUG com.demo.DemoDataAdapter  - findByQuery('dev.demo.Associated Data') completed
11780 [http-8081-exec-2] DEBUG com.demo.DemoDataAdapter  - findByQuery('dev.demo.Associated Data') completed
11780 [http-8081-exec-5] DEBUG com.demo.DemoDataAdapter  - findByQuery('dev.demo.Associated Data') completed
11801 [http-8081-exec-4] DEBUG com.demo.DemoDataAdapter  - findById('dev.demo.Sine Wave') starting
11802 [http-8081-exec-5] DEBUG com.demo.DemoDataAdapter  - findById('dev.demo.Sine Wave') starting
11802 [http-8081-exec-4] DEBUG com.demo.DemoDataAdapter  - findById('dev.demo.Sine Wave','1.31492263E9') completed
11804 [http-8081-exec-1] DEBUG com.demo.DemoDataAdapter  - findById('dev.demo.Sine Wave') starting
11804 [http-8081-exec-2] DEBUG com.demo.DemoDataAdapter  - findById('dev.demo.Sine Wave') starting
11803 [http-8081-exec-7] DEBUG com.demo.DemoDataAdapter  - findById('dev.demo.Sine Wave') starting
11803 [http-8081-exec-5] DEBUG com.demo.DemoDataAdapter  - findById('dev.demo.Sine Wave','1.31492261E9') completed
11808 [http-8081-exec-7] DEBUG com.demo.DemoDataAdapter  - findById('dev.demo.Sine Wave','1.314922595E9') completed
11807 [http-8081-exec-2] DEBUG com.demo.DemoDataAdapter  - findById('dev.demo.Sine Wave','1.314922625E9') completed
11806 [http-8081-exec-1] DEBUG com.demo.DemoDataAdapter  - findById('dev.demo.Sine Wave','1.314922635E9') completed

Custom Query Implementation / findByQuery()

The incomplete record technique is useful when there are not too many records, but each can be costly to calculate or transfer. When the data set has too many records to completely retrieve efficiently, implementing a custom query can provide significant performance gains. The default query mechanism uses fetchRecords() to pull in all records, then runs a post-processing routine to filter out the records not desired. When the data sets are not too large, this saves the developer time writing additional routines. This example will demonstrate how to implement a custom query method by overriding findByQuery().

The signature of findByQuery() is identical to fetchRecords(), and much of the implementation will look familiar. The complexity behind findByQuery() is in properly converting the query into the form the adapter can use.

[java,n] public void findByQuery(ServiceRequestFacade request, ServiceResponseFacade response) { logger.debug("findByQuery('"+ request.getEntity().getNamespace().getFQNamespace()+ "') starting");

try { if (request.getEntity().getEntityName().equals(ENTITY_SINE_WAVE)) { Query query = request.getQuery(); if (query!= null && !query.isEmpty()) { // only intercept non-empty queries String filter = query.getFilter(); if (filter != null && !filter.isEmpty()) { // only processing a filter rule for this example logger.debug("findByQuery: request.getEntity() = " + request.getEntity().getEntityName()) ; String queryString = query.toGQLString(request.getEntity()); Up to this point is boilerplate code that will extract the queryString from the request, if any. If there is none, the code will call the super implementation of findByQuery(). This may be because the query is one that specifies a group by or pivot operation, not a filter operation. [java,n]

int duration = getSettingAsInt(SETTING_DURATION, 600); int dataInterval = getSettingAsInt(SETTING_DATA_INTERVAL, 10);

long currentMs = System.currentTimeMillis(); long currentSec = currentMs/1000;

long startSec = ((currentSec - duration) / dataInterval) * dataInterval; // align to interval

long lowBoundary = startSec; long highBoundary = currentSec; The above code section sets up some defaults that will be used for the sine wave calculations, and is completely specific to this contrived example. [java,n]

// For "efficient" examples, filter must be in the form of // WHERE `Key` <OPERATOR> <VALUE> [AND `Key` <OPERATOR> <VALUE>] // with no order by or group by following it, where <OPERATOR> must be either '>=' or '<='. // Any other forms will be passed to the super.findByQuery() implementation // In a real adapter, you will likely perform a translation of the query string into // the language/syntax of the system being queried.

String whereClause = queryString.replaceFirst(".* WHERE (.*)", "$1");

String[] andExprs = whereClause.split("AND"); for (int i=0; i<andExprs.length; i++) { String expression = andExprs[i].trim(); String[] exprParts = expression.split(" ");

String attrName = exprParts[0].replaceAll("`", ""); if (attrName.equals("Key")) { String operator = exprParts[1]; long timeValue = Long.parseLong(exprParts[2]); if (operator.equals(">=")) { lowBoundary = timeValue; } else if (operator.equals("<=")) { highBoundary = timeValue; } else { logger.debug("findByQuery: ignoring - " + expression); super.findByQuery(request, response); return; } } else { logger.debug("findByQuery: ignoring - " + expression); super.findByQuery(request, response); return; } } The previous section of code parses a specific range of filters, namely those in the form of 'Key' >= something or 'Key' <= something, and updates the variables defining the range for the sine wave calculation. If any filters are encountered that do not match the narrow range defined, the method resorts to calling super.findByQuery() and returning. [java,n]

List<GenericRecord> recordList = new ArrayList<GenericRecord>();

lowBoundary = (lowBoundary / dataInterval) * dataInterval; // align to interval

// Now that lowBoundary and highBoundary are set, return records for (long timeSec = lowBoundary; timeSec <= highBoundary; timeSec += dataInterval) { EntityDef entity = request.getEntity(); fillSineRecords(recordList, timeSec, entity, true, "Set in findByQuery()"); }

response.addGenericRecords(recordList); return; } } } The above section is almost identical to that from fetchRecords(), with the exception that these records are returned complete, while the new fetchRecords() did not fill the description returned incomplete records. This is just for illustrative purposes. findByQuery() can return complete or incomplete records, as dictacted by the type and size of data being returned. [java,n]

Query query = request.getQuery(); if (query!= null && !query.isEmpty()) { // only intercept non-empty queries logger.debug("findByQuery(): queryString = " + query.toGQLString(request.getEntity())); } // only run this if not processed above super.findByQuery(request, response); } finally { logger.debug("findByQuery('"+ request.getEntity().getNamespace().getFQNamespace()+ "') completed"); } } The method finishes with some logging and a default execution of super.findByQuery() if not handled by the above code.

To invoke findByQuery(), the Sine Wave data collection must be edited to apply a filter. The best way to do this is to first preview the Sine Wave Entity, and copy one of the key values by first selecting it and then clicking Copy to Clipboard. This value can then be modified and used in the filter expression. Alternately, just use values like 1000 and 2000 for the Key, and the data will be calculated for times in 1970.

Once the filters are created and only have Key <= or Key >= expressions, the custom query will be executed. This can be verified by looking at the preview, where the description will now by "Set in findByQuery()".

The Tomcat console will also provide confirmation in the form of log message, as below:

16243 [http-8081-exec-4] DEBUG com.demo.DemoDataAdapter  - findByQuery('dev.demo.Sine Wave') starting
16245 [http-8081-exec-4] DEBUG com.demo.DemoDataAdapter  - findByQuery: request.getEntity() = Sine Wave
16249 [http-8081-exec-4] DEBUG com.demo.DemoDataAdapter  - findByQuery(): queryString = SELECT * WHERE `Key` >= 1000 AND  `Key` <= 2000
16257 [http-8081-exec-4] DEBUG com.demo.DemoDataAdapter  - findByQuery('dev.demo.Sine Wave') completed

A couple notes:

• The associations will be blank if the time range is not in last 10 minutes, because the Associated Data Entity only fills the last 10 minutes by default (as determined by the settings for the Data Source).
• If the filter does not match the allowed pattern, findByQuery() will use the super implementation, which will in turn fetch the data using fetchRecords(). Because fetchRecords() will use the same time range as described above, a filter of "Key >= 1000 and Key <= 2000" (and therefore handled by the custom method) will show data from 1970, while a filter of "Key > 1000 and Key < 2000" (and handled by the super.findByQuery()) will return no records, since fetchRecords() would have returned no records for that time range.