Appboard/old/advanced data adapter

Prerequisites

These examples use the same system configuration as the Creating An AppBoard Data Adapter examples. Review the required environment and verify it using the steps on that page before continuing.

Advanced Data Adapter Concepts

The previous examples demonstrate how to develop the necessary callback methods for the AppBoard system to interrogate an adapter for its Entities and related settings. It also demonstrated the use of fetchRecords() to retrieve all the content for a given Entity. While this approach is sufficient for many adapters, the advanced topics in this walkthrough will illustrate how to be more efficient in memory and network bandwidth usage through the use of other methods to retrieve the data.

These examples will demonstrate the following concepts:

1. Partial record retrieval -- Instead of retrieving all the attributes for each record, the adapter can return only some of the attributes, and when needed, the client will request the remainder
2. Custom query implementation -- Default query mechanism relies on the server having all records for an entity. If the adapter's data source permits filtered retrieval, the adapter can intercept incoming queries and handle them more efficiently.
3. (Future) More discrete cache control

Refactoring fetchRecords()

To make the comparison between the previous fetchRecords() implementation and the new methods that will return the data records, the original adapter class was refactored. The new method is:

[java,N] public void fetchRecords(ServiceRequestFacade request, ServiceResponseFacade response) throws Exception { logger.debug("fetchRecords('"+request.getEntity().getNamespace().getFQNamespace()+"') starting");

List<GenericRecord> recordList = new ArrayList<GenericRecord>();

int duration = getSettingAsInt(SETTING_DURATION, 600); int dataInterval = getSettingAsInt(SETTING_DATA_INTERVAL, 10);

long currentMs = System.currentTimeMillis(); long currentSec = currentMs/1000;

long startSec = ((currentSec - duration) / dataInterval) * dataInterval; // align to interval

for (long timeSec = startSec; timeSec <= currentSec; timeSec += dataInterval) { EntityDef entity = request.getEntity(); fillSineRecords(recordList, timeSec, entity, true, "Set in fetchRecords()"); }

response.addGenericRecords(recordList);

logger.debug("fetchRecords('"+request.getEntity().getNamespace().getFQNamespace()+"') completed"); }

The main change is that the GenericRecords are no longer constructed in fetchRecords(), but instead in a method fillSineRecords(). That method, and those it calls, are:

[java,N] private void fillSineRecords(List<GenericRecord> recordList, long timeSec, EntityDef entity, boolean sineComplete, String sineDescription) { int value = getSineValue(timeSec);

Date timestamp = new Date(timeSec * 1000);

boolean isWaveEntity = entity.getNamespace().getEntityName().equals(ENTITY_SINE_WAVE); if (isWaveEntity) { GenericRecord record = getSineRecord(timeSec, entity, value, timestamp, sineComplete, sineDescription); recordList.add(record); } else { int assocOffset = getSettingAsInt(SETTING_ASSOCIATED_DATA_POINTS, 3);

for (int i = -assocOffset; i <= assocOffset; i++) { GenericRecord record = new GenericRecord(entity); record.setAttributeValue(ATTR_TIMESTAMP_OFFSET, Long.toString(timeSec) + ':' + Integer.toString(i)); record.setAttributeValue(ATTR_WAVE_KEY, timeSec); record.setAttributeValue(ATTR_TIMESTAMP, timestamp); record.setAttributeValue(ATTR_OFFSET, i); record.setAttributeValue(ATTR_DESCRIPTION, "Offset " + i + " from value " + value + " is " + (value + i));

recordList.add(record); } } }

private int getSineValue(long timeSec) {

int wavePeriod = getSettingAsInt(SETTING_WAVE_PERIOD, 60); int waveAmp = getSettingAsInt(SETTING_WAVE_AMPLITUDE, 50); int waveOffset = getSettingAsInt(SETTING_WAVE_OFFSET, 50);

double phase = ((double) (timeSec % wavePeriod)) / ((double) wavePeriod) * 2.0d * Math.PI; int value = (int) (Math.sin(phase) * waveAmp + waveOffset); return value; }

private GenericRecord getSineRecord(long timeSec, EntityDef entity, int value, Date timestamp, boolean setComplete, String description) { GenericRecord record = new GenericRecord(entity); record.setAttributeValue(ATTR_KEY, timeSec); record.setAttributeValue(ATTR_TIMESTAMP, timestamp); record.setAttributeValue(ATTR_VALUE, value); if (setComplete) { record.setAttributeValue(ATTR_DESCRIPTION, description); } record.setComplete(setComplete); return record; }

Note the only new method exposed, which is GenericRecord.setComplete(boolean isComplete). This method marks the record as either complete or not, with the default being complete. If the record is complete, the client will not request any missing attribute values, which is the behavior seen so far. The next step will explore using incomplete records.

To verify the refactoring works, preview the Sine Wave Data Source. The preview should look similar to:

Note that the description field shows "Set in fetchRecords()". This will change after the next step.

Incomplete Records / findById()

Incomplete records are any GenericRecords that have been marked with record.setComplete(false), which informs the client that when accessing the details of that record, the client will need to first request the rest of the record. This is done automatically by the client through the findById() method.

If records are marked as incomplete in the fetchRecords() method, findById() must be implemented to get the missing data to the client and to prevent the client from asking the server for an update every time any attribute of the record is access. This is because the retrieval is triggered by a check of the record on the client on every access to any attribute value, and if the record is incomplete, findById() is called with the intention of getting a complete record.

findById() is similar to fetchRecords(), but instead of returning all the GenericRecords for the Entity, only a single record needs to be returned. As a side note, you may fill the response object with additional records other than the one requested by ID (may be useful if the target record has associations that are being filled). findById() must first determine the appropriate key for the record requested, and then return a completed GenericRecord. The code for this is: [java,N] public GenericRecord findByID(ServiceRequestFacade request, ServiceResponseFacade response) { logger.debug("findById('"+ request.getEntity().getNamespace().getFQNamespace()+ "') starting");

Map<String, Object> paramMap = request.getParams(); if (paramMap == null || paramMap.size() < 0) { response.setStatusCode(-5); response.setMessage("No Query Parameters defined"); return null; } Number id = null; Object idObj = paramMap.get(ID); if (idObj instanceof Number) { id = (Number) idObj; } else { id = Double.parseDouble(idObj.toString()); }

long timeSec = id.longValue(); Date timestamp = new Date(timeSec * 1000); EntityDef entity = request.getEntity();

int value = getSineValue(timeSec);

GenericRecord record = getSineRecord(timeSec, entity, value, timestamp, true, "Set in findById()");

logger.debug("findById('"+ request.getEntity().getNamespace().getFQNamespace()+ "','"+id+"') completed");

return record; }

The first half of the method extracts the ID parameter from the request and performs some error checking (if no ID exists, this is the error response), then converts the ID object to a Number (since the primary key for the Sine Wave Entity is the numeric field Key -- recall new AttributeDef(ATTR_KEY, true, AttributeType.INT)).

Once the key is retrieved, it is converted to a meaningful value for this Entity, namely the time in seconds, then that is used, along with the helper methods for sine wave calculations, to create and return a GenericRecord. Not that this record will have setComplete(true) due to the parameter for the getSineRecord() call.

The last step is to modify the fetchRecords() method to have it set the record as incomplete. This will force the client to make a secondary call to retrieve the complete records.

[java,N] public void fetchRecords(ServiceRequestFacade request, ServiceResponseFacade response) throws Exception { ... fillSineRecords(recordList, timeSec, entity, false, "Set in fetchRecords()"); ... }

The new behavior can be verified through a couple changes. First, a preview of the Sine Wave Entity will show "Set in findById()" for the description, and careful observation will show that the descriptions fill in after the other attributes, since that is the field left incomplete in the getSineRecord() helper method.

The other way is through the logger. If the following code is added to the adapter class: [java,n] static { logger.addAppender(new ConsoleAppender(new PatternLayout(PatternLayout.TTCC_CONVERSION_PATTERN), "system.out")); logger.setLevel(Level.DEBUG); } The logger statements will output to the console as the adapter is exercised. When the preview of Sine Wave is displayed and then scrolled down, the description fields will need filled, resulting in blocks of calls to findById(), as the following except from the Tomcat console illustrates:

11335 [http-8081-exec-3] DEBUG com.demo.DemoDataAdapter - findById('dev.demo.Sine Wave') starting 11335 [http-8081-exec-2] DEBUG com.demo.DemoDataAdapter - findById('dev.demo.Sine Wave') starting 11335 [http-8081-exec-5] DEBUG com.demo.DemoDataAdapter - findById('dev.demo.Sine Wave') starting 11335 [http-8081-exec-7] DEBUG com.demo.DemoDataAdapter - findById('dev.demo.Sine Wave') starting 11335 [http-8081-exec-6] DEBUG com.demo.DemoDataAdapter - findById('dev.demo.Sine Wave') starting 11335 [http-8081-exec-4] DEBUG com.demo.DemoDataAdapter - findById('dev.demo.Sine Wave') starting 11340 [http-8081-exec-3] DEBUG com.demo.DemoDataAdapter - findById('dev.demo.Sine Wave','1.314922665E9') completed 11340 [http-8081-exec-6] DEBUG com.demo.DemoDataAdapter - findById('dev.demo.Sine Wave','1.31492265E9') completed 11340 [http-8081-exec-5] DEBUG com.demo.DemoDataAdapter - findById('dev.demo.Sine Wave','1.31492266E9') completed 11340 [http-8081-exec-7] DEBUG com.demo.DemoDataAdapter - findById('dev.demo.Sine Wave','1.314922645E9') completed 11340 [http-8081-exec-2] DEBUG com.demo.DemoDataAdapter - findById('dev.demo.Sine Wave','1.31492267E9') completed 11340 [http-8081-exec-4] DEBUG com.demo.DemoDataAdapter - findById('dev.demo.Sine Wave','1.314922655E9') completed 11482 [http-8081-exec-7] DEBUG com.demo.DemoDataAdapter - findByQuery('dev.demo.Associated Data') starting 11482 [http-8081-exec-3] DEBUG com.demo.DemoDataAdapter - findByQuery('dev.demo.Associated Data') starting 11482 [http-8081-exec-2] DEBUG com.demo.DemoDataAdapter - findByQuery('dev.demo.Associated Data') starting 11482 [http-8081-exec-4] DEBUG com.demo.DemoDataAdapter - findByQuery('dev.demo.Associated Data') starting 11482 [http-8081-exec-5] DEBUG com.demo.DemoDataAdapter - findByQuery('dev.demo.Associated Data') starting 11482 [http-8081-exec-6] DEBUG com.demo.DemoDataAdapter - findByQuery('dev.demo.Associated Data') starting 11488 [http-8081-exec-5] DEBUG com.demo.DemoDataAdapter - findByQuery(): queryString = SELECT * WHERE `Wave Key`=1314922660 11486 [http-8081-exec-4] DEBUG com.demo.DemoDataAdapter - findByQuery(): queryString = SELECT * WHERE `Wave Key`=1314922655 11485 [http-8081-exec-2] DEBUG com.demo.DemoDataAdapter - findByQuery(): queryString = SELECT * WHERE `Wave Key`=1314922670 11483 [http-8081-exec-3] DEBUG com.demo.DemoDataAdapter - findByQuery(): queryString = SELECT * WHERE `Wave Key`=1314922665 11483 [http-8081-exec-7] DEBUG com.demo.DemoDataAdapter - findByQuery(): queryString = SELECT * WHERE `Wave Key`=1314922645 11493 [http-8081-exec-3] DEBUG com.demo.DemoDataAdapter - fetchRecords('dev.demo.Associated Data') starting 11492 [http-8081-exec-4] DEBUG com.demo.DemoDataAdapter - fetchRecords('dev.demo.Associated Data') starting 11492 [http-8081-exec-5] DEBUG com.demo.DemoDataAdapter - fetchRecords('dev.demo.Associated Data') starting 11492 [http-8081-exec-2] DEBUG com.demo.DemoDataAdapter - fetchRecords('dev.demo.Associated Data') starting 11489 [http-8081-exec-6] DEBUG com.demo.DemoDataAdapter - findByQuery(): queryString = SELECT * WHERE `Wave Key`=1314922650 11494 [http-8081-exec-7] DEBUG com.demo.DemoDataAdapter - fetchRecords('dev.demo.Associated Data') starting 11511 [http-8081-exec-5] DEBUG com.demo.DemoDataAdapter - fetchRecords('dev.demo.Associated Data') completed 11517 [http-8081-exec-3] DEBUG com.demo.DemoDataAdapter - fetchRecords('dev.demo.Associated Data') completed 11502 [http-8081-exec-6] DEBUG com.demo.DemoDataAdapter - fetchRecords('dev.demo.Associated Data') starting 11519 [http-8081-exec-4] DEBUG com.demo.DemoDataAdapter - fetchRecords('dev.demo.Associated Data') completed 11513 [http-8081-exec-7] DEBUG com.demo.DemoDataAdapter - fetchRecords('dev.demo.Associated Data') completed 11512 [http-8081-exec-2] DEBUG com.demo.DemoDataAdapter - fetchRecords('dev.demo.Associated Data') completed 11522 [http-8081-exec-6] DEBUG com.demo.DemoDataAdapter - fetchRecords('dev.demo.Associated Data') completed 11778 [http-8081-exec-6] DEBUG com.demo.DemoDataAdapter - findByQuery('dev.demo.Associated Data') completed 11779 [http-8081-exec-3] DEBUG com.demo.DemoDataAdapter - findByQuery('dev.demo.Associated Data') completed 11779 [http-8081-exec-4] DEBUG com.demo.DemoDataAdapter - findByQuery('dev.demo.Associated Data') completed 11779 [http-8081-exec-7] DEBUG com.demo.DemoDataAdapter - findByQuery('dev.demo.Associated Data') completed 11780 [http-8081-exec-2] DEBUG com.demo.DemoDataAdapter - findByQuery('dev.demo.Associated Data') completed 11780 [http-8081-exec-5] DEBUG com.demo.DemoDataAdapter - findByQuery('dev.demo.Associated Data') completed 11801 [http-8081-exec-4] DEBUG com.demo.DemoDataAdapter - findById('dev.demo.Sine Wave') starting 11802 [http-8081-exec-5] DEBUG com.demo.DemoDataAdapter - findById('dev.demo.Sine Wave') starting 11802 [http-8081-exec-4] DEBUG com.demo.DemoDataAdapter - findById('dev.demo.Sine Wave','1.31492263E9') completed 11804 [http-8081-exec-1] DEBUG com.demo.DemoDataAdapter - findById('dev.demo.Sine Wave') starting 11804 [http-8081-exec-2] DEBUG com.demo.DemoDataAdapter - findById('dev.demo.Sine Wave') starting 11803 [http-8081-exec-7] DEBUG com.demo.DemoDataAdapter - findById('dev.demo.Sine Wave') starting 11803 [http-8081-exec-5] DEBUG com.demo.DemoDataAdapter - findById('dev.demo.Sine Wave','1.31492261E9') completed 11808 [http-8081-exec-7] DEBUG com.demo.DemoDataAdapter - findById('dev.demo.Sine Wave','1.314922595E9') completed 11807 [http-8081-exec-2] DEBUG com.demo.DemoDataAdapter - findById('dev.demo.Sine Wave','1.314922625E9') completed 11806 [http-8081-exec-1] DEBUG com.demo.DemoDataAdapter - findById('dev.demo.Sine Wave','1.314922635E9') completed