MetaDataDesign

Thinking out loud here on the wiki for design around meta data. My first thinking is around long-term where we want to be as an end-goal

Long Term

Each client/driver like our modbus one, bacnet one, etc. etc. Has different meta data associated with it.

modbus for instance has a type of device(say ModBusDevice) which has properties and then there are many instances of that device deployed out on a campus or building(say ModBusDeviceInstance) which has it's own properties.

Bacnet is similar so you have BacnetDevice and BacnetDeviceInstance. Some of the properties are the same and some of the properties are different. For instance location of instances is a property both BacnetDeviceInstance and ModbusDeviceInstance have.

Ideally, drivers are storing all meta data in databus, either just being readers or as in bacnet, a writer and a reader since it discovers devices on the network.

Then there is client/driver developer who really wants to use something like playorm and easily have a BacnetDevice entity and BacnetDeviceInstance entity so he can query into the table and find entities faster. At the same time, we want these entities exposed to customers through databus as well so they can query the same entities. A user of databus would also like to update the meta data of these entities as well and databus does not have and will not want to know about BacnetDevice entity specifically BUT we do have the meta data for it so we do know it's format.

The real question becomes as drivers/clients define/use entities like this, how do they tell databus about these new tables (all of these tables should probably be virtual and in one CF called metadata or something as well). These new meta tables need to show up in a databus database for end users to query...and possibly modify data in those tables as well.

All of the above implies clients are connecting to cassandra through PlayOrm which I believe is ok for most customers including NREL. Databus sees and displays that information then. Every time a new table is registered, the client/driver just needs to send databus a request that this new table exists along with what database this table is virtually a part of. Reading from this table is then no different than reading from any of the RELATIONAL tables so minimal additions/changes should be needed for something like this.

On top of this, there is search and we had designed a search engine that can create search pages for unknown schemas which is great. There is a page for each set of meta data that exists in the system. Typically we are finding each client/driver has two tables for meta data like ModbusDevice and ModbusDeviceInstance where one is the properties of a device like type, description, model no, and the instance is the more detailed of an instance like serial no, location, etc. One ModbusDevice has many Instances installed on a campus. Our search engine does one search page for ModbusDevice and ModbusDeviceInstance and does one search page for BacnetDevice and BacnetDeviceInstance and as many other pages as you want. I need to talk to justin about how this will work with the above strategy. Also, these search pages are faceted search pages so filters show up after a search and you can filter all results in a certain location.

This search works for our meta data RELATIONAL tables write now and not the above tables at all since databus is being avoided.....maybe relational tables is the only way to go, but each client needs it's specific meta data.

Shorter Term

The Longer term does need modifications to databus so in the short term, we should consider a quicker solution. I am not sure how we can take a playorm entity and map it to an existing relational table so the short term solution would need to be scripted to port it's data to a new format at a later time to obtain the longer term goal.

For the shorter term goal, I can't say it better than Matthew did in his email

I've been thinking about a couple of Databus things I'd like to get some feedback on. I'm going to email them to you, but please let me know if you think I should post this on the github wiki, or something else. Anyway, here is an idea for handling modbus configuration in DataBus. I'd be interested in your feedback and its not urgent.

Problem: Configuration information for sensors and meters based on standard industry protocols is complex. Duplication of configuration information should be avoided, and centralization of configuration information would improve ease-of-use and reliability for a Databus system.

Solution: This description is specific to modbus devices but can be extended to a variety of meter client applications. Create a Database named SENSOR_CONFIG in Databus. Create a Table (type RELATIONAL_TABLE) which has columns which correspond to the entries in the meters.csv file (primary key is Serial Number). Create another Table to hold the modbus configuration information based on the meterModels.json file (primary key is Model Name). Create a robot with read-only access to the SENSOR_CONFIG Database and read/write access to the Database for storing meter readings. The modbus client application is then configured with: -The Databus host url -The Database and Tables with the modbus configuration information for this client -The Database for storing the meter data -The credentials of the robot

I think this can be implemented without changing any existing features of Databus; it just requires the modbus client to read Tables. What do you think?

Here is an idea for handling metadata in DataBus. I'd be interested in your feedback and its not urgent.

Problem: Metadata for data collected from sensors and meters is required for record keeping and system management. This might include the units for data returned by a sensor, the location, installation details, etc. This information changes rarely, and does not need to be included with every entry of sensor data, but it should be stored available for use.

Solution: Create a Table (type RELATIONAL_TABLE) in every Databus Database named "<Name_of_Database>_metadata". Whenever a Table is added to the Database, a corresponding entry is added to the _metadata Table. The metadata Table has columns:

TableName (primary key) Units SensorType Serial_Number Description Location [etc]

The columns are all strings, and with no standard format except for TableName, which is the name of the Table in the Database, and Units, which is a comma-separated list of text units corresponding to the data columns in the Table.

A user can implement this system now on their own. If DataBus supported this method by automatically creating the metadata Table in each Database and adding entries for each Table that was created, then internal mechanisms (Databus charts) and external applications could be designed to use the metadata Table.

Questions: What do you think? How do NREL users manage the metadata now?