Incoming data structures
Registering your own data structures goes hand in hand with write your own collector guide.
Most likely you will be extending UDR micro service as it is the entry point of the RCB Cyclops framework. However, if you are playing with the Open Source code, you can use this guide for other micro services as well.
First, navigate to
core/udr/src/main/java/ch/icclab/cyclops/consume/data/model
and create a new class that implements DataMapping interface. The name of the class needs to correspond to the _class field of your JSON messages.
Return String name of the JSON field that holds Long timestamp, or null if you want to have timestamp generated for you automatically.
Return TimeUnit of the respective timestamp field, or null if you want to have default behaviour (automatic generation is based on milliseconds).
Return List<String> including tag names of your JSON message, or null if everything should be persisted as fields. To understand the difference between tags and fields, please read the following article.
In case that you need to preprocess, harmonise or simply update incoming data frames, you can do so by implementing function that takes Map original parameter, and returns Map object. This could prove to be useful if your system is sending time stamps as Strings and you need to convert them into Long first. If you don't want to preprocess your data frames, simply return null and the original object will be used instead.
Return true if your preference is to automatically publish incoming data frames to micro service's exchange, once they have been successfully persisted in time series database. This can be useful for cases of stream processing.
A rule of thumb is that if you are ingesting Usage Records records in UDR micro service and generating UDR records via GenerateUDR command, then you should have stream processing for this data frame set to false.
Based on your preference in the previous method, you can select whether you want to have messages published over micro service's broadcast exchange (if returning false), or sent out using a routing key (class name) to its dispatch exchange (for true).
An example would be a Usage Data class for UDR micro service, that corresponds to the following JSON:
{
"_class": "OpenStackCeilometerCPU",
"time": 1467642128,
"account": "martin",
"usage": 20,
"unit": "s",
"metadata": {
"source": "VM1",
"project_id": "123"
}
}
Here we are specifying that class in question is OpenStackCeilometerCPU, numeric time field is time in TimeUnit.SECONDS, and we want to have account stored as tag (everything else as fields).
import ch.icclab.cyclops.consume.data.DataMapping;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.TimeUnit;
public class OpenStackCeilometerCPU implements DataMapping {
@Override
public String getTimeField() {
return "time";
}
@Override
public TimeUnit getTimeUnit() {
return TimeUnit.SECONDS;
}
@Override
public List<String> getTagNames() {
List<String> list = new ArrayList<>();
list.add("account");
return list;
}
@Override
public Map preProcess(Map original) {
return null;
}
@Override
public Boolean shouldPublish() {
return false;
}
@Override
public Boolean doNotBroadcastButRoute() {
return false;
}
}
After this point you will need to recompile selected micro service and redeploy everything, to do so read the following guide.
If the compilation fails, or your mapping doesn't work, make sure you are implementing mentioned interface, class name corresponds to _class field and Java class is located as instructed above.
