The web client allows the user to build Json queries to Skyminer. This section gives some further information on the parameter within the Json query when the Search mode is activated.
Correlation search queries have five parameter categories:
reference_series: it is the query for the series that should be searched. It should be a complete KairosDB query metric with only one metric and that generate only one series (no grouping without vertical aggregation).
searched_series: it is the query for the set of series that should be compared to the reference series. It should be a complete KairosDB query metric, possibly with many metrics and tag group bys.
similarity_measure: similarity measure is the method used to measure similarity between two series. Currently implemented are discarding_linear_correlation and dtw. A similarity measure goes with a reliability threshold (reliability_threshold) that indicates the minimum number of samples used in the correlation that are necessary. If it is under the threshold, correlation score is set to 0. Data would typically be under this threshold if one of the series had only a few samples or if both series had unsynchronized sampling. You can find details on similarity measures and their parameter in Details on similarity measures.
Discarding linear correlation is the classical Pearson correlation.
DTW is the dynamic time warping distance measure.
result_format: it is the only optional parameter of the correlation query. It has three parameters:
threshold: only returns correlated series that have correlation scores above this threshold. Usually a double between 0 and 1.
best: integer n indicating that only the best n series should be returned.
computation_method: implemented are sequential and threaded. Threaded uses every available cores on the server.
The following Json shows the structure of the query:
{
"reference_series": {… KairosDB query …},
"searched_series": {… KairosDB query …},
"similarity_measure": { … Similarity measure parameters (“threshold”,…) …},
"result_format": { … Result format parameters (“best”,”threshold”,…) …},
"computation_method": { … Computation method parameters (“name”,…) … }
}
The result is separated in two parts:
query: the query that actually generated this response
results: The actual response to the query
The results sub-object has four parameters:
reference_series: the description of the reference series, in the same format as in usual KairosDB responses.
correlated_series: a table with p elements describing the p series that have been correlated to the reference series.
correlation_score: a table with p numbers (usually between 0 and 1) representing the correlation scores of the p correlated series.
correlation_reliability: a table with p numbers representing an indicator of the reliability of the correlation scores of the p correlated series.
Here is an example response:
{
"query": {
… A correlation query as presented before …
},
"results": {
"reference_series": {
"name": "kairosdb.protocol.http_request_count",
"tags": {
"host": ["skyminer01"],
"method": ["metricnames","query","search","tags"]
},
"query_index": 0
},
"correlated_series": [
{
"name": "kairosdb.protocol.http_request_count",
"group_by": [
{
"name": "tag",
"tags": ["method"],
"group": {"method": "tags”}
}
],
"tags": {
"host": ["skyminer01"],
"method": ["tags"]
},
"query_index": 0
},
{
"name": "kairosdb.protocol.http_request_count",
"group_by": [
{
"name": "tag",
"tags": ["method"],
"group": {"method": "query"}
}
],
"tags": {
"host": ["skyminer01"],
"method": ["query"]
},
"query_index": 0
},
{
"name": "kairosdb.protocol.http_request_count",
"group_by": [
{
"name": "tag",
"tags": ["method"],
"group": {"method": "metricnames"}
}
],
"tags": {
"host": ["skyminer01"],
"method": [ "metricnames" ]
},
"query_index": 0
},
{
"name": "kairosdb.protocol.http_request_count",
"group_by": [
{
"name": "tag",
"tags": ["method"],
"group": {"method": "search"}
}
],
"tags": {
"host": ["skyminer01"],
"method": ["search"]
},
"query_index": 0
}
],
"correlation_score": [0.95011,0.89177,0.77440,0],
"correlation_reliability": [60,60,60,1]
}
}
The largest part of the response is the description of the correlated series. To avoid generating overly long responses it is crucial to properly define the best and threshold parameters in the result format.
This section gives some further information on the parameter within the Json query when the Matrix mode is activated.
Correlation search queries have four parameters, because they do not have a reference series.
{
"searched_series": {… KairosDB query …},
"similarity_measure": {… Similarity measure parameters (“threshold”,…) …},
"result_format": {… Result format parameters (“best”,”threshold”,”sparse”,…) …},
"computation_method": {… Computation method parameters (“name”,…) …}
}
There are 2 other differences between matrix query API and search query API:
result_format has two extra boolean options called sparse and return_perfect_scores. When the best option and/or the threshold option are used, Skyminer only returns the correlations that match this criteria. But there will be cross correlations between the series involved in this selected correlations that might interest the user. By setting sparse to false, those are also returned in the response.
computation_method: implemented methods are memory_cached and threaded_memory_cached. The second one is usually faster.
The result is separated is separated in the same two parts as for Search queries: quer and results.
The results sub-object has two parameters:
correlated_series: a table with p elements describing the p series that have been correlated to the reference series.
correlation_score: a table with p numbers (usually between 0 and 1) representing the correlation scores of the p correlated series.
Here is an example response:
{
"query": {
… A correlation query as presented before …
},
"results": {
"correlated_series": [
{
"name": "kairosdb.protocol.http_request_count",
"group_by": [
{
"name": "tag",
"tags": ["method"],
"group": {"method": "tags”}
}
],
"tags": {
"host": ["skyminer01"],
"method": ["tags"]
},
"query_index": 0
},
{
"name": "kairosdb.protocol.http_request_count",
"group_by": [
{
"name": "tag",
"tags": ["method"],
"group": {"method": "query"}
}
],
"tags": {
"host": ["skyminer01"],
"method": ["query"]
},
"query_index": 0
},
{
"name": "kairosdb.protocol.http_request_count",
"group_by": [
{
"name": "tag",
"tags": ["method"],
"group": {"method": "metricnames"}
}
],
"tags": {
"host": ["skyminer01"],
"method": [ "metricnames" ]
},
"query_index": 0
},
{
"name": "kairosdb.protocol.http_request_count",
"group_by": [
{
"name": "tag",
"tags": ["method"],
"group": {"method": "search"}
}
],
"tags": {
"host": ["skyminer01"],
"method": ["search"]
},
"query_index": 0
}
],
"correlation_score": [[0,1,0.95011,20], [0,2,0.86545,20], [1,2,0.47011,10]],
}
}