Limits

Overview

When querying Skyminer, the data returned can be very large. To prevent the response time from being excessively long, it is usually a good idea to limit the number of points returned by Skyminer.

Description

Queries can become overwhelming either horizontally, that is to say that there are too many datapoints for one of the metrics returned, or vertically, when a group by returns too many groups.

Horizontal limits

There are two mechanism to limit the number of datapoints in each series returned.

  • You can use the limit parameter of the metric. This will stop returning points after the limit is reached, but will not throw an exception.

  • You can use the limit aggregator. This will stop returning points and return an error message if the limit is reached.

Safeguard

Safeguard is a mechanism used to facilitate the usage of horizontal and vertical limits.

In the query UI, the padlock button can be used to define 3 parameters:

  • Number of series: maximum number of groups returned by a group_by (vertical limit)

  • Limit before aggregation: maximum number of datapoints for each series, before aggregation (horizontal limit)

  • Limit after aggregation: maximum number of datapoints for each series, after aggregation (horizontal limit)

../_images/limits-ui.png

These limits are set by default to avoid accidental disproportionally large queries.

Usage in API

The different kinds of limits can be set in different ways. This section does not present new API parameters as limit mechanisms have already been introduced in the general presentation of the metric object and in the section on aggregators.

Horizontal limits

The limit aggregator can be used as any aggregator. The name of the aggregator is limit and there is only one parameter:

  • limit: the maximum number of datapoints for each series returned.

This aggregator throws an error when the limit is reached, and returns a response with status code 500:

{
    "errors": [
        "Limit Aggregator: query exceeded limit of 10000 data points, please increase limits or add filtering/downsampling"
    ]
}

An alternative is the limit attribute, a root attribute of the metric object. It does not throw an error when the limit is reached but stops returning datapoints.

Vertical limits

The vertical limits are done through the group_limit attribute of the metric. It sets the maximum number of groups returned by the group by.

...
{
    "tags": {... tags for filtering ...},
    "name": "abc.123",
    "limit": 1000,
    "group_limit": 100,
    "aggregators": [... a list of aggregators ...]
}
...

An error message is returned if the limit is overstepped with status code 500:

{
    "errors": [
        "Groups Limit: exceeded limit of 100 groups in a single metric query, please increase limits, change grouping or add filters on tags"
    ]
}

Safeguard

Alternatively you can use the safeguard object to define those limits:

{
    "safeguard": {
        "group_limit": 100,
        "limit_before_aggregation": 10000000,
        "limit_after_aggregation": 10000
    },
    "metrics": [
        ...
    ]
}