Go Back to Announcements List
31st October 2024

[Info] Catalogue API Change: parameters' limits

Dear Customer,

We would like to inform you about an upcoming change to our OData, OpenSearch and STAC API interfaces, effective 12 November 2024.

Starting from 12 November 2024, the number of skipped results will be limited to 10 000 items. Currently, there is no limit on the number of skipped items.

This limit will affect the following parameters in each Catalogue API interface:

  • OData interface: maximum value for the skip parameter will be set to 10 000
  • OpenSearch interface: maximum value for '(page - 1) * maxRecords + index - 1' will be set to 10 000, where by deafult maxRecords = 20, page = 1 and index = 1; maximum value for 'index' will be set to 10001
  • STAC interface: maximum value for '(page - 1) * limit' will be set to 10 000, where by deafult page = 1 and limit = 20

Please find below a list of parameters for each Catalogue API that will be affected by the change as well as the response which will be returned when the limit is exceeded.

The affected query parameters for OData Catalogue interface:OData Catalogue API

skip

OData API request

https://datahub.creodias.eu/odata/v1/Products?$filter=Collection/Name eq 'SENTINEL-3' and ContentDate/Start gt 2024-09-01&$skip=10001&$count=True

Current Response

200 OK
 
{
    "@odata.context": "$metadata#Products",
    "@odata.count": 457588,
    "value": [...],
    "@odata.nextLink": "https://datahub.creodias.eu/odata/v1/Products?%24filter=Collection%2FName+eq+%27SENTINEL-3%27+and+ContentDate%2FStart+gt+2024-09-01&%24skip=10021&%24count=True"
}

New Response

422 Unprocessable Entity
 
{
    "detail": [
        {
            "type": "less_than_equal",
            "loc": [
                "query",
                "$skip"
            ],
            "msg": "Input should be less than or equal to 10000",
            "input": "10001",
            "ctx": {
                "le": 10000
            }
        }
    ]
}

The @odata.nextLink field, which is normally included in the API response to provide the next link for pagination, will not be shown if the skip parameter in the next link exceeds a limit of 10,000. Here's a step-by-step breakdown:

  1. OData query Example:
  2. $skip=10000 means you're telling the API to skip the first 10,000 records.
  3. $top=1 means you want to retrieve only 1 record in the response.
  4. Pagination Logic: The OData API usually provides a @odata.nextLink in the response, which gives the URL to retrieve the next set of records in the sequence. This next link works by adjusting the skip value to keep moving forward through the data.
  5. Skip + Top: In this example, skip + top = 10000 + 1 = 10001. This value exceeds the skip limit of 10,000.
  6. Impact: Because this skip value (10001) exceeds the limit of 10,000, the API will not include the @odata.nextLink in the response. Essentially, the API is signaling that it cannot paginate beyond this point.

Important Note

After the change, when the skip value exceeds 10,000, the API will no longer return the @odata.nextLink for further pagination, meaning you can't retrieve records beyond this limit via pagination using @odata.nextLink.
The affected query parameters for OpenSearch interface:
OpenSearch Catalogue API

maxRecords
page
index

OpenSearch API request

https://datahub.creodias.eu/resto/api/collections/Sentinel1/search.json?maxRecords=15&exactCount=1&index=12&page=667&productType=IW_SLC__1S

Current Response

200 OK
 
{
  "type": "FeatureCollection",
  "properties": {
    "id": "a70c9f16-fc45-5ecb-b1e9-4433f14946f4",
    "totalResults": 2381730,
    "exactCount": 1,
    "startIndex": 10002,
    "itemsPerPage": 15,
    "query": {
      "originalFilters": {
        "productType": "IW_SLC__1S",
        "collection": "SENTINEL-1"
      },
      "appliedFilters": {
        "productType": "IW_SLC__1S",
        "collection": "SENTINEL-1"
      },
      "processingTime": 10.239139726
    },
    "links": [
      {
        "rel": "self",
        "type": "application/json",
        "title": "self",
        "href": "https://datahub.creodias.eu/resto/api/collections/Sentinel1/search.json?maxRecords=15&exactCount=1&index=12&page=667&productType=IW_SLC__1S"
      },
      {
        "rel": "search",
        "type": "application/opensearchdescription+xml",
        "title": "OpenSearch Description Document",
        "href": "https://datahub.creodias.eu/resto/api/collections/Sentinel1/describe.xml"
      },
      {
        "rel": "previous",
        "type": "application/json",
        "title": "previous",
        "href": "https://datahub.creodias.eu/resto/api/collections/Sentinel1/search.json?maxRecords=15&exactCount=1&index=12&productType=IW_SLC__1S&page=666"
      },
      {
        "rel": "first",
        "type": "application/json",
        "title": "first",
        "href": "https://datahub.creodias.eu/resto/api/collections/Sentinel1/search.json?maxRecords=15&exactCount=1&index=12&productType=IW_SLC__1S&page=1"
      },
      {
        "rel": "next",
        "type": "application/json",
        "title": "next",
        "href": "https://datahub.creodias.eu/resto/api/collections/Sentinel1/search.json?maxRecords=15&exactCount=1&index=12&productType=IW_SLC__1S&page=668"
      },
      {
        "rel": "last",
        "type": "application/json",
        "title": "last",
        "href": "https://datahub.creodias.eu/resto/api/collections/Sentinel1/search.json?maxRecords=15&exactCount=1&index=12&productType=IW_SLC__1S&page=158782"
      }
    ]
  },
    "features": [...]
}

New Response

400 Bad Request
 
{
    "detail": {
        "ErrorMessage": "Validation error.",
        "ErrorCode": 400,
        "ErrorDetail": [
            {
                "loc": [
                    "page",
                    "maxRecords",
                    "index"
                ],
                "msg": "The '(page - 1) * maxRecords + index - 1' should be less than 10000. By default, 'maxRecords' is set to 20, while both 'page' and 'index' are set to 1 if not specified."
            }
        ],
        "RequestID": "5ec27fc6-2b2d-4513-b61e-ccc09b8f0bba"
    }
}

Index limit in OpenSearch Catalogue API
OpenSearch API request

https://datahub.creodias.eu/resto/api/collections/Sentinel1/search.json?&index=10002&productType=IW_SLC__1S

Current Response

200 OK
 
{
  "type": "FeatureCollection",
  "properties": {
    "id": "a70c9f16-fc45-5ecb-b1e9-4433f14946f4",
    "totalResults": null,
    "exactCount": 0,
    "startIndex": 10002,
    "itemsPerPage": 20,
    "query": {
      "originalFilters": {
        "productType": "IW_SLC__1S",
        "collection": "SENTINEL-1"
      },
      "appliedFilters": {
        "productType": "IW_SLC__1S",
        "collection": "SENTINEL-1"
      },
      "processingTime": 0.296336496
    },
    "links": [
      {
        "rel": "self",
        "type": "application/json",
        "title": "self",
        "href": "https://datahub.creodias.eu/resto/api/collections/Sentinel1/search.json?&index=10002&productType=IW_SLC__1S"
      },
      {
        "rel": "search",
        "type": "application/opensearchdescription+xml",
        "title": "OpenSearch Description Document",
        "href": "https://datahub.creodias.eu/resto/api/collections/Sentinel1/describe.xml"
      },
      {
        "rel": "next",
        "type": "application/json",
        "title": "next",
        "href": "https://datahub.creodias.eu/resto/api/collections/Sentinel1/search.json?index=10002&productType=IW_SLC__1S&page=2"
      }
    ]
  },
  "features": [...]
}

New Response

400 Bad Request
 
{
  "detail": {
    "ErrorMessage": "Validation error.",
    "ErrorCode": 400,
    "ErrorDetail": [
      {
        "loc": [
          "index"
        ],
        "msg": "Input should be less than or equal to 10001."
      }
    ],
    "RequestID": "09ff2890-c925-497a-a11d-95b308437b8e"
  }
}

Important Note

After the change, when the '(page - 1) * maxRecords + index - 1' value exceeds 10,000, the API will no longer return the 'next' link for further pagination, meaning you can't retrieve records beyond this limit via pagination using 'next' link.
The affected query parameters for STAC Catalogue API interface:
STAC Catalogue API

limit
page

STAC API request

https://datahub.creodias.eu/stac/search?limit=1000&page=12&sortby=datetime

Current Response

200 OK
 
"type": "FeatureCollection",
"features": [...],
"links": [
    {
        "rel": "next",
        "href": "https://datahub.creodias.eu/stac/search?limit=1000&page=13&sortby=datetime",
        "type": "application/json"
    },
    {
        "rel": "prev",
        "href": "https://datahub.creodias.eu/stac/search?limit=1000&page=11&sortby=datetime",
        "type": "application/json"
    },
    {
        "rel": "first",
        "href": "https://datahub.creodias.eu/stac/search?limit=1000&page=1&sortby=datetime",
        "type": "application/json"
    },
    {
        "rel": "self",
        "href": "https://datahub.creodias.eu/stac/search?limit=1000&page=12&sortby=datetime",
        "type": "application/json"
    },
    {
        "rel": "root",
        "href": "https://datahub.creodias.eu/stac",
        "type": "application/json"
    }
 ]
}

New Response

400 Bad Request
 
{
    "code": "400",
    "description": "The '(page - 1) * limit' should be less than 10000. By default, 'limit' is set to 20 and 'page' is set to 1 if not specified."
    "request_id": "4e681b43-5b47-4437-a0b1-f11b3c36a24a"
}

Important Note

After the change, when the '(page - 1) * limit' value exceeds 10,000, the API will no longer return the 'next' link for further pagination, meaning you can't retrieve records beyond this limit via pagination using 'next' link.

We recommend reviewing the upcoming changes to Catalogue OData, OpenSearch and STAC API interfaces described above to avoid disruption to your current scripts or apps.

Kind regards,
CREODIAS Team