Getting Updates

On API is a set of streaming API endpointsfiles to stay up to date with the latest data.  Gracenote assigns each customer their own data stream containing only the data pertinent to their subscription.  Anytime a change is made on any of the On API endpoints, an incremental ‘Update ID’ is assigned.  You can keep your data up to date by periodically querying for updates that have occurred since the last ingested ‘Update ID.’

Each On API endpoint response includes a nextUpdateId field and a maxUpdateId field within the streamData node. Use the nextUpdateId field to start your next update query. Use the maxUpdateId field to determine when to stop the query.

Important: Due to the large volume of data entitlements, Gracenote engineering has enabled a limit parameter to control batch size of each request. We recommend ingesting with a limit of 1000 records per query across all endpoint s except Lineups. The maximum allowable limit for the Lineups endpoint is 10.

We do our best to optimize the API for all of our global customers. However, if you encounter performance issues like the response taking too long, or not getting a response at all, we recommend retrying with a reduced batch size.

General Update Process

Use this process to update endpointsfiles:

  1. Retrieve full datasets use On API with a request parameter of updateId=0.
  2. For each iteration, use the nextUpdateId provided under streamData to receive the next dataset.
  3. Use the limit parameter to control ingest batch size, iterating until no further updates are returned in response. You can confirm this by checking whether the maxUpdateId under streamData provided in the last request matches your updateId. Also, if there are no further updates available, nextUpdateId will not be provided in the returned response.
  4. Retrieve updates in hourly, or other interval, using the received maxUpdateId +1. Iterate until no further updates are returned.
  5. You can restart ingestion at any point by reverting to an earlier updateId, or reseeding data from updateId=0.

Note: Also see endpoint -Specific Update Requests.

Update Sequence

After you have seeded all data and are caught up with the latest endpointsfiles using updateIdfor each entity, use the following sequence of API requests to receive updates:

  1. Programs
  2. Sources
  3. Schedules
  4. Celebrities
  5. Other Core endpointsfiles
  6. Sports-related (Teams, Venues, Organizations, Universities)

Data Update Frequency

Data is continuously being added and updated to the Gracenote database and made available via On API. The metrics below are for general guidance purposes and are based on past experience of data flow, but more or less data may be observed at any time, given operational and editorial activities. You should plan for potential spikes in data output and for the potential of no available updates during periods of time. Note that all metrics below are based on delivery of updates after receiving an initial full dataset.

Entity

Daily Average

High

Sources

175

300

Programs

150,000

400,000

Celebrities

1,500

5,000

Schedules*

120,000

250,000

* Schedule updates are delivered based on station+day.  At midnight UTC, one full new day is delivered for each channel included in the customer dataset, to extend the schedule window.  Typical mid-day editorial schedule updates is delivered throughout the day as they occur. Due to the nature of schedule updates, frequency of updates will vary based on the customer dataset. Numbers stated above are for North American (USA+CAN) schedule data.

API Response Limits

Gracenote recommends a maximum limit of 1000 across all endpoint s except Lineups. The maximum allowable limit for the Lineups endpoint is 10. You can override this value with the "limit" parameter if smaller payloads are preferred for processing.

You are expected to continue to request updates from the API, incrementing the updateIdvalue per entity using the nextUpdateId value available in the response header or streamData element in the body, until no more updates are returned and then request again based on your desired polling frequency.

API Update Helpers

All API responses contain the streamData elements that are intended to make the processing of requesting updates simpler. This information is available in the response header as well as the body for clients that process the response body separately from the update retrieval process.

The nextUpdateId provides the updateIdthat should be specified in the next request to pull the next set of records. If the current response is the last set of updates available for an endpoint, the nextUpdateId will not exist.

The maxUpdateId specifies the maximum updateIdavailable for the specific endpoint at the time the response was provided to the client. It is possible that this is not the actual maximum updateIdas new updates can continue to flow in.

Data Recovery

Gracenote offers options for data recovery.  In extreme cases of complete data loss, you can restart ingestion from updateId=0. The On API data stream only maintains the newest (highest updateId) version of an object, so you will not be streaming through every historical update. However, this data refresh can be a lengthy process. 

In the event that ingestion stopped, but existing data was maintained, you can resume ingestion based on your last known update ID.  If you do not know the last updateIdused, you can instead use a date/time parameter to restart ingestion and then switch back to using update IDs.

Alternatively, you can FTP seed files directly. See Getting Seed Files for instructions.

FAQ

This section answers frequently-asked questions about updates.

Do you support for HTTPS request?

Clients can make calls via HTTP or HTTPS. No additional configuration is required for SSL.

What is the maximum allowable limit to retrieve objects across all On API endpoint s?

Gracenote recommends a maximum limit of 1000 across all endpoint s except Lineups. The maximum allowable limit for the Lineups endpoint is 10. This means any API queries for the Lineups endpoint s requesting more than 10 objects will be automatically throttled to return 10 objects.

How do we get an initial On API dataset?

Use On API to make single-threaded, sequential requests based on the last updateIdreceived. For a full set, use updateId=0. See Getting Datasets and Updates.

Can I make multi-threaded requests to the API using different updateIdranges to get an initial dataset?

We do not currently provide a means to partition updateIdranges for seeding purposes. This is being considered as a future roadmap item. Current seeding requires single-threaded, sequential requests based on the last updateIdreceived.

Are update tokens (updateId) assigned per method, or across all methods?

The updateIdtoken is unique per method. The last updateIdreceived should be stored per method for delta retrievals. Sample storage of updateId:

Entity

Max updateIdreceived
Programs 20248923
Schedules 49275382
Sources 3029384
Celebrities 12048273
ProgramMappings 287077806

How and when are additional schedule days added to my Schedules dataset?

A daily process runs at midnight UTC to add one additional schedule day per station to your dataset. You will see a batch of updates at that time corresponding to the additional schedule days. This daily process ensures that you can use a continuous process to pull both editorial updates and new schedules days using same updateId.

What does updateDate mean? Can I use it to get deltas since a certain timestamp?

The updateDate is included for informational purposes only, and represents the timestamp the entity was updated or became available to the customer dataset. For instance, when new schedule days become available shortly after midnight UTC each day, they are marked with updateDate at that point to indicate when they were assigned new updateIds for retrieval. Any subsequent updates to those schedule days will receive a new updateIdand updateDate. The updateDate is not available as a request parameter.

How can I tell when I have received all available updates?

When a response no longer has a nextUpdateId or the maxUpdateId is the same as your updateIdtoken, quest based on most recent updateIdreturns no objects, this is indication that all available updates have been received. If your system is polling for new updates, the response will contain no objects if there are no new updates available. Also, the maxUpdateId in the streamData element will match your updateIdtoken.