Catalog
The catalog is a resource that records all package operations on a package source, such as creations and deletions. The catalog resource has the Catalog type in the service index. You can use this resource to query for all published packages.
Note
Because the catalog is not used by the official NuGet client, not all package sources implement the catalog.
Note
Currently, the nuget.org catalog is not available in China. For more details, see NuGet/NuGetGallery#4949.
Versioning
The following @type value is used:
| @type value | Notes |
|---|---|
| Catalog/3.0.0 | The initial release |
Base URL
The entry point URL for the following APIs is the value of the @id property associated with the aforementioned
resource @type values. This topic uses the placeholder URL {@id}.
HTTP methods
All URLs found in the catalog resource support only the HTTP methods GET and HEAD.
Catalog index
The catalog index is a document in a well-known location that has a list of catalog items, ordered chronologically. It is the entry point of the catalog resource.
The index is comprised of catalog pages. Each catalog page contains catalog items. Each catalog item represents an event concerning a single package at a point in time. A catalog item can represent a package that was created, unlisted, relisted, or deleted from the package source. By processing the catalog items in chronological order, the client can build an up-to-date view of every package that exists on the V3 package source.
In short, catalog blobs have the following hierarchical structure:
- Index: the entry point for the catalog.
- Page: a grouping of catalog items.
- Leaf: a document representing a catalog item, which is a snapshot of a single package's state.
Each catalog object has a property called the commitTimeStamp representing when the item was added to the catalog. Catalog items are added to a catalog page in batches called commits. All catalog items in the same commit have the same commit timestamp (commitTimeStamp) and commit ID (commitId). Catalog items placed in the same commit represent events that happened around the same point in time on the package source. There is no ordering within a catalog commit.
Because each package ID and version is unique, there will never be more than one catalog item in a commit. This ensures that catalog items for a single package can always be unambiguously ordered with respect to commit timestamp.
There is never be more than one commit to the catalog per commitTimeStamp. In other words, the commitId is redundant with the commitTimeStamp.
In contrast to the package metadata resource, which is indexed by package ID, the catalog is indexed (and queryable) only by time.
Catalog items are always added to the catalog in a monotonically increasing, chronological order. This means that if a catalog commit is added at time X then no catalog commit will ever be added with a time less than or equal to X.
The following request fetches the catalog index.
GET {@id}
The catalog index is a JSON document that contains an object with the following properties:
| Name | Type | Required | Notes |
|---|---|---|---|
| commitId | string | yes | A unique ID associated with the most recent commit |
| commitTimeStamp | string | yes | A timestamp of the most recent commit |
| count | integer | yes | The number of pages in the index |
| items | array of objects | yes | A array of objects, each object representing a page |
Each element in the items array is an object with some minimal details about each page. These page objects do not
contain the catalog leaves (items). The order of the elements in this array is not defined. Pages can be ordered by the
client in memory using their commitTimeStamp property.
As new pages are introduced, the count will be incremented and new objects will appear in the items array.
As items are added to the catalog, the index's commitId will change and the commitTimeStamp will increase. These
two properties are essentially a summary over all page commitId and commitTimeStamp values in the items array.
Catalog page object in the index
The catalog page objects found in the catalog index's items property have the following properties:
| Name | Type | Required | Notes |
|---|---|---|---|
| @id | string | yes | The URL to fetch catalog page |
| commitId | string | yes | A unique ID associated with the most recent commit in this page |
| commitTimeStamp | string | yes | A timestamp of the most recent commit in this page |
| count | integer | yes | The number of items in the catalog page |
In contrast to the package metadata resource which in some cases inlines leaves
into the index, catalog leaves are never inlined into the index and must always be fetched by using the page's @id
URL.
Sample request
GET https://api.nuget.org/v3/catalog0/index.json
Sample response
{
"commitId": "3d698852-eefb-48ed-8f55-9ee357540d20",
"commitTimeStamp": "2017-10-31T23:33:17.0954363Z",
"count": 3,
"items": [
{
"@id": "https://api.nuget.org/v3/catalog0/page0.json",
"commitId": "3a4df280-3d86-458e-a713-4c91ca261fef",
"commitTimeStamp": "2015-02-01T06:30:11.7477681Z",
"count": 540
},
{
"@id": "https://api.nuget.org/v3/catalog0/page1.json",
"commitId": "8bcd3cbf-74f0-47a2-a7ae-b7ecc50005d3",
"commitTimeStamp": "2015-02-01T06:39:53.9553899Z",
"count": 540
},
{
"@id": "https://api.nuget.org/v3/catalog0/page2.json",
"commitId": "3d698852-eefb-48ed-8f55-9ee357540d20",
"commitTimeStamp": "2017-10-31T23:33:17.0954363Z",
"count": 47
}
]
}
Catalog page
The catalog page is a collection of catalog items. It is a document fetched using one of the @id values found in the
catalog index. The URL to a catalog page is not intended to be predictable and should be discovered using only the
catalog index.
New catalog items are added to the page in the catalog index only with the highest commit timestamp or to a new page. Once a page with a higher commit timestamp is added to the catalog, older pages are never added to or changed.
The catalog page document is a JSON object with the following properties:
| Name | Type | Required | Notes |
|---|---|---|---|
| commitId | string | yes | A unique ID associated with the most recent commit in this page |
| commitTimeStamp | string | yes | A timestamp of the most recent commit in this page |
| count | integer | yes | The number of items in the page |
| items | array of objects | yes | The catalog items in this page |
| parent | string | yes | A URL to the catalog index |
Each element in the items array is an object with some minimal details about the catalog item. These item objects do
not contain all of the catalog item's data. The order of the items in the page's items array is not defined. Items
can be ordered by the client in memory using their commitTimeStamp property.
The number of catalog items in a page is defined by server implementation. For nuget.org, there are at most 550 items in each page, although the actual number may be smaller for some pages depending on the size of the next commit batch at the point in time.
As new items are introduced, the count is incremented and new catalog item objects appear in the items
array.
As items are added to the page, the commitId changes and the commitTimeStamp increases. These two
properties are essentially a summary over all commitId and commitTimeStamp values in the items array.
Catalog item object in a page
The catalog item objects found in the catalog page's items property have the following properties:
| Name | Type | Required | Notes |
|---|---|---|---|
| @id | string | yes | The URL to fetch the catalog item |
| @type | string | yes | The type of the catalog item |
| commitId | string | yes | The commit ID associated with this catalog item |
| commitTimeStamp | string | yes | The commit timestamp of this catalog item |
| nuget:id | string | yes | The package ID that this leaf is related to |
| nuget:version | string | yes | The package version that this leaf is related to |
The @type value will be one of the following two values:
nuget:PackageDetails: this corresponds toPackageDetailstype in the catalog leaf document.nuget:PackageDelete: this corresponds to thePackageDeletetype in the catalog leaf document.
For more details about what each type means, see the corresponding items type below.
Sample request
GET https://api.nuget.org/v3/catalog0/page2926.json
Sample response
{
"commitId": "616117f5-d9dd-4664-82b9-74d87169bbe9",
"commitTimeStamp": "2017-10-31T23:30:32.4197849Z",
"count": 5,
"parent": "https://api.nuget.org/v3/catalog0/index.json",
"items": [
{
"@id": "https://api.nuget.org/v3/catalog0/data/2017.10.31.23.30.32/util.biz.payments.0.0.4-preview.json",
"@type": "nuget:PackageDetails",
"commitId": "616117f5-d9dd-4664-82b9-74d87169bbe9",
"commitTimeStamp": "2017-10-31T23:30:32.4197849Z",
"nuget:id": "Util.Biz.Payments",
"nuget:version": "0.0.4-preview"
},
{
"@id": "https://api.nuget.org/v3/catalog0/data/2017.10.31.23.28.02/util.biz.0.0.4-preview.json",
"@type": "nuget:PackageDetails",
"commitId": "820340b2-97e3-4f93-b82e-bc85550a6560",
"commitTimeStamp": "2017-10-31T23:28:02.788239Z",
"nuget:id": "Util.Biz",
"nuget:version": "0.0.4-preview"
},
{
"@id": "https://api.nuget.org/v3/catalog0/data/2017.10.31.22.31.22/sourcecode.clay.data.1.0.0-preview1-00258.json",
"@type": "nuget:PackageDetails",
"commitId": "cae34527-ffc7-4e96-884f-7cf95a32dbdd",
"commitTimeStamp": "2017-10-31T22:31:22.5169519Z",
"nuget:id": "SourceCode.Clay.Data",
"nuget:version": "1.0.0-preview1-00258"
},
{
"@id": "https://api.nuget.org/v3/catalog0/data/2017.10.31.22.31.22/sourcecode.clay.1.0.0-preview1-00258.json",
"@type": "nuget:PackageDetails",
"commitId": "cae34527-ffc7-4e96-884f-7cf95a32dbdd",
"commitTimeStamp": "2017-10-31T22:31:22.5169519Z",
"nuget:id": "SourceCode.Clay",
"nuget:version": "1.0.0-preview1-00258"
},
{
"@id": "https://api.nuget.org/v3/catalog0/data/2017.10.31.22.31.22/sourcecode.clay.json.1.0.0-preview1-00258.json",
"@type": "nuget:PackageDetails",
"commitId": "cae34527-ffc7-4e96-884f-7cf95a32dbdd",
"commitTimeStamp": "2017-10-31T22:31:22.5169519Z",
"nuget:id": "SourceCode.Clay.Json",
"nuget:version": "1.0.0-preview1-00258"
}
]
}
Catalog leaf
The catalog leaf contains metadata about a specific package ID and version at some point in time. It is a document
fetched using the @id value found in a catalog page. The URL to a catalog leaf is not intended to be predictable
and should be discovered using only a catalog page.
The catalog leaf document is a JSON object with the following properties:
| Name | Type | Required | Notes |
|---|---|---|---|
| @type | string or array of strings | yes | The type(s) of the catalog item |
| catalog:commitId | string | yes | A commit ID associated with this catalog item |
| catalog:commitTimeStamp | string | yes | The commit timestamp of this catalog item |
| id | string | yes | The package ID of the catalog item |
| published | string | yes | The published date of the package catalog item |
| version | string | yes | The package version of the catalog item |
Item types
The @type property is a string or array of strings. For convenience, if the @type value is a string, it should be
treated as any array of size one. Not all possible values for @type are documented. However, each catalog item has exactly one of the two following string type values:
PackageDetails: represents a snapshot of package metadataPackageDelete: represents a package that was deleted
Package details catalog items
Catalog items with the type PackageDetails contain a snapshot of package metadata for a specific package (ID and
version combination). A package details catalog item is produced when a package source encounters any of the
following scenarios:
- A package is pushed.
- A package is relisted.
- A package is unlisted.
- A package is deprecated.
- A package is undeprecated.
- A package is reflowed.
- A package's vulnerability status is updated.
A package reflow is an administrative gesture that essentially generates a fake push of an existing package with no changes to the package itself. On nuget.org, a reflow is used after fixing a bug in one of the background jobs which consume the catalog.
Clients consuming the catalog items should not attempt to determine which of these scenarios produced the catalog item. Instead, the client should simply update any maintained view or index with the metadata contained in the catalog item. Furthermore, duplicate or redundant catalog items should be handled gracefully (idempotently).
Package details catalog items have the following properties in addition to those included on all catalog leaves.
| Name | Type | Required | Notes |
|---|---|---|---|
| authors | string | no | |
| created | string | no | A timestamp of when the package was first created. Fallback property: published. |
| dependencyGroups | array of objects | no | The dependencies of the package, grouped by target framework (same format as the package metadata resource) |
| deprecation | object | no | The deprecation associated with the package (same format as the package metadata resource) |
| description | string | no | |
| iconUrl | string | no | |
| isPrerelease | boolean | no | Whether or not the package version is prerelease. Can be detected from version. |
| language | string | no | |
| licenseUrl | string | no | |
| listed | boolean | no | Whether or not the package is listed |
| minClientVersion | string | no | |
| packageHash | string | yes | The hash of the package, encoding using standard base 64 |
| packageHashAlgorithm | string | yes | |
| packageSize | integer | yes | The size of the package .nupkg in bytes |
| packageTypes | array of objects | no | The package types specified by the author. |
| projectUrl | string | no | |
| releaseNotes | string | no | |
| requireLicenseAgreement | boolean | no | Assume false if excluded |
| summary | string | no | |
| tags | array of strings | no | |
| title | string | no | |
| verbatimVersion | string | no | The version string as it's originally found in the .nuspec |
| vulnerabilities | array of objects | no | The security vulnerabilities of the package |
The package version property is the full version string after normalization. This means that SemVer 2.0.0 build data can
be included here.
The created timestamp is when the package was first received by the package source, which is typically a short
time before the catalog item's commit timestamp.
The packageHashAlgorithm is a string defined by the server implementation representing the hashing algorithm used to
produce the packageHash. nuget.org always used the packageHashAlgorithm value of SHA512.
The packageTypes property will only be present if a package type was specified by the author. If it is present, it will always have at least one (1) entry. Each item in the packageTypes array is a JSON object with the following properties:
| Name | Type | Required | Notes |
|---|---|---|---|
| name | string | yes | The name of the package type. |
| version | string | no | The version of the package type. Only present if the author explicitly specified a version in the nuspec. |
The published timestamp is the time when the package was last listed.
Note
On nuget.org, the published value is set to the year 1900 when the package is unlisted.
Vulnerabilities
An array of vulnerability objects. Each vulnerability has the following properties:
| Name | Type | Required | Notes |
|---|---|---|---|
| advisoryUrl | string | yes | Location of security advisory for the package |
| severity | string | yes | Severity of advisory: "0" = Low, "1" = Moderate, "2" = High, "3" = Critical |
If the severity property contains values other than those listed here, the severity of the advisory is to be treated as Low.
Sample request
GET https://api.nuget.org/v3/catalog0/data/2015.02.01.11.18.40/windowsazure.storage.1.0.0.json
Sample response
{
"@type": [
"PackageDetails",
"catalog:Permalink"
],
"authors": "NuGet.org Team",
"catalog:commitId": "49fe04d8-5694-45a5-9822-3be61bda871b",
"catalog:commitTimeStamp": "2015-02-01T11:18:40.8589193Z",
"created": "2011-12-02T20:21:23.74Z",
"description": "This package is an example for the V3 protocol.",
"deprecation": {
"reasons": [
"Legacy",
"HasCriticalBugs",
"Other"
],
"message": "This package is an example--it should not be used!",
"alternatePackage": {
"id": "Newtonsoft.JSON",
"range": "12.0.2"
}
},
"iconUrl": "https://www.nuget.org/Content/gallery/img/default-package-icon.svg",
"id": "NuGet.Protocol.V3.Example",
"isPrerelease": false,
"language": "en-US",
"licenseUrl": "http://www.opensource.org/licenses/ms-pl",
"packageHash": "2edCwKLcbcgFJpsAwa883BLtOy8bZpWwbQpiIb71E74k5t2f2WzXEGWbPwntRleUEgSrcxJrh9Orm/TAmgO4NQ==",
"packageHashAlgorithm": "SHA512",
"packageSize": 118348,
"packageTypes": [
{
"@id": "https://api.nuget.org/v3/catalog0/data/2015.02.01.11.18.40/windowsazure.storage.1.0.0.json#packagetypes/DotnetTool",
"@type": "PackageType",
"name": "DotnetTool"
}
],
"projectUrl": "https://github.com/NuGet/NuGetGallery",
"published": "1900-01-01T00:00:00Z",
"requireLicenseAcceptance": false,
"title": "NuGet V3 Protocol Example",
"version": "1.0.0",
"dependencyGroups": [
{
"@id": "https://api.nuget.org/v3/catalog0/data/2015.02.01.11.18.40/windowsazure.storage.1.0.0.json#dependencygroup",
"@type": "PackageDependencyGroup",
"dependencies": [
{
"@id": "https://api.nuget.org/v3/catalog0/data/2015.02.01.11.18.40/windowsazure.storage.1.0.0.json#dependencygroup/aspnet.suppressformsredirect",
"@type": "PackageDependency",
"id": "aspnet.suppressformsredirect",
"range": "[0.0.1.4, )"
},
{
"@id": "https://api.nuget.org/v3/catalog0/data/2015.02.01.11.18.40/windowsazure.storage.1.0.0.json#dependencygroup/webactivator",
"@type": "PackageDependency",
"id": "WebActivator",
"range": "[1.4.4, )"
},
{
"@id": "https://api.nuget.org/v3/catalog0/data/2015.02.01.11.18.40/windowsazure.storage.1.0.0.json#dependencygroup/webapi.all",
"@type": "PackageDependency",
"id": "WebApi.All",
"range": "[0.5.0, )"
}
],
"targetFramework": ".NETFramework4.6"
}
],
"tags": [
"NuGet",
"V3",
"Protocol",
"Example"
],
"vulnerabilities": [
{
"@id": "https://api.nuget.org/v3/catalog0/data/2015.02.01.11.18.40/windowsazure.storage.1.0.0.json#vulnerability/GitHub/999",
"@type": "Vulnerability",
"advisoryUrl": "https://github.com/advisories/ABCD-1234-5678-9012",
"severity": "2"
}
]
}
Package delete catalog items
Catalog items with the type PackageDelete contain a minimal set of information indicating to catalog clients that a
package has been deleted from the package source and is no longer available for any package operation (such as
restore).
Note
It is possible for a package to be deleted and later republished using the same package ID and version. On nuget.org, this is a very rare case as it breaks the official client's assumption that a package ID and version imply a specific package content. For more information about package deletion on nuget.org, see our policy.
Package delete catalog items have no additional properties in addition to those included on all catalog leaves.
The version property is the original version string found in the package .nuspec.
The published property is the time when package was deleted, which is typically as short time before the catalog
item's commit timestamp.
Sample request
GET https://api.nuget.org/v3/catalog0/data/2017.11.02.00.40.00/netstandard1.4_lib.1.0.0-test.json
Sample response
{
"@type": [
"PackageDelete",
"catalog:Permalink"
],
"catalog:commitId": "19fec5b4-9335-4e4b-bd50-8d5d3f734597",
"catalog:commitTimeStamp": "2017-11-02T00:40:00.1969812Z",
"id": "netstandard1.4_lib",
"originalId": "netstandard1.4_lib",
"published": "2017-11-02T00:37:43.7181952Z",
"version": "1.0.0-test"
}
Cursor
Overview
This section describes a client concept that, although is not necessarily mandated by the protocol, should be part of any practical catalog client implementation.
Because the catalog is an append-only data structure indexed by time, the client should store a cursor locally,
representing up to what point in time the client has processed catalog items. Note that this cursor value should never
be generated using the client's machine clock. Instead, the value should come from a catalog object's
commitTimestamp value.
Every time the client wants to process new events on the package source, it need only query the catalog for all catalog items with a commit timestamp greater than its stored cursor. After the client successfully processes all new catalog items, it records the latest commit timestamp of catalog items just processed as its new cursor value.
Using this approach, the client can be sure to never miss any package events that occurred on the package source. Additionally, the client never has to reprocess old events prior to the cursor's recorded commit timestamp.
This powerful concept of cursors is used for many of nuget.org background jobs and is used to keep the V3 API itself up-to-date.
Initial value
When the catalog client is starting for the very first time (and therefore has no cursor value), it should use a default
cursor value of .NET's System.DateTimeOffset.MinValue or some such analogous notion of minimum representable
timestamp.
Iterating over catalog items
To query for the next set of catalog items to process, the client should:
- Fetch the recorded cursor value from a local store.
- Download and deserialize the catalog index.
- Find all catalog pages with a commit timestamp greater than the cursor.
- Declare an empty list of catalog items to process.
- For each catalog page matched in step 3:
- Download and deserialized the catalog page.
- Find all catalog items with a commit timestamp greater than the cursor.
- Add all matching catalog items to the list declared in step 4.
- Sort the catalog item list by commit timestamp.
- Process each catalog item in sequence:
- Download and deserialize the catalog item.
- React appropriately to the catalog item's type.
- Process the catalog item document in a client-specific fashion.
- Record the last catalog item's commit timestamp as the new cursor value.
With this basic algorithm, the client implementation can build up a complete view of all packages available on the package source. The client need only execute this algorithm periodically to always be aware of the latest changes to the package source.
Note
This is the algorithm that nuget.org uses to keep the Package Metadata, Package Content, Search and Autocomplete resources up to date.
Dependent cursors
Suppose there are two catalog clients that have an inherent dependency where one client's output depends on another client's output.
Example
For example, on nuget.org a newly published package should not appear in the search resource before it appears in the package metadata resource. This is because the "restore" operation performed by the official NuGet client uses the package metadata resource. If a customer discovers a package using the search service, they should be able to successfully restore that package using the package metadata resource. In other words, the search resource depends on the package metadata resource. Each resource has a catalog client background job updating that resource. Each client has its own cursor.
Since both resources are built off of the catalog, the cursor of the catalog client that updates the search resource must not go beyond the cursor of the package metadata catalog client.
Algorithm
To implement this restriction, simply modify the algorithm above to be:
- Fetch the recorded cursor value from a local store.
- Download and deserialize the catalog index.
- Find all catalog pages with a commit timestamp greater than the cursor less than or equal to the dependency's cursor.
- Declare an empty list of catalog items to process.
- For each catalog page matched in step 3:
- Download and deserialized the catalog page.
- Find all catalog items with a commit timestamp greater than the cursor less than or equal to the dependency's cursor.
- Add all matching catalog items to the list declared in step 4.
- Sort the catalog item list by commit timestamp.
- Process each catalog item in sequence:
- Download and deserialize the catalog item.
- React appropriately to the catalog item's type.
- Process the catalog item document in a client-specific fashion.
- Record the last catalog item's commit timestamp as the new cursor value.
Using this modified algorithm, you can build a system of dependent catalog clients all producing their own specific indexes, artifacts, etc.