All Tutorials

Bulk import data with the GraphQL Admin API

All Tutorials

Bulk import data with the GraphQL Admin API

Bulk import data with the GraphQL Admin API

Importing large volumes of data using traditional and synchronous APIs is slow, complex to run, and difficult to manage. Instead of manually running a GraphQL mutation multiple times and managing a client-side throttle, you can run a bulk mutation operation.

Using the GraphQL Admin API, you can bulk import large volumes of data asychronously. When the operation is complete, the results are delivered in a JSON Lines (JSONL) file that Shopify makes available at a URL.

This guide introduces the bulkOperationRunMutation and shows you how to use it to bulk import data into Shopify.

Requirements

Limitations

  • You can run only one bulk operation of each type (bulkOperationRunMutation or bulkOperationRunQuery) at a time per shop.

    This limit is in place because operations are asynchronous and long-running. To run a subsequent bulk mutation operation for a shop, you need to either cancel the running operation or wait for it to finish.

  • You can supply only one of the supported GraphQL API mutations to the bulkOperationRunMutation at a time:

  • The mutation that's passed into bulkOperationRunMutation is limited to one connection field, which is defined by the GraphQL Admin API schema.

How bulk importing data works

You initiate a bulk operation by supplying a mutation string in the bulkOperationRunMutation. Shopify then executes that mutation string asynchronously as a bulk operation.

Most GraphQL Admin API requests that you make are subject to rate limits, but the bulkOperationRunMutation request isn't. Because you're only making low-cost requests for creating operations, polling their status, or canceling them, bulk mutation operations are an efficient way to create data compared to standard GraphQL API requests.

The following diagram shows the steps involved in bulk importing data into Shopify:

Workflow for bulk importing data

  1. Create a JSONL file and include GraphQL variables: Include the variables for the mutation in a JSONL file format. Each line in the JSONL file represents one input unit. The mutation runs once on each line of the input file.

  2. Upload the file to Shopify: Before you upload the file, you must reserve a link by running the stagedUploadsCreate mutation. After the space has been reserved, you can upload the file by making a request using the information returned from the stagedUploadsCreate response.

  3. Create a bulk mutation operation: After the file has been uploaded, you can run bulkOperationRunMutation to create a bulk mutation operation. The bulkOperationRunMutation imports data in bulk by running the supplied GraphQL API mutation with the file of variables uploaded in the last step.

  4. Poll the status of the operation: While the operation is running, you need to poll to see its progress using the currentBulkOperation field. The objectCount field on the bulkOperation object increments to indicate the operation's progress, and the status field returns a boolean value that states whether the operation is completed.

  5. Retrieve the results: When a bulk mutation operation is completed, a JSONL output file is available for download at the URL specified in the url field.

Create a JSONL file and include GraphQL variables

When adding the required GraphQL variables to a new JSONL file, you need to format the variables so that they'll be accepted by the corresponding bulk operation GraphQL API.

For example, you might want to import a large quantity of products. Each attribute of a product must be mapped to existing fields defined in the GraphQL input object ProductInput. In the JSONL file, each line represents one product input. The GraphQL Admin API runs once on each line of the input file. One input should take up one line only, no matter how complex the input object structure is.

The following example shows a sample JSONL file that is used to create 10 products in bulk:

{ "input": { "title": "Sweet new snowboard 1", "productType": "Snowboard", "vendor": "JadedPixel" } }
{ "input": { "title": "Sweet new snowboard 2", "productType": "Snowboard", "vendor": "JadedPixel" } }
{ "input": { "title": "Sweet new snowboard 3", "productType": "Snowboard", "vendor": "JadedPixel" } }
{ "input": { "title": "Sweet new snowboard 4", "productType": "Snowboard", "vendor": "JadedPixel" } }
{ "input": { "title": "Sweet new snowboard 5", "productType": "Snowboard", "vendor": "JadedPixel" } }
{ "input": { "title": "Sweet new snowboard 6", "productType": "Snowboard", "vendor": "JadedPixel" } }
{ "input": { "title": "Sweet new snowboard 7", "productType": "Snowboard", "vendor": "JadedPixel" } }
{ "input": { "title": "Sweet new snowboard 8", "productType": "Snowboard", "vendor": "JadedPixel" } }
{ "input": { "title": "Sweet new snowboard 9", "productType": "Snowboard", "vendor": "JadedPixel" } }
{ "input": { "title": "Sweet new snowboard 10", "productType": "Snowboard", "vendor": "JadedPixel" } }

Upload the file to Shopify

After you've created the JSONL file, and included the GraphQL variables, you can upload the file to Shopify. Before uploading the file, you need to first generate the upload URL and parameters.

Generate the uploaded URL and parameters

You can use the stagedUploadsCreate mutation to generate the values that you need to authenticate the upload. The mutation returns an array of stagedMediaUploadTarget instances.

An instance of stagedMediaUploadTarget has the following key properties:

  • parameters: The parameters that you use to authenticate an upload request.
  • url: The signed URL where you can upload the JSONL file that includes GraphQL variables.

The mutation accepts an input of type stagedUploadInput, which has the following fields:

Field Type Description
resource enum Specifies the resource type to upload. To use bulkOperationRunMutation, the resource type must be BULK_MUTATION_VARIABLES.
filename string The name of the file to upload.
mimeType string The media type of the file to upload. To use bulkOperationRunMutation, the mimeType must be "text/jsonl".
httpMethod enum The HTTP method to be used by the staged upload. To use bulkOperationRunMutation, the httpMethod must be POST.

Example

The following example uses the stagedUploadsCreate mutation to generate the values required to upload a JSONL file and be consumed by the bulkOperationRunMutation. You must first run the stagedUploadsCreate mutation with no variables, and then separately send a POST request to the staged upload URL with the JSONL data:

Request

POST /admin/api/2021-07/graphql.json

mutation {
  stagedUploadsCreate(input:{
    resource: BULK_MUTATION_VARIABLES,
    filename: "bulk_op_vars",
    mimeType: "text/jsonl",
    httpMethod: POST
  }){
    userErrors{
      field,
      message
    },
    stagedTargets{
      url,
      resourceUrl,
      parameters {
        name,
        value
      }
    }
  }
}

View response

JSON response:

{
  "data": {
    "stagedUploadsCreate": {
      "userErrors": [],
      "stagedTargets": [
        {
          "url": "https://shopify.s3.amazonaws.com",
          "resourceUrl": null,
          "parameters": [
            {
              "name": "key",
              "value": "tmp/21759409/bulk/89e620e1-0252-43b0-8f3b-3b7075ba4a23/bulk_op_vars"
            },
            {
              "name": "Content-Type",
              "value": "text/jsonl"
            },
            {
              "name": "success_action_status",
              "value": "201"
            },
            {
              "name": "acl",
              "value": "private"
            },
            {
              "name": "policy",
              "value": "eyJleHBpcmF0aW9uIjoiMjAyMS0wMS0yOFQyMDowNTo0NloiLCJjb25kaXRpb25zIjpbeyJidWNrZXQiOiJzaG9waWZ5In0sWyJjb250ZW50LWxlbmd0aC1yYW5nZSIsMSwyMDk3MTUyMF0seyJrZXkiOiJ0bXAvMjE3NTk0MDkvYnVsay84OWU2MjBlMS0wMjUyLTQzYjAtOGYzYi0zYjcwNzViYTRhMjMvYnVsa19vcF92YXJzIn0seyJDb250ZW50LVR5cGUiOiJ0ZXh0L2pzb25sIn0seyJzdWNjZXNzX2FjdGlvbl9zdGF0dXMiOiIyMDEifSx7ImFjbCI6InByaXZhdGUifSx7IngtYW16LWNyZWRlbnRpYWwiOiJBS0lBSllNNTU1S1ZZRVdHSkRLUS8yMDIxMDEyOC91cy1lYXN0LTEvczMvYXdzNF9yZXF1ZXN0In0seyJ4LWFtei1hbGdvcml0aG0iOiJBV1M0LUhNQUMtU0hBMjU2In0seyJ4LWFtei1kYXRlIjoiMjAyMTAxMjhUMTkwNTQ2WiJ9XX0="
            },
            {
              "name": "x-amz-credential",
              "value": "AKIAJYM555KVYEWGJDKQ/20210128/us-east-1/s3/aws4_request"
            },
            {
              "name": "x-amz-algorithm",
              "value": "AWS4-HMAC-SHA256"
            },
            {
              "name": "x-amz-date",
              "value": "20210128T190546Z"
            },
            {
              "name": "x-amz-signature",
              "value": "5d063aac44a108f2e38b8294ca0e82858e6f44baf835eb81c17d37b9338b5153"
            }
          ]
        }
      ]
    }
  },
  "extensions": {
    "cost": {
      "requestedQueryCost": 11,
      "actualQueryCost": 11
    }
  }
}

Upload the JSONL file

After you generate the parameters and URL for an upload, you can upload the JSONL file using a POST request. You must use a multipart form, and include all parameters as form inputs in the request body.

To generate the parameters for the multipart form, start with the parameters returned from the stagedUploadsCreate mutation. Then, add the file attachment.

POST request

curl --location --request POST 'https://shopify.s3.amazonaws.com' \
--form 'key="tmp/21759409/bulk/89e620e1-0252-43b0-8f3b-3b7075ba4a23/bulk_op_vars"' \
--form 'x-amz-credential="AKIAJYM555KVYEWGJDKQ/20210128/us-east-1/s3/aws4_request"' \
--form 'x-amz-algorithm="AWS4-HMAC-SHA256"' \
--form 'x-amz-date="20210128T190546Z"' \
--form 'x-amz-signature="5d063aac44a108f2e38b8294ca0e82858e6f44baf835eb81c17d37b9338b5153"' \
--form 'policy="eyJleHBpcmF0aW9uIjoiMjAyMS0wMS0yOFQyMDowNTo0NloiLCJjb25kaXRpb25zIjpbeyJidWNrZXQiOiJzaG9waWZ5In0sWyJjb250ZW50LWxlbmd0aC1yYW5nZSIsMSwyMDk3MTUyMF0seyJrZXkiOiJ0bXAvMjE3NTk0MDkvYnVsay84OWU2MjBlMS0wMjUyLTQzYjAtOGYzYi0zYjcwNzViYTRhMjMvYnVsa19vcF92YXJzIn0seyJDb250ZW50LVR5cGUiOiJ0ZXh0L2pzb25sIn0seyJzdWNjZXNzX2FjdGlvbl9zdGF0dXMiOiIyMDEifSx7ImFjbCI6InByaXZhdGUifSx7IngtYW16LWNyZWRlbnRpYWwiOiJBS0lBSllNNTU1S1ZZRVdHSkRLUS8yMDIxMDEyOC91cy1lYXN0LTEvczMvYXdzNF9yZXF1ZXN0In0seyJ4LWFtei1hbGdvcml0aG0iOiJBV1M0LUhNQUMtU0hBMjU2In0seyJ4LWFtei1kYXRlIjoiMjAyMTAxMjhUMTkwNTQ2WiJ9XX0="' \
--form 'acl="private"' \
--form 'Content-Type="text/jsonl"' \
--form 'success_action_status="201"' \
--form 'file=@"/Users/username/Documents/bulk_mutation_tests/products_long.jsonl"'

GraphQL variables in JSONL file

{ "input": { "title": "Sweet new snowboard 1", "productType": "Snowboard", "vendor": "JadedPixel" } }
{ "input": { "title": "Sweet new snowboard 2", "productType": "Snowboard", "vendor": "JadedPixel" } }
{ "input": { "title": "Sweet new snowboard 3", "productType": "Snowboard", "vendor": "JadedPixel" } }
{ "input": { "title": "Sweet new snowboard 4", "productType": "Snowboard", "vendor": "JadedPixel" } }
{ "input": { "title": "Sweet new snowboard 5", "productType": "Snowboard", "vendor": "JadedPixel" } }
{ "input": { "title": "Sweet new snowboard 6", "productType": "Snowboard", "vendor": "JadedPixel" } }
{ "input": { "title": "Sweet new snowboard 7", "productType": "Snowboard", "vendor": "JadedPixel" } }
{ "input": { "title": "Sweet new snowboard 8", "productType": "Snowboard", "vendor": "JadedPixel" } }
{ "input": { "title": "Sweet new snowboard 9", "productType": "Snowboard", "vendor": "JadedPixel" } }
{ "input": { "title": "Sweet new snowboard 10", "productType": "Snowboard", "vendor": "JadedPixel" } }

Create a bulk mutation operation

After you upload the file, you can run bulkOperationRunMutation to import data in bulk. You must supply the corresponding mutation and the URL that you obtained in the previous step.

The bulkOperationRunMutation mutation takes the following arguments:

Field Type Description
mutation string Specifies the GraphQL API mutation that you want to run in bulk. Valid values: productCreate, collectionCreate, productUpdate, productUpdateMedia, productVariantUpdate
stagedUploadPath string The path to the file of inputs in JSONL format to be consumed by stagedUploadsCreate

Example

In the following example, you want to run the following productCreate mutation in bulk:

mutation call($input: ProductInput!) {
  productCreate(input: $input) {
    product {
      id
      title
      variants(first: 10) {
        edges {
          node {
            id
            title
            inventoryQuantity
          }
        }
      }
    }
    userErrors {
      message
      field
    }
  }
}

To run the productCreate mutation in bulk, pass the mutation as a string into bulkOperationRunMutation:

Request

POST /admin/api/2021-07/graphql.json

mutation {
  bulkOperationRunMutation(
    mutation: "mutation call($input: ProductInput!) { productCreate(input: $input) { product {id title variants(first: 10) {edges {node {id title inventoryQuantity }}}} userErrors { message field } } }",
    stagedUploadPath: "tmp/21759409/bulk/89e620e1-0252-43b0-8f3b-3b7075ba4a23/bulk_op_vars") {
    bulkOperation {
      id
      url
      status
    }
    userErrors {
      message
      field
    }
  }
}

View response

JSON response:

{
  "data": {
    "bulkOperationRunMutation": {
      "bulkOperation": {
        "id": "gid://shopify/BulkOperation/206005076024",
        "url": null,
        "status": "CREATED"
      },
      "userErrors": []
    }
  },
  "extensions": {
    "cost": {
      "requestedQueryCost": 10,
      "actualQueryCost": 10
    }
  }
}

Poll the status of the operation

Because bulk operations are asynchronous and can be long running for large input files, you need to poll to find out when one is completed. The easiest way to poll is to query the currentBulkOperation field.

You can adjust your polling intervals based on the amount of data that you import. To learn about other possible operation statuses, refer to the BulkOperationStatus reference documentation.

To poll the status of the operation, use the following example request:

Request

POST /admin/api/2021-07/graphql.json

query {
 currentBulkOperation(type: MUTATION) {
    id
    status
    errorCode
    createdAt
    completedAt
    objectCount
    fileSize
    url
    partialDataUrl
 }
}

View response

JSON response:

{
  "data": {
    "currentBulkOperation": {
      "id": "gid://shopify/BulkOperation/206005076024",
      "status": "COMPLETED",
      "errorCode": null,
      "createdAt": "2021-01-28T19:10:59Z",
      "completedAt": "2021-01-28T19:11:09Z",
      "objectCount": "16",
      "fileSize": "6155",
      "url": "https://storage.googleapis.com/shopify-tiers-assets-prod-us-east1/iqtpj52yuoa7prkbpzp9gwn27kw3?GoogleAccessId=assets-us-prod%40shopify-tiers.iam.gserviceaccount.com&Expires=1612465869&Signature=KOhlcYhLve3NLr6rfVbAeY02crFAM3rMrDNfTSlgT%2FScI%2B8o%2B%2FdO99F3UseC837uWA6FzfrNhxdRNqhBN%2F8ekBTW7IyPRD6ho5phfE8MTaev4ltQrJygJTDbjXfX5KLJOuY8siH%2FDrc4gctZsMsNaf2%2FYp%2FaDzBzjfxJge8i8he69t0uZ39FBXrMxCeMVd6lU8%2FbgMuO80rTHjgI%2BlC8g2%2FWiHyq5rSTDLIxxGWRCddMfPcaivdWVdMubMa0wOt9W9R2mfjuTAgUBexUkJwhvrkdof%2Bg00gU1g4dIBWlUSO5D9tdrv9bmIy7FceopNufrpwnD1NXU8Narsx2yEQ6aA%3D%3D&response-content-disposition=attachment%3B+filename%3D%22bulk-206005076024.jsonl%22%3B+filename%2A%3DUTF-8%27%27bulk-206005076024.jsonl&response-content-type=application%2Fjsonl",
      "partialDataUrl": null
    }
  },
  "extensions": {
    "cost": {
      "requestedQueryCost": 1,
      "actualQueryCost": 1
    }
  }
}

Retrieve the results

When a bulk mutation operation is finished, you can download a result data file.

If an operation successfully completes, then the url field contains a URL where you can download the data file. If an operation fails, but some data was retrieved before the failure occurred, then a partially complete data file is available at the URL specified in the partialDataUrl field.

In either case, the returned URLs are authenticated and expire after one week.

After you've downloaded the data, you can parse it according to the JSONL format. Since both input and response files are in JSONL, each line in the final asset file represents the response of running the mutation on the corresponding line in the input file.

Operation success

The following example shows the response for a product that was successfully created:

{"data":{"productCreate":{"product":{"id":"gid:\/\/shopify\/Product\/5602345320504","title":"Monday morning snowboard 1","variants":{"edges":[{"node":{"id":"gid:\/\/shopify\/ProductVariant\/35645836853304","title":"First","inventoryQuantity":0}},{"node":{"id":"gid:\/\/shopify\/ProductVariant\/35645836886072","title":"Second","inventoryQuantity":0}}]}},"userErrors":[]}},"__lineNumber":0}

Operation failures

Bulk operations can fail for any of the reasons that a regular GraphQL API mutation would fail, such as not having permission to access certain APIs. For this reason, the best approach is to run a single GraphQL mutation first to make sure that it works before running a mutation as part of a bulk operation.

If a bulk operation does fail, then its status field returns FAILED and the errorCode field returns a code such as one of the following:

  • ACCESS_DENIED: There are missing access scopes. Run the mutation normally (outside of a bulk operation) to get more details on which field is causing the issue.
  • INTERNAL_SERVER_ERROR: Something went wrong on Shopify's server and we've been notified of the error. These errors might be intermittent, so you can try making your request again.
  • TIMEOUT: One or more mutation timeouts occurred during execution. Try removing some fields from your query so that it can run successfully. These timeouts might be intermittent, so you can try submitting the query again.

To learn about the other possible operation error codes, refer to the BulkOperationErrorCode reference documentation.

Validation error

If the input has the correct format, but one or more values failed the validation of the product creation service, then the response looks like the following:

{"data"=>{"productCreate"=>{"userErrors"=>[{"message"=>"Some error message", "field"=>["some field"]}]}}}

Unrecognizable field error

If the input has an unrecognizable field, then the response looks like the following:

{"errors"=>[{"message"=>"Variable input of type ProductInput! was provided invalid value for myfavoriteaddress (Field is not defined on ProductInput)", "locations"=>[{"line"=>1, "column"=>13}], "extensions"=>{"value"=>{"myfavoriteaddress"=>"test1"}, "problems"=>[{"path"=>["myfavoriteaddress"], "explanation"=>"Field is not defined on ProductInput"}]}}]}

Cancel an operation

To cancel an in-progress bulk operation, run the bulkOperationCancel mutation and supply the operation ID as an input variable:

Request

POST /admin/api/2021-07/graphql.json

mutation {
  bulkOperationCancel(id: "gid://shopify/BulkOperation/1") {
    bulkOperation {
      status
    }
    userErrors {
      field
      message
    }
  }
}

Next steps