Skip to main content

Crawl an API

Currently, we offer factories for two types of APIs: REST and GraphQL. These factories allow you to ingest data from APIs with minimal effort. Here are the guides to help you get started:

  • REST API Crawler: This guide will walk you through the process of creating a REST API crawler using our rest factory. It will show you how to define the endpoints you want to fetch, create a configuration object, and build the asset.
  • GraphQL API Crawler: This guide will explain how to use the GraphQL Resource Factory to bootstrap a graphql_factory asset in Dagster that fetches all fields from a GraphQL endpoint automatically.

Reach out to us on Discord for help.