In Apache Airflow, downloading files via HTTP is a core data engineering task. While the HttpOperator (formerly SimpleHttpOperator ) is perfect for API interactions, it is primarily designed to return the response body as a string. For larger files or persistent storage, you often need to combine it with other tools. 1. Using the Standard HttpOperator
: Large files (e.g., 100MB+) should not be stored in XCom as they can bloat your metadata database. airflow http operator download file
The HttpOperator is best for small payloads. It returns the response as text, which Airflow automatically pushes to —a temporary storage for passing small amounts of data between tasks. Best for : Small JSON files or text-based API responses. In Apache Airflow, downloading files via HTTP is
: Use the requests library inside a Python function. It returns the response as text, which Airflow
: Can be configured within the HTTP Connection settings in Airflow.