Add the following snippet to your pom.xml file. Ensure the version number matches your Spark installation (e.g., 3.5.0 ).
When submitting a job to a cluster, you don't always need to bundle the JAR in your fat-jar. You can use the --packages flag to download it at runtime: download spark-sql-kafka-0-10_2.12
This artifact is the official connector that allows (using Scala 2.12) to read from and write to Apache Kafka using the Structured Streaming API. Core Integration Methods Add the following snippet to your pom
Guide to Downloading and Integrating spark-sql-kafka-0-10_2.12 download spark-sql-kafka-0-10_2.12
spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.5.0 your-app.jar Use code with caution. Compatibility Matrix