Table of Contents
Big Data Tools
Return to JetBrains Plugins, JetBrains
A bundle of plugins for data engineers and other specialists engaged with big data workloads. Installed in your favorite JetBrains IDE, Big Data Tools helps develop, visualize, debug, and monitor big data pipelines built in Scala, Python, and SQL. Subscribe to our News | Read our Blog | Join the Slack Community Use Big Data Tools for:
- Exploratory analysis, visualization, and prototyping jobs in Zeppelin notebooks.
- Running and monitoring Spark or Flink jobs directly from your IDE.
- Working with Amazon EMR clusters.
- Viewing big data files, such as CSV, Parquet, ORC, and Avro.
- Producing and consuming messages with Kafka.
- Previewing Hive Metastore databases.
- Getting insights about your Hadoop environment.
- Built-in tools and integrations:
- Supported languages: Scala, Python, SQL.
Notebooks: Zeppelin. Monitoring: Hadoop, Kafka, Spark, Hive Metastore, Flink, AWS Glue. Remote file storages: AWS S3, Google Cloud Storage, Microsoft Azure, Tencent Cloud Object Storage (COS), DigitalOcean Spaces, Alibaba OSS, Hadoop Distributed File System (HDFS), and more. File systems: HDFS, Local, SFTP. Data processing platforms: AWS EMR.
Getting started
- These instructions link to the Big Data Tools documentation for IntelliJ IDEA. If you use the plugin with a different JetBrains IDE, please use one of the links below instead: PyCharm | DataSpell | DataGrip To start using Big Data Tools in IntelliJ IDEA:
Install the Big Data Tools plugin. Install the required language plugins (Scala or Python). Create a new project in your IDE. Connect to a particular server, storage, or service. Voila! You’re ready to start working on your project.