Azure Synapse development (code writing) - IDE or notebooks?

Michael Urinovsky 11 Reputation points Microsoft Employee
2022-03-31T13:28:46.87+00:00

My team is going to implement Data Engineering use cases on Azure Synapse using Spark. We are considering 2 types of code development: IDE based or Notebooks based. What would be considered as a preferred way of work for a Data Engineer? What are the pros and cons for each method from your perspective?

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,378 questions
{count} vote

1 answer

Sort by: Most helpful
  1. PRADEEPCHEEKATLA 90,651 Reputation points Moderator
    2022-04-04T05:33:40.537+00:00

    Hello @Michael Urinovsky ,

    Thanks for the question and using MS Q&A platform.

    As per my experience Azure Synapse Notebooks experience is better compared to the IDE. And this perspective can be varied from one user to other.

    Azure Synapse Notebook:

    A Synapse notebook is a web interface for you to create files that contain live code, visualizations, and narrative text. Notebooks are a good place to validate ideas and use quick experiments to get insights from your data. Notebooks are also widely used in data preparation, data visualization, machine learning, and other Big Data scenarios.

    With a Synapse notebook, you can:

    • Get started with zero setup effort.
    • Keep data secure with built-in enterprise security features.
    • Analyze data across raw formats (CSV, txt, JSON, etc.), processed file formats (parquet, Delta Lake, ORC, etc.), and SQL tabular data files against Spark and SQL.
    • Be productive with enhanced authoring capabilities and built-in data visualization.

    IDE - IntelliJ or VSCode:

    You can use the Azure Toolkit for IntelliJ plug-in to develop Apache Spark applications, which are written in Scala, and then submit them to a serverless Apache Spark pool directly from the IntelliJ integrated development environment (IDE). You can use the plug-in in a few ways:

    • Develop and submit a Scala Spark application on a Spark pool.
    • Access your Spark pools resources.
    • Develop and run a Scala Spark application locally.

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    1 person found this answer helpful.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.