.json data via http

Question

Hello,
In pyspark I know how to read a .json file and display it into a dataframe and then land the result into .parquet file.
now I am given a http address which is supposed to return .json data.
I am not sure how to call this http to then have the .json data into a dataframe.

Questions
1-
Do you know how to do this in databricks pyspark?
2-
Alternatively, how can I do this in ADF?

Thank you

Answer 1

Hello @arkiboys ,
Thanks for the question and using MS Q&A platform.
As we understand the ask here is know how we can convert a json response to dataframe in pyspark , please do let us know if its not accurate.

Since we are talking about dataframe you will have to the work in databricks or Synapse notebook . I have tested the below code with the weather API and works .

import numpy as np
import pandas as pd
url = "http://api.weatherstack.com/current?access_key=<<accessKey>>&query=Sammamish"
df = pd.read_json(url)

You can use ADF to call tyhe notebook , but the code will run on ADb or Synapse cluster

Please do let me if you have any queries.
Thanks
Himanshu

---

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators