Rest API Data stream to event hub

Dondapati, Navin 281 Reputation points
2020-12-30T16:21:21.083+00:00

Hi Guys,

We are using below rest api get method of accuweather to fetch location info

curl --location --request GET 'http://dataservice.accuweather.com/locations/v1/adminareas/countryCode?apikey=VU86G'

PostMan output:
http://dataservice.accuweather.com/locations/v1/adminareas/countryCode?apikey=VU86G

{
"ID": "AMA",
"LocalizedName": "Amazonas",
"EnglishName": "Amazonas",
"Level": 1,
"LocalizedType": "Department",
"EnglishType": "Department",
"CountryID": "countryCode"
},
{
"ID": "ANT",
"LocalizedName": "Antioquia",
"EnglishName": "Antioquia",
"Level": 1,
"LocalizedType": "Department",
"EnglishType": "Department",
"CountryID": "countryCode"
}

We want to use the url as producer in eventhub and pass through stream analytics and load to blob storage,

How do i make above data as producer to eventhub and make sure it load data every 4 hours?

Regards,
Navin

Azure Event Hubs
Azure Event Hubs
An Azure real-time data ingestion service.
644 questions
Azure Stream Analytics
Azure Stream Analytics
An Azure real-time analytics service designed for mission-critical workloads.
359 questions
0 comments No comments
{count} votes

Accepted answer
  1. MartinJaffer-MSFT 26,091 Reputation points
    2020-12-31T22:10:52.887+00:00

    Hello and welcome back to Microsoft Q&A @Anonymous .

    Data gets pushed to Event hubs. Event hubs does not pull the data for you.
    This means you will need to write a producer, and run the producer on a schedule.
    Alternatively you could write a long-running producer that waits between batches.

    There a number of languages available. For this example, I will build off the python quickstart.

    import asyncio  
    from azure.eventhub.aio import EventHubProducerClient  
    from azure.eventhub import EventData  
      
    from time import sleep  
    #import urllib  
      
    async def run():  
        # Create a producer client to send messages to the event hub.  
        # Specify a connection string to your event hubs namespace and  
        # the event hub name.  
        producer = EventHubProducerClient.from_connection_string(conn_str="EVENT HUBS NAMESPACE - CONNECTION STRING", eventhub_name="EVENT HUB NAME")  
        async with producer:  
            # Create a batch.  
            event_data_batch = await producer.create_batch()  
      
            #get the data  
            #use pycurl http://pycurl.io/  
            #or use liburl  
            # or use requests https://requests.readthedocs.io/en/master/  
            data = request_data(url)  
    
            # Add events to the batch.  
            for record in data:  
                event_data_batch.add(EventData(record))  
      
            # Send the batch of events to the event hub.  
            await producer.send_batch(event_data_batch)  
      
            # make the process wait for 4 hours between iterations  
            sleep(4*60*60)  
      
    loop = asyncio.get_event_loop()  
    loop.run_until_complete(run())  
    

    I would also like to note the data you shared above appears to be dimension data defining the regions of world, rather than event data describing the weather. This type of data changes very slowly / infrequently. It would be better used in a reference table in Stream Analytics.


1 additional answer

Sort by: Most helpful
  1. PRADEEPCHEEKATLA-MSFT 90,146 Reputation points Microsoft Employee
    2020-12-31T08:01:56.447+00:00

    Hello @Anonymous ,

    Unfortunately, REST API is not supported as the producer/consumer side in Azure Event Hubs.

    For more detail, you have refer the similar threads which address a similar issue:

    https://stackoverflow.com/questions/53433372/is-it-possible-to-receive-events-from-azure-event-hubs-using-rest-api

    https://stackoverflow.com/questions/32332714/how-to-send-and-consume-messages-in-event-hub-via-rest-api

    Hope this helps. Do let us know if you any further queries.

    ------------

    • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification.
    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.