How to grab only certain sections of data from a csv file

Question

How to grab only certain sections of data from a csv file

King Java 790

I am trying to grab certain rows (and columns) of data from csv file.

As you can see below, I am trying to grab only certain rows and columns of data from csv file (two spots).

What is best solution?

User's image

Should I use Python script to run or is there a better/quicker way in ADF?

Thanks.

Smaran Thoomu 24,110 Reputation points Microsoft External Staff Moderator

2023-09-18T12:02:18.0566667+00:00

@King Java Just checking in to see if the below answer provided by @Sina Salam helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Accepted answer

0 additional answers

Your answer

Smaran Thoomu 24,110 Reputation points Microsoft External Staff Moderator

2023-09-18T12:02:18.0566667+00:00

@King Java Just checking in to see if the below answer provided by @Sina Salam helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 1

Sina Salam 22,031 Volunteer Moderator

Hello @King Java

Welcome to Microsoft Q&A and thank you for posting your questions here.

Concerning your question, you were asking if you should use Python script to run it or use your Azure Data Factory (ADF) better or quicker way.

Absolutely, you can use Python. However, to grab specific sections of data from a CSV file using Azure Data Factory (ADF), because of Azure ecosystem Azure Data Factory (ADF) is a powerful tool to consider, very fast and easy. It allows you to create data pipelines and perform Extract, Transform, Load (ETL) operations on various data sources, including CSV files.

Therefore, if all your configurations are okay, while adding Data Flow Transformations or after you can do the followings:

Inside the Data Flow activity, add transformations to filter rows and select specific columns.
- You can use the Filter and Select transformations for this purpose for example, to filter rows where the 'Role' column is 'Manager' and select specific columns like 'Name' and 'Salary', you would configure these transformations to your specific needs accordingly.
You can define filter conditions using expressions in the Filter transformation.
Then, after applying the transformations, configure the Sink dataset. This dataset defines where the filtered and selected data will be written. You can specify the path, format, and destination for the output CSV file.

I hope this is helpful! PS: Do not hesitate to let me know if you have any other questions.

Please remember to "Accept Answer" if answer helped, so that others in the community facing similar issues can easily find the solution.

Best Regards,

Sina Salam

King Java 790 Reputation points

2023-09-08T23:05:59.3366667+00:00
@Sina Salam

Thank you for help.

Please see below:

Two yellowed boxed areas are where I need to grab data.

Basically, I need to have these logics:

Find row column where it has "Admissions"

Go down 3 rows

Select column B to column F until there is no value on column B

Find row column where it has "Readmissions"

Go down 3 rows

Select column B to column F until there is no value on column B

Is this possible in ADF?

Thanks.
Sina Salam 22,031 Reputation points Volunteer Moderator

2023-09-11T11:56:36.33+00:00

Hi @King Java

Thank you for asking.

Your first question was "Should I use Python script to run or is there a better/quicker way in ADF?"

I have made you to understand that it is possible to use Python and you can also, grab specific sections of data from a CSV file using Azure Data Factory (ADF) with steps.

Acknowledge that first then, moving forward If you will need to write logic, provide information about which option above is suitable or which language you would like to use.

If you could write your clean "pseudo code" and provide option you would like to use, then no impossibility.

Success,

Sina
Smaran Thoomu 24,110 Reputation points Microsoft External Staff Moderator

2023-09-12T13:14:56.16+00:00

Hi @King Java
Just checking in to see if the above answer provided by @Sina Salam helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
King Java 790 Reputation points

2023-09-12T17:26:42.4+00:00

@Smaran Thoomu Thank you for your help.

Can you give me some example of how to express the logic in ADF?

Which object do I use?

Thanks.
Smaran Thoomu 24,110 Reputation points Microsoft External Staff Moderator

2023-09-14T14:25:39.1333333+00:00

Hi @King Java
I would suggest using the Filter transformation to remove unwanted rows and then apply the conditional split transformation to split the rows into different datasets. Ideally, perform this within a dataflow.
Kindly refer this link for your reference. https://www.youtube.com/watch?v=9bCyKUNfNl8&list=PLsJW07-_K61IFEbcrImrHeLaS6C_voe0E&index=26
Smaran Thoomu 24,110 Reputation points Microsoft External Staff Moderator

2023-09-17T17:46:50.1833333+00:00

@King Java Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.

Share via

How to grab only certain sections of data from a csv file

0 additional answers

Your answer