HTTP file download missing columns

Question

I'm using an http linked service and basic authentication to download a file. It requires username and password. When I download the file from a browser with the URL and credentials the file has more columns compared to the file download via ADF pipeline. The URL is exactly the same. Why am I missing column in the ADF pipeline?

Answer 1

@Winston TheFifth

Thanks For Reaching out MS Q&A

Check the mapping in the copy activity: One possible reason for the missing columns is that the mapping in the copy activity is not correct or updated. You can try to select all columns (*) in the source and then map those you want to the sink schema, or clear the mapping and let the ADF copy component auto map to columns in the sink schema.

Use pre-copy script to create columns if not exist: Another possible reason is that the sink table does not have the same schema as the source file. You can use a pre-copy script in the copy activity to create columns if they do not exist in the sink table.

Use dynamic mapping with metadata and lookup activities: A more advanced solution is to use dynamic mapping with metadata and lookup activities to compare the number of columns in the source file and the sink table, and then copy the data accordingly.

Hope this helps. Do let us know if you any further queries.