Is there way I can check where It's causing issue from?
Yes when the backup is running look at the wait_type from sys.dm_exec_requests
select wait_type, wait_resource,* from sys.dm_exec_requests where command like '%backup%'
In most cases slow backup is due to storage issue because backup reads from storage and writes to storage. I would check disk I/O for drives using dmv
sys.dm_io_virtual_file_stats. You can also fire disk related perfmon counters. Specially disk for
Avg disk queue length.
Paul Randal has script created to get I/O stalls on Disk please refer.
Moral of story is get you disks checked thoroughly.