Is there way I can check where It's causing issue from?
Yes when the backup is running look at the wait_type from sys.dm_exec_requests
select wait_type, wait_resource,* from sys.dm_exec_requests where command like '%backup%'
In most cases slow backup is due to storage issue because backup reads from storage and writes to storage. I would check disk I/O for drives using dmv sys.dm_io_virtual_file_stats
. You can also fire disk related perfmon counters. Specially disk for Avg disk queue length
.
Paul Randal has script created to get I/O stalls on Disk please refer.
Moral of story is get you disks checked thoroughly.