Hi @Saiyad Rahim ,
Is it true that SCOM can not do Memory Used in MB or GB?
SCOM monitors by default the memory in available Megabytes (MB).
Also, what Performance counter do I use if I want to monitor if a server is using more than 80% memory?
You would need to use the Memory\Available MBytes performance counter, you can use the management pack described in Jonathan's blog post over here to monitor percentage of memory: http://blog.scomskills.com/monitor-percent-memory-used-on-windows/
(If the reply was helpful please don't forget to upvote or accept as answer, thank you)