Hello,
First of all please review the Runbook server installation log: %LocalAppData%\Microsoft System Center 2012\Orchestrator\LOGS\SetupWizard.log.
Are you installing Orchestrator and SQL on the same server?
Has the account running Orchestrator Setup sysadmin-rights to the SQL Database?
Is the collation SQL_Latin1_General_CP1_CI_AS?