VMSS instance failes to connect to DB on first startup

Жаренков Данила 1 Reputation point
2019-12-09T11:48:57.567+00:00

Hello,

I've got an image for scale set, it's Win Server with defined system service, which starts Spring Boot application. This application uses Hibernate to connect to SQL Server DB (it's in the same resource group).

Issue i got - when VMSS scale out and creates new instance - spring boot service fails on startup. Issue with secure connection to database

WARN: HHH000342: Could not obtain connection to query metadata : The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "Unexpected rethrowing".  
Dec 09, 2019 9:51:23 AM org.hibernate.dialect.Dialect   
INFO: HHH000400: Using dialect: org.hibernate.dialect.SQLServerDialect  
Dec 09, 2019 9:51:24 AM org.hibernate.engine.jdbc.spi.SqlExceptionHelper logExceptions  
WARN: SQL Error: 1038, SQLState: S0004  
Dec 09, 2019 9:51:24 AM org.hibernate.engine.jdbc.spi.SqlExceptionHelper logExceptions  
ERROR: An object or column name is missing or empty. For SELECT INTO statements, verify each column has a name. For other statements, look for empty alias names. Aliases defined as "" or [] are not allowed. Change the alias to a valid name.  
[main] WARN org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in class path resource [com/kmslh/spot/db/config/SpotDBConfig.class]: Invocation of init method failed; nested exception is javax.persistence.PersistenceException: [PersistenceUnit: default] Unable to build Hibernate SessionFactory; nested exception is org.hibernate.exception.SQLGrammarException: Error accessing table metadata  
Dec 09, 2019 9:51:24 AM org.apache.catalina.core.StandardService stopInternal  
INFO: Stopping service [Tomcat]  
[main] INFO org.springframework.boot.autoconfigure.logging.ConditionEvaluationReportLoggingListener -   
  
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.  
[main] ERROR org.springframework.boot.SpringApplication - Application run failed  
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in class path resource [com/kmslh/spot/db/config/SpotDBConfig.class]: Invocation of init method failed; nested exception is javax.persistence.PersistenceException: [PersistenceUnit: default] Unable to build Hibernate SessionFactory; nested exception is org.hibernate.exception.SQLGrammarException: Error accessing table metadata  
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1778)  
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)  
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)  
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)  
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)  
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)  
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)  
	at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1105)  
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)  
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)  
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140)  
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:742)  
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:389)  
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:311)  
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1213)  
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1202)  
	at com.kmslh.spot.htmlconverter.HtmlConverter.main(HtmlConverter.java:11)  
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)  
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:47)  
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:86)  
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)  
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)  
Caused by: javax.persistence.PersistenceException: [PersistenceUnit: default] Unable to build Hibernate SessionFactory; nested exception is org.hibernate.exception.SQLGrammarException: Error accessing table metadata  
	at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.buildNativeEntityManagerFactory(AbstractEntityManagerFactoryBean.java:402)  
	at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.afterPropertiesSet(AbstractEntityManagerFactoryBean.java:377)  
	at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.afterPropertiesSet(LocalContainerEntityManagerFactoryBean.java:341)  
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1837)  
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1774)  
	... 24 more  
Caused by: org.hibernate.exception.SQLGrammarException: Error accessing table metadata  
	at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:106)  
	at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)  
	at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)  
	at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)  
	at org.hibernate.tool.schema.extract.internal.InformationExtractorJdbcDatabaseMetaDataImpl.convertSQLException(InformationExtractorJdbcDatabaseMetaDataImpl.java:98)  
	at org.hibernate.tool.schema.extract.internal.InformationExtractorJdbcDatabaseMetaDataImpl.getTables(InformationExtractorJdbcDatabaseMetaDataImpl.java:341)  
	at org.hibernate.tool.schema.extract.internal.DatabaseInformationImpl.getTablesInformation(DatabaseInformationImpl.java:120)  
	at org.hibernate.tool.schema.internal.GroupedSchemaMigratorImpl.performTablesMigration(GroupedSchemaMigratorImpl.java:65)  
	at org.hibernate.tool.schema.internal.AbstractSchemaMigrator.performMigration(AbstractSchemaMigrator.java:207)  
	at org.hibernate.tool.schema.internal.AbstractSchemaMigrator.doMigration(AbstractSchemaMigrator.java:114)  
	at org.hibernate.tool.schema.spi.SchemaManagementToolCoordinator.performDatabaseAction(SchemaManagementToolCoordinator.java:184)  
	at org.hibernate.tool.schema.spi.SchemaManagementToolCoordinator.process(SchemaManagementToolCoordinator.java:73)  
	at org.hibernate.internal.SessionFactoryImpl.(SessionFactoryImpl.java:309)  
	at org.hibernate.boot.internal.SessionFactoryBuilderImpl.build(SessionFactoryBuilderImpl.java:462)  
	at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:935)  
	at org.hibernate.jpa.HibernatePersistenceProvider.createContainerEntityManagerFactory(HibernatePersistenceProvider.java:141)  
	at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.createNativeEntityManagerFactory(LocalContainerEntityManagerFactoryBean.java:365)  
	at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.buildNativeEntityManagerFactory(AbstractEntityManagerFactoryBean.java:390)  
	... 28 more  
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: An object or column name is missing or empty. For SELECT INTO statements, verify each column has a name. For other statements, look for empty alias names. Aliases defined as "" or [] are not allowed. Change the alias to a valid name.  
	at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:259)  
	at com.microsoft.sqlserver.jdbc.TDSTokenHandler.onEOF(tdsparser.java:256)  
	at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:108)  
	at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:28)  
	at com.microsoft.sqlserver.jdbc.SQLServerConnection$1ConnectionCommand.doExecute(SQLServerConnection.java:2754)  
	at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7344)  
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2713)  
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectionCommand(SQLServerConnection.java:2759)  
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.setCatalog(SQLServerConnection.java:3076)  
	at com.microsoft.sqlserver.jdbc.SQLServerDatabaseMetaData.switchCatalogs(SQLServerDatabaseMetaData.java:345)  
	at com.microsoft.sqlserver.jdbc.SQLServerDatabaseMetaData.getResultSetFromStoredProc(SQLServerDatabaseMetaData.java:297)  
	at com.microsoft.sqlserver.jdbc.SQLServerDatabaseMetaData.getResultSetWithProvidedColumnNames(SQLServerDatabaseMetaData.java:321)  
	at com.microsoft.sqlserver.jdbc.SQLServerDatabaseMetaData.getTables(SQLServerDatabaseMetaData.java:493)  
	at org.hibernate.tool.schema.extract.internal.InformationExtractorJdbcDatabaseMetaDataImpl.getTables(InformationExtractorJdbcDatabaseMetaDataImpl.java:329)  
	... 40 more  

But if i'm just restarting my service- it works fine.
The vnet of VMSS is added to DB firewall rules.
More over - i added option to system service Recovery to restart service on failure. And again, the second run is successful.

Could it be an issue, that my VM in scale set starts a little bit earlier than it's IP have been added to vnet?

Azure Virtual Machines
Azure Virtual Machines
An Azure service that is used to provision Windows and Linux virtual machines.
7,130 questions
Azure Virtual Machine Scale Sets
Azure Virtual Machine Scale Sets
Azure compute resources that are used to create and manage groups of heterogeneous load-balanced virtual machines.
347 questions
{count} votes