Azure Container Instance failing to connect to Azure DB for MySQL Instance

Question

I have spun up an ACI based off a custom image hosting a strapi application, that uses the mysql js client to interface with the DB. When accessing the Azure DB for MySQL instance via a local MySQL client or a local docker container, the connection to the database is successful. When I attempt to start the Azure Container Instance of the same image, I receive an error related to the MySQL client and the container fails to properly start.

Error:

error Error: UNKNOWN_CODE_PLEASE_REPORT: An internal error has occurred. Please retry or report your issues. at Handshake.Sequence._packetToError (/usr/src/app/node_modules/mysql/lib/protocol/sequences/Sequence.js:47:14) at Handshake.ErrorPacket (/usr/src/app/node_modules/mysql/lib/protocol/sequences/Handshake.js:123:18) at Protocol._parsePacket (/usr/src/app/node_modules/mysql/lib/protocol/Protocol.js:291:23) at Parser._parsePacket (/usr/src/app/node_modules/mysql/lib/protocol/Parser.js:433:10) at Parser.write (/usr/src/app/node_modules/mysql/lib/protocol/Parser.js:43:10) at Protocol.write (/usr/src/app/node_modules/mysql/lib/protocol/Protocol.js:38:16) at TLSSocket.ondata (_stream_readable.js:717:22) at TLSSocket.emit (events.js:315:20) at addChunk (_stream_readable.js:295:12) at readableAddChunk (_stream_readable.js:271:9) at TLSSocket.Readable.push (_stream_readable.js:212:10) at TLSWrap.onStreamRead (internal/stream_base_commons.js:186:23)

The environment variables used to connect are:

DATABASE_HOST
money-heroes-preview.mysql.database.azure.com

DATABASE_PORT
3306

DATABASE_NAME
money-heroes

DATABASE_USER
mh_admin@money-heroes-preview

DATABASE_PASSWORD
*********************

DATABASE_SSL
true

This error can be observed even with the DB instance allowing access to Azure Services, with a blanket IP whitelist and SSL Enforcing turned off.

Answer 1

@NaomiBeckford-4720 This is very helpful. The message which we are most concerned about are

--> (65062) ClientAuthenticationPacket {
clientFlags: 457679,
maxPacketSize: 0,
charsetNumber: 33,
filler: undefined,
user: undefined,
scrambleBuff: <Buffer 54 dd d2 75 87 da 39 2a 3b 53 4b 0e 71 62 2a 19 05 7a cd 63>,
database: 'money-heroes',
protocol41: true
}
<-- (65062) ErrorPacket {
fieldCount: 255,
errno: 9999,
sqlStateMarker: '#',
sqlState: '28000',
message: 'An internal error has occurred. Please retry or report your issues.\u0000'
}

The error code is SQL State 28000 and on reading about it , this error is related to access denied for the database to the user. Please check database exist and user has access to that database (user being database user which you are connecting with).

Once the database is created and access has been granted, it should work now.

Please 'Accept as answer' if it helped, so that it can help others in the community looking for help on similar topics

Answer 2

@Naomi Beckford Thank you for your query!!!

I am suspecting the issue is with docker image. I would suggest you to try running image locally with connection pointing towards azure database which would help to pinpoint the exact problem.

If locally also you are facing the same error, it might be docker image is not reading environment variable and is always trying to connect to localhost for MySQL connection and hence working locally with local database and failing in ACI.

Another suspect is SSL connection. Since the client is trying to use SSL connection, it might be failing if it is not able to verify the certificate for it. I would advice to disable SSL connection in client also and if that works, read the documentation of enabling SSL connection and enable it.

Hope this helps.

Please 'Accept as answer' if it helped, so that it can help others in the community looking for help on similar topics

Answer 3

I have attempted all of the above, but with no reprieve.

• Switching SSL on/off doesn't allow a connection via the docker container.
• I checked ENV variables by attempting to ping a non-existent database, which gave me a more descriptive error.
• Running locally does not allow me to debug the issue either, the same error is still present when running the docker image locally via docker run.

The error seems to occur during the handshake with the database. Is there anyway I can see what's going on when it happens? To see if Azure DB for MySQL is hitting a problem?

Answer 4

@Naomi Beckford Since locally also the problem is reproducible and the error is in the connection to MySQL instance of azure, I would suggest few approaches to identify the source of the problem and solve it accordingly

1. Try using MySQL CLI client to connect to MySQL instance directly. Most likely it would work and MySQL client connection should work. If this is not working, something is wrong in the setting of MySQL server (maybe try provisioning a new MySQL server and incrementally add new settings such as database, users to it.
2. If MySQL client was able to connect, try using Node js mysql client to connect to database (do it locally without any docker in order to debug it easily). If possible first try with same package which is being used in the docker image and then use any other alternative package to connect. I am suspecting some problem in the package which is trying to connect if after disabling SSL connection also this does not work.

We are just trying to identify which step in the equation is causing issue so that issue can be fixed.

Hope this helps

Please 'Accept as answer' if it helped, so that it can help others in the community looking for help on similar topics

Answer 5

Hey, I'm doing some debugging with some lightweight client code to see whats going on and I've managed to get some more debug info that may interest you. Still trying to figure out what it means as of yet but I'll share with you guys now.

<-- HandshakeInitializationPacket {
  protocolVersion: 10,
  serverVersion: '5.6.42.0',
  threadId: 65062,
  scrambleBuff1: <Buffer fe fe fe fe fe fe fe fe>,
  filler1: <Buffer 00>,
  serverCapabilities1: 43791,
  serverLanguage: 33,
  serverStatus: 2,
  serverCapabilities2: 63,
  scrambleLength: 21,
  filler2: <Buffer 00 00 00 00 00 00 00 00 00 00>,
  scrambleBuff2: <Buffer fe fe fe fe fe fe fe fe fe fe fe fe>,
  filler3: <Buffer 00>,
  pluginData: 'mysql_native_password',
  protocol41: true
}
--> (65062) SSLRequestPacket {
  clientFlags: 457679,
  maxPacketSize: 0,
  charsetNumber: 33
}
--> (65062) ClientAuthenticationPacket {
  clientFlags: 457679,
  maxPacketSize: 0,
  charsetNumber: 33,
  filler: undefined,
  user: undefined,
  scrambleBuff: <Buffer 54 dd d2 75 87 da 39 2a 3b 53 4b 0e 71 62 2a 19 05 7a cd 63>,
  database: 'money-heroes',
  protocol41: true
}
<-- (65062) ErrorPacket {
  fieldCount: 255,
  errno: 9999,
  sqlStateMarker: '#',
  sqlState: '28000',
  message: 'An internal error has occurred. Please retry or report your issues.\u0000'
}
/Users/chrisevans/Documents/GitHub/mysql-client-test/index.js:22
  if (error) throw error;
             ^
Error: UNKNOWN_CODE_PLEASE_REPORT: An internal error has occurred. Please retry or report your issues.
    at Handshake.Sequence._packetToError (/Users/chrisevans/Documents/GitHub/mysql-client-test/node_modules/mysql/lib/protocol/sequences/Sequence.js:47:14)
    at Handshake.ErrorPacket (/Users/chrisevans/Documents/GitHub/mysql-client-test/node_modules/mysql/lib/protocol/sequences/Handshake.js:123:18)
    at Protocol._parsePacket (/Users/chrisevans/Documents/GitHub/mysql-client-test/node_modules/mysql/lib/protocol/Protocol.js:291:23)
    at Parser._parsePacket (/Users/chrisevans/Documents/GitHub/mysql-client-test/node_modules/mysql/lib/protocol/Parser.js:433:10)
    at Parser.write (/Users/chrisevans/Documents/GitHub/mysql-client-test/node_modules/mysql/lib/protocol/Parser.js:43:10)
    at Protocol.write (/Users/chrisevans/Documents/GitHub/mysql-client-test/node_modules/mysql/lib/protocol/Protocol.js:38:16)
    at TLSSocket.ondata (_stream_readable.js:717:22)
    at TLSSocket.emit (events.js:315:20)
    at addChunk (_stream_readable.js:295:12)
    at readableAddChunk (_stream_readable.js:271:9)
    --------------------
    at Protocol._enqueue (/Users/chrisevans/Documents/GitHub/mysql-client-test/node_modules/mysql/lib/protocol/Protocol.js:144:48)
    at Protocol.handshake (/Users/chrisevans/Documents/GitHub/mysql-client-test/node_modules/mysql/lib/protocol/Protocol.js:51:23)
    at Connection.connect (/Users/chrisevans/Documents/GitHub/mysql-client-test/node_modules/mysql/lib/Connection.js:116:18)
    at Object.<anonymous> (/Users/chrisevans/Documents/GitHub/mysql-client-test/index.js:19:12)
    at Module._compile (internal/modules/cjs/loader.js:1138:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1158:10)
    at Module.load (internal/modules/cjs/loader.js:986:32)
    at Function.Module._load (internal/modules/cjs/loader.js:879:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
    at internal/main/run_main_module.js:17:47 {
  code: 'UNKNOWN_CODE_PLEASE_REPORT',
  errno: 9999,
  sqlMessage: 'An internal error has occurred. Please retry or report your issues.\u0000',
  sqlState: '28000',
  fatal: true
}