Synapse - ADLS - Spark Notebook - Invalid JobSessionToken : IDX10249: X509SecurityKey validation failed. The associated certificate has expired.

Andrii Shymko 0 Reputation points
2023-08-17T13:27:03.95+00:00

Hello!

We use PySpark code in Synapse Notebook to read files from ADLS Gen2. For that we use mssparkutils.credentials.getSecret() and mssparkutils.credentials.getConnectionStringOrCreds() commands (see below)

account_access_key = mssparkutils.credentials.getSecret(akvName, secretName, linkedService)
...
sas_token = mssparkutils.credentials.getConnectionStringOrCreds(linkedService)

The necessary access like Storage Blob Data Contributor are provided to the Synapse Service principle. The Pipeline with the PySpark code notebook worked fine for 2.5 months, but recently the notebook in the pipeline is failed with the following error:

Py4JJavaError: An error occurred while calling z:mssparkutils.credentials.getSecret. : com.microsoft.azure.synapse.tokenlibrary.TokenLibrary$NonRetryableStatusException$1: POST failed with 'Unauthorized' (401) and message: {"result":"UserError","errorId":"Unauthorized","errorMessage":"Invalid JobSessionToken : IDX10249: X509SecurityKey validation failed. The associated certificate has expired. ValidTo (UTC): '8/12/2023 11:52:43 PM', Current time (UTC): '8/17/2023 12:54:55 PM'.. TraceId : 92f6a324-04ff-45c1-aa5e-d03c57f78179 | client-request-id : 55e09202-c679-4d0a-ac11-54c907c5b9ba. Error Component : TokenService"} 	at com.microsoft.azure.synapse.tokenlibrary.TokenLibrary.$anonfun$invokeTokenService$7(TokenLibrary.scala:470) 	at com.twitter.util.Future.$anonfun$flatMap$1(Future.scala:1808) 	at com.twitter.util.Promise$FutureTransformer.liftedTree1$1(Promise.scala:240) 	at com.twitter.util.Promise$FutureTransformer.k(Promise.scala:240) 	at com.twitter.util.Promise$Transformer.apply(Promise.scala:215) 	at com.twitter.util.Promise$WaitQueue.com$twitter$util$Promise$WaitQueue$$run(Promise.scala:91) 	at com.twitter.util.Promise$WaitQueue$$anon$1.run(Promise.scala:86) 	at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:198) 	at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:157) 	at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:274) 	at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:109) 	at com.twitter.util.Promise$WaitQueue.runInScheduler(Promise.scala:86) 	at com.twitter.util.Promise.updateIfEmpty(Promise.scala:778) 	at com.twitter.util.Promise.update(Promise.scala:750) 	at com.twitter.util.Promise.setValue(Promise.scala:726) 	at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:123) 	at com.twitter.finagle.netty4.transport.ChannelTransport$$anon$2.channelRead(ChannelTransport.scala:168) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) 	at shadenetty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) 	at com.twitter.finagle.netty4.http.handler.UnpoolHttpHandler$.channelRead(UnpoolHttpHandler.scala:32) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) 	at shadenetty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) 	at com.twitter.finagle.netty4.http.handler.ClientExceptionMapper$.channelRead(ClientExceptionMapper.scala:35) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) 	at shadenetty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) 	at com.twitter.finagle.netty4.http.handler.HeaderValidatorHandler$.channelRead(HeaderValidatorHandler.scala:51) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) 	at shadenetty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) 	at shadenetty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) 	at shadenetty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) 	at shadenetty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) 	at shadenetty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) 	at shadenetty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) 	at shadenetty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) 	at shadenetty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) 	at shadenetty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) 	at shadenetty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) 	at shadenetty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) 	at shadenetty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) 	at shadenetty.handler.ssl.SslHandler.unwrap(SslHandler.java:1432) 	at shadenetty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199) 	at shadenetty.handler.ssl.SslHandler.decode(SslHandler.java:1243) 	at shadenetty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) 	at shadenetty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) 	at shadenetty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) 	at shadenetty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) 	at shadenetty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) 	at shadenetty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) 	at shadenetty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) 	at shadenetty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:648) 	at shadenetty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:583) 	at shadenetty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:500) 	at shadenetty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) 	at shadenetty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 	at com.twitter.finagle.util.BlockingTimeTrackingThreadFactory$$anon$1.run(BlockingTimeTrackingThreadFactory.scala:23) 	at shadenetty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) 	at java.lang.Thread.run(Thread.java:750)

The Linked Service has "Account key" authentication type. I checked Linked Service connection and it works properly.

I created a new Linked Service with another type of authentication ("System Assigned Managed Identity") and used instead of the existing one but the same issue occurred.

Please help me to understand a root cause of this issue.

Thank you in advance!

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,385 questions
{count} votes

1 answer

Sort by: Most helpful
  1. PRADEEPCHEEKATLA 90,661 Reputation points Moderator
    2023-09-13T10:22:59.17+00:00

    @Anonymous - I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

    Issue: Synapse - ADLS - Spark Notebook - Invalid JobSessionToken : IDX10249: X509SecurityKey validation failed. The associated certificate has expired.

    Error:

    Py4JJavaError: An error occurred while calling z:mssparkutils.credentials.getSecret. : com.microsoft.azure.synapse.tokenlibrary.TokenLibrary$NonRetryableStatusException$1: POST failed with 'Unauthorized' (401) and message: {"result":"UserError","errorId":"Unauthorized","errorMessage":"Invalid JobSessionToken : IDX10249: X509SecurityKey validation failed. The associated certificate has expired. ValidTo (UTC): '8/12/2023 11:52:43 PM', Current time (UTC): '8/17/2023 12:54:55 PM'.. TraceId : 92f6a324-04ff-45c1-aa5e-d03c57f78179 | client-request-id : 55e09202-c679-4d0a-ac11-54c907c5b9ba. Error Component : TokenService"} 	at com.microsoft.azure.synapse.tokenlibrary.TokenLibrary.$anonfun$invokeTokenService$7(TokenLibrary.scala:470) 	at com.twitter.util.Future.$anonfun$flatMap$1(Future.scala:1808) 	at com.twitter.util.Promise$FutureTransformer.liftedTree1$1(Promise.scala:240) 	at com.twitter.util.Promise$FutureTransformer.k(Promise.scala:240) 	at com.twitter.util.Promise$Transformer.apply(Promise.scala:215) 	at com.twitter.util.Promise$WaitQueue.com$twitter$util$Promise$WaitQueue$$run(Promise.scala:91) 	at com.twitter.util.Promise$WaitQueue$$anon$1.run(Promise.scala:86) 	at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:198) 	at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:157) 	at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:274) 	at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:109) 	at com.twitter.util.Promise$WaitQueue.runInScheduler(Promise.scala:86) 	at com.twitter.util.Promise.updateIfEmpty(Promise.scala:778) 	at com.twitter.util.Promise.update(Promise.scala:750) 	at com.twitter.util.Promise.setValue(Promise.scala:726) 	at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:123) 	at com.twitter.finagle.netty4.transport.ChannelTransport$$anon$2.channelRead(ChannelTransport.scala:168) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) 	at shadenetty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) 	at com.twitter.finagle.netty4.http.handler.UnpoolHttpHandler$.channelRead(UnpoolHttpHandler.scala:32) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) 	at shadenetty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) 	at com.twitter.finagle.netty4.http.handler.ClientExceptionMapper$.channelRead(ClientExceptionMapper.scala:35) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) 	at shadenetty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) 	at com.twitter.finagle.netty4.http.handler.HeaderValidatorHandler$.channelRead(HeaderValidatorHandler.scala:51) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) 	at shadenetty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) 	at shadenetty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) 	at shadenetty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) 	at shadenetty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) 	at shadenetty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) 	at shadenetty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) 	at shadenetty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) 	at shadenetty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) 	at shadenetty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) 	at shadenetty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) 	at shadenetty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) 	at shadenetty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) 	at shadenetty.handler.ssl.SslHandler.unwrap(SslHandler.java:1432) 	at shadenetty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199) 	at shadenetty.handler.ssl.SslHandler.decode(SslHandler.java:1243) 	at shadenetty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) 	at shadenetty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) 	at shadenetty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) 	at shadenetty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) 	at shadenetty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) 	at shadenetty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) 	at shadenetty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) 	at shadenetty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) 	at shadenetty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:648) 	at shadenetty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:583) 	at shadenetty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:500) 	at shadenetty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) 	at shadenetty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 	at com.twitter.finagle.util.BlockingTimeTrackingThreadFactory$$anon$1.run(BlockingTimeTrackingThreadFactory.scala:23) 	at shadenetty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) 	at java.lang.Thread.run(Thread.java:750)
    
    
    

    Solution: The issue was auto-resolved. I did nothing with Azure App registration (we don't have any Azure App registrations) or any other steps to fix this problem.

    If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

    I hope this helps!

    If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.


    Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.