Can't import ML tools in Azure notebook unless I select Python 3.8 kernel.

Question

I have been using learning studio and have a CPU compute that when started has several options for the kernel. I can only import keras or tensorflow if I select Python 3.8. This version of keras is 2.11.0. I need to use the trained models on a VM that has Python3.10 with keras 3.4.1.

If I select other kernels !pip list | grep keras => 3.4.1 but I can't import keras.

How I've been selecting the kernel is by using the drop down menu upper right stopping the compute instance and then restarting it. Is there a better way and why can't I import keras when I use other kernels and pip shows it as installed?

Answer 1

Hello Phelps, Ronald (Ron) (CIV),

Welcome to the Microsoft Q&A and thank you for posting your questions here.

Problem

I understand that you are having issue with importing Keras when using different kernels in your learning studio environment, unless you select Python 3.8 kernel it does not work though, you would like to use Python 3.10. You need also if there is a better way and why can't you import keras when you have use other kernels and pip shows it as installed.

Solution

Some users have reported similar issues after upgrading TensorFlow versions, and a workaround is to try importing it as from tensorflow.python.keras.layers import Dense.

About Python 3.10 Compatibility, you have mentioned needing to use Python 3.10 with Keras 3.4.1. If you're upgrading your code to work with Python 3.10, ensure that you also update any TensorFlow-related imports. For example, change import tensorflow.python.keras.backend as K` to import tensorflow.compat.v1.keras.backend as K.

Thirdly, restarting the compute instance by selecting a different kernel from the drop-down menu is a common way to switch kernels. However, there might be a more efficient approach like instead of restarting the entire instance, consider using Jupyter magic commands to switch kernels within a notebook cell. For example:

      source activate myenv  # Activate the desired environment
       jupyter kernelspec list  # List available kernels
    

Fourthly, ensure that Keras is installed in the Python environment associated with the kernel you're using. You can check this by running pip list | grep keras and if Keras is not installed, use python -m pip install keras to install it.

Last but not the least, consider creating a separate Python environment for each backend (e.g., TensorFlow, JAX) to avoid version conflicts. For instance, you can create a JAX GPU environment with Conda:

     conda create -y -n keras-jax python=3.10
     conda activate keras-jax
     pip install -r requirements-jax-cuda.txt
     pip install --upgrade keras

I put together a more comprehensive solution together in the code below like a pseudocode, read the comment to identify each of the stage to implement.

# Verify Kernel Configuration
   import sys
   print(sys.version)
   import keras
   print(keras.__version__)
   
# Reinstall Keras in the Active Kernel using bash: 
   !pip install --upgrade pip
   !pip uninstall keras -y
   !pip install keras==3.4.1
  
# Check for Conflicts using bash:
   !pip check
  
# Create and Use a Virtual Environment using bash
   !python3.10 -m venv myenv
   !source myenv/bin/activate
   !pip install keras==3.4.1
 
# Restart Kernel
   - Restart the kernel after making changes to the environment

# Check Python Pat using Python
   import sys
   print(sys.executable)
   
# Error Logs: Log errors to identify any further issues.
   - Check specific error messages for clues.
# Import Resolution Issue 
   - Try workarounds such as `from tensorflow.python.keras.layers import Dense`.
# Python 3.10 Compatibility
   - Ensure TensorFlow-related imports are updated.
# Kernel Selection Using Jupyter Commands using bash
 
    %%bash
    source activate myenv
    jupyter kernelspec list
    

References

Source: Python - Import "tensorflow.keras" could not be resolved. Accessed, 7/10/2024.

Source: Trying to upgrade code to work with Python 3.10 with Tensor flow. Accessed, 7/10/2024.

Source: Python 3.x - Error when importing 'keras' from 'tensorflow' Accessed, 7/10/2024.

Source: Getting started with Keras. Accessed, 7/10/2024.

Accept Answer

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

** Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful ** so that others in the community facing similar issues can easily find the solution.

Best Regards,

Sina Salam