A Minimal Python Environment Setting for Scientific Machine Learning on macOS/Linux/Windows (as of September 8, 2023)

I prefer to create and maintain the lightest, most portable Python environment for Scientific Machine Learning. Although the Jupyter Lab Desktop App is available, it is premature yet. One thing I want from the Jupyter Lab Desktop App (JLab App) is to use Conda to install and update Python libraries. Currently, JLab only supports pip installation. (A new version of the JLab App was released in February 2023, and I have not tested it yet.) So, I still use Miniconda to set up Python environments on macOS and Linux. Miniconda is a slim Python distribution containing the minimum amount of packages necessary for a Python installation that can use Conda.

(Remark: You may follow the same procedure on the Windows system. I recommend using Anaconda Powershell Prompt after the Miniconda install.)

1. Install Miniconda

Download the latest Miniconda installer. In your terminal window, run:

bash Miniconda3-latest-MacOSX-x86_64.sh

on macOS, or

bash Miniconda3-latest-Linux-x86_64.sh

on Linux.

For Windows, just double click the installer.

Accept the license agreement and default options. At the end of the installation, you can choose whether or not to “initialize Miniconda3 by running conda init?” The default is “no.” Choose “yes” to activate Miniconda.

2. Install Jupyter Lab

I primarily work in the Jupyter Lab environment, so I need to install Jupyter Lab. On terminal (Linux/MacOS) or Anaconda Powercell Prompt (Windows), type

conda install jupyterlab

3. Create Conda Environment

Keep the base conda environment minimal, and use one or more conda environments to install the package you need for the task or project you’re working on. To create an environment, use the conda create command and then activate the environment:

conda create -n env-name
conda activate env-name

4. Install Essential Python Libraries With Conda

Then, run the conda install command to install the different packages and software you want to include in the installation. Here is the list of essential packages for Scientific Machine Learning practice:

NumPy, SciPy, Matplotlib, Plotly, Pandas, Tensorflow, Keras, Tensorflow-Probability, pyDOE2

conda install -c numpy scipy matplotlib plotly pandas tensorflow 
conda install -c conda-forge pydoe2
conda install keras, tensorflow-probability

(If your machine has a GPU capability, then you may install the corresponding packages, such as “tensorflow-gpu” instead of “tensorflow”.)

5. Install Jupyter Kernel under the Conda Environment

To install all dependencies needed to use Jupyter:

conda install ipykernel

Then, You install the kernel for this environment. I usually use the same kernel name as the environment name here (i.e. env-name in this example).

ipython kernel install --user --name=env-name

6. Save the Conda Environment to a “yml” File

If you want a record of what is installed in your environment or want to reproduce your work environment with a history of the exact setup of your environment, conda can create a file, usually called, environment.yml, that describes the precise versions of all the packages you have installed in an environment. To create an environment.yml file from your currently-activated environment, run:

conda env export > environment.yml

When something goes wrong or dependencies are broken, you may use “environment.yml” to reproduce your workflow:

conda env create -f environment.yml

Deactivate the current environment:

conda deactivate

If you want to upgrade to a newer version or add new dependencies by changing the yml file, then you can update it by

conda env update -f environment.yml --prune

Here — pruneoption causes conda to remove any dependencies no longer required from the environment. Before updating the yml file, you better save the current environment setting as, for example, “environment-old.yml” so you can retrieve the setting when the updated environment goes wrong.

Remark 1. You better save and update the yml file in your home directory (e.g., /home/kim/).

Remark 2. On Linux (Ubuntu), updating the yml file doesn’t upgrade Tensorflow to a new version. Activate the environment and upgrade Tensorflow by typing conda update tensorflow on the command line.

Remark 3. You may also update an individual package on the command line outside the virtual environment. For example, conda update -n myenv tensorflow

7. Work under Jupyter Lab

If you want to work in a Jupyter environment, start Jupyter Lab

Jupyter Lab

You can choose kernel “environment” on the Jupyter launching pad to do your work in the corresponding conda environment. (You don’t need to activate the conda environment on the command line before launching Jupyter Lab.)

References

• CREATE A PORTABLE PYTHON INSTALLATION WITH MINICONDA
• Stop Using Anaconda for Your Data Science Projects
• How To Use Miniconda With Python and JupyterLab
• How to Install TensorFlow and Keras with GPU support on Windows (Remark: At the Install Step 4, the ‘cuDNN’ package is in zip format, so you might need to install manually following this procedure and create the folder ‘C:\Program Files\NVIDIA\CUDNN\v8.4’.)