How do Jupyter notebook components work together?

[fog37]

TL;DR Summary

understanding the inner workings of jupyter notebook and similar apps

Hello everyone,
As mentioned in past threads, jupyter notebook is a popular application that is used to code in Python (and other languages as well). The jupyter notebook installation includes 3 software components working together:

1) kernel: it interact with the web server
2) server: web server that interacts with the client web app
3) client: web app written in JS that interacts with the browser (i.e. it uses the browser to display)

I read that the web server has 2 roles: be a ZMQ-Websocket bridge (?) and serve as a backend for the browser...

I am confused on how those 3 components work together.....The jupyter notebook server process on the local computer launches the browser that then connects to the server instance. The web server, at the request of the web client, invokes the python interpreter to get its results that are then relayed back to the web client using web protocols. And where is the kernel? The jupyter kernel is iPython (what is that? Isn't it just the previous version of Jupyter notebook to interactively run code?
Does the browser connect to the local server, and the server connects with the jupyter kernel (which is NOT the python interpreter running the Python code), and the kernel connect with the Python interpreter? And the results are then sent back up the chain to the browser to display?

I have created a virtual environment and launched jupyter notebook from it. The notebook opens but I realized that, to use the modules/libraries inside the environment, I need to first create a separate jupyter kernel for that specific environment by installing ipykernel and then creating and naming a new jupyter kernel. I then select the new specific kernel in my notebook and everything works just fine with imports... Why do I need to create a separate jupyter kernel for each different environment? The interpreter being used is the one associated to the active virtual environment...

Thank you!

 

[PeterDonis]

Why do I need to create a separate jupyter kernel for each different environment?

Because, as you yourself state, the kernel is Python code, which is importing other Python modules, and for the kernel you are using to import modules that are installed in a particular virtual environment, the kernel itself, as Python code, needs to be installed in that same virtual environment. That's the whole point of virtual environments: to install all of the Python code you want to use, and to work together with each other, in the same environment so everything can "see" everything else.

 

[fog37]

Because, as you yourself state, the kernel is Python code, which is importing other Python modules, and for the kernel you are using to import modules that are installed in a particular virtual environment, the kernel itself, as Python code, needs to be installed in that same virtual environment. That's the whole point of virtual environments: to install all of the Python code you want to use, and to work together with each other, in the same environment so everything can "see" everything else.

Fair :) Even that jupyter kernel must be the environment....my incorrect assumption was that by simply downloading jupyter notebook in the environment and launching it from the environment everything would have worked just fine, under the assumption that installing the jupyter notebook in the environment would have also taken care of the jupyter kernel...

 

[pbuk]

I am confused on how those 3 components work together

Not much to add to @PeterDonis's answer, except the diagram and explanation under this link might help:
https://docs.jupyter.org/en/latest/...hitecture.html#the-jupyter-notebook-interface

 

[fog37]

Not much to add to @PeterDonis's answer, except the diagram and explanation under this link might help:
https://docs.jupyter.org/en/latest/...hitecture.html#the-jupyter-notebook-interface

Ok, thank you.

So the kernel sits between the interpreter and the server. The server talks to the client which is run inside the browser.

Just downloading jupyter notebook into my environment does not automatically download also the kernel, PeterDonis states...

 

[pbuk]

So the kernel sits between the interpreter and the server. The server talks to the client which is run inside the browser.

I'm not sure that is right: in particular I'm not sure what you mean by "interpreter" - there is nothing labelled like that on the linked diagram.

If you want to say that the kernel "sits between" anything then that would be between the Jupyter Server and the operating system.

Just downloading jupyter notebook into my environment does not automatically download also the kernel

I don't know what you mean by "downloading", but to be clear the Jupyter Notebook doesn't run your Python (or other language e.g. R) code, it sends your code to the iPython (or R etc.) Kernel. It is the iPython Kernel that actually runs your code, so only the venv that the Kernel is running in matters.

 

[fog37]

I'm not sure that is right: in particular I'm not sure what you mean by "interpreter" - there is nothing labelled like that on the linked diagram.

If you want to say that the kernel "sits between" anything then that would be between the Jupyter Server and the operating system.I don't know what you mean by "downloading", but to be clear the Jupyter Notebook doesn't run your Python (or other language e.g. R) code, it sends your code to the iPython (or R etc.) Kernel. It is the iPython Kernel that actually runs your code, so only the venv that the Kernel is running in matters.

Hi,

by "interpreter" I simply mean the Python interpreter that take Python written code and converts it only bytecode, etc.
It is true that "interpreter" is not mentioned in the diagram. I guess the iPython kernel is what accomplishes that the interpreter does....Doesn't the virtual environment itself have its own Python interpreter version? I guess jupyter notebooks don't use that interpreter and use the iPython kernel instead as you mention...that was my confusion: thinking that the code would be run by the Python interpreter in the virtual environment and not by the kernel...

 

[PeterDonis]

Doesn't the virtual environment itself have its own Python interpreter version?

A virtual environment runs a modified version of the Python interpreter that uses the environment's installed modules and packages.

I guess jupyter notebooks don't use that interpreter and use the iPython kernel instead as you mention

IPython is a front end that uses whatever Python interpreter is visible to it, so if it is installed in a virtual environment, it will use the virtual environment's interpreter. The IPython "kernel" is just the program that interfaces between whatever is using the IPython front end (such as a Jupyter notebook) and the Python interpreter, which is what actually executes the Python byte code.

 

[fog37]

Hello and thanks again for your help. I have tested a few things and here my conclusions and a couple of observations. Please feel free to make corrections:

1. Anaconda comes with jupyter notebook, jupyter lab, VS code and other IDEs preinstalled.
2. When we create a new virtual environment (called myenv) in the Anaconda terminal using conda, we don't need to install jupyter notebook or jupyter lab again (since they already exist) inside the environment.,, What we need to do is a) install the package ipykernel, b) create a brand new jupyter kernel called myenvkernel, c) install the various libraries using either conda or pip....

1st observation: I had to install pip inside my new (myenv) environment. If I don't do that and type pip install <package>, the package gets install just fine BUT in the (base) environment, not in my new (myenv), which is where I want the package to be. So after installing pip via conda install pip, all my package installs end up in (myenv) as intended.

3. To launch jupyterlab we simply type jupyter lab in the new activated environment (myenv). A new notebook open. I choose the kernel myenvkernel and all works fine, i.e. the notebook has access only to the packages installed in (myenv).

2nd observation: while in jupyterlab with the myenvkernel running, I can open another new notebook and change the kernel to iPython3, which is the kernel of the (base) environment...Just like that, my new notebook gets connected with the (base) environment and it can use all its internal modules/packages....I didn't need to first deactivate (myenv) and then activate (base). I could do all that from the notebook by switching kernels which switched environment for the notebooks...

Side question: what is the different between the Anaconda terminal and the regular Windows terminal?

Thank you again!

 

[PeterDonis]

I had to install pip inside my new (myenv) environment.

Yes, that's true of every virtual environment with Python. AFAIK most tools that create virtual environments do this automatically. I'm a bit surprised that Anaconda's tools don't.

 

[fog37]

Yes, that's true of every virtual environment with Python. AFAIK most tools that create virtual environments do this automatically. I'm a bit surprised that Anaconda's tools don't.

What if we installed a jupyter notebook inside every virtual environment that I use? That would be redundant maybe but I guess we would not have to create a separate kernel for each virtual environment. Each environment would have its own jupyter notebook with its automatically created kernel. I think I have seen some instructor online opt for approach for his students.

That said, I think having a single jupyter notebook (i.e. not installing it for each environment) and creating a different kernel for each virtual environment seems a better choice.

 

[PeterDonis]

What if we installed a jupyter notebook inside every virtual environment that I use?

From what you're describing, it looks like Anaconda's virtual environment tools might be automatically doing that.

That said, I think having a single jupyter notebook (i.e. not installing it for each environment) and creating a different kernel for each virtual environment seems a better choice.

I'm not sure how this would be possible. If you're using Python in a virtual environment, you need to have everything Python is going to have to import installed in that virtual environment.

 

[fog37]

From what you're describing, it looks like Anaconda's virtual environment tools might be automatically doing that.I'm not sure how this would be possible. If you're using Python in a virtual environment, you need to have everything Python is going to have to import installed in that virtual environment.

That makes sense. Your point is that I should not be able to launch any IDE program that is not installed inside the virtual environment itself...
But I can. I didn't install jupyter lab via pip or conda. However, when typing conda list, I see that there is a jupyter_client and jupyter_core installed:

Back to your point: if an environment is activated, we can still launch a Python script .py from it which will have access to the environment resources in the environment. The script .py is a program which is not located inside the environment but that seems ok. Why couldn't be the same rationale apply to when we launch an IDE (a program) that is not installed inside the environment itself?

 

[PeterDonis]

Your point is that I should not be able to launch any IDE program that is not installed inside the virtual environment itself...

No, just that if you launch a Python program when an environment is not activated, it will not be using the Python modules or packages that are installed in the virtual environment. If you have other versions of the same Python modules or packages installed elsewhere on your system, it will use those.

I didn't install jupyter lab via pip or conda. However, when typing conda list, I see that there is a jupyter_client and jupyter_core installed:

That might mean, as I said, that Anaconda installed these things automatically, without your having to tell it to. Or it might mean that you are using versions of these things that are installed elsewhere on your system, not in the environment. That depends on whether or not the environment was activated when you ran the script. See below.

if an environment is activated, we can still launch a Python script .py from it which will have access to the environment resources in the environment.

Yes.

The script .py is a program which is not located inside the environment but that seems ok

If the environment is activated, any Python program you run from that shell is "located inside the environment". It doesn't matter where the program is installed in your file system. It matters what Python interpreter and modules and packages get loaded.

Why couldn't be the same rationale apply to when we launch an IDE (a program) that is not installed inside the environment itself?

"Not installed inside the environment itself" doesn't apply to scripts you run directly. It applies to the Python interpreter and Python modules and packages. Activating an environment means that any Python program you run from that shell will use the Python interpreter and modules and packages that are installed in the environment that is activated. See above.

 

[fog37]

No, just that if you launch a Python program when an environment is not activated, it will not be using the Python modules or packages that are installed in the virtual environment. If you have other versions of the same Python modules or packages installed elsewhere on your system, it will use those.That might mean, as I said, that Anaconda installed these things automatically, without your having to tell it to. Or it might mean that you are using versions of these things that are installed elsewhere on your system, not in the environment. That depends on whether or not the environment was activated when you ran the script. See below.Yes.If the environment is activated, any Python program you run from that shell is "located inside the environment". It doesn't matter where the program is installed in your file system. It matters what Python interpreter and modules and packages get loaded."Not installed inside the environment itself" doesn't apply to scripts you run directly. It applies to the Python interpreter and Python modules and packages. Activating an environment means that any Python program you run from that shell will use the Python interpreter and modules and packages that are installed in the environment that is activated. See above.

Yes, thank you. I think we agree:

On my machine, Jupyter lab was already existing somewhere before I created my new virtual environment (ven1) because when I downloaded Anaconda jupyter came preinstalled...

With the environment (ven1) activated, typing (ven1) ../../fog37/ pip install jupyter notebook essentially re-installs the jupyter IDE (inside (ven1) I guess, which is not a good place since only packages and modules should be inside ven1) which is unnecessary since it is already installed.

 

[PeterDonis]

(inside (ven1) I guess, which is not a good place since only packages and modules should be inside ven1

Isn't what you are trying to install a Python package that reads, writes, and edits jupyter notebooks?

 

[pbuk]

@fog37 you do seem to be rather confused. What is it you are actually trying to do that you can't do by pressing 'Start' and typing 'Jupyter'?

 

[fog37]

@fog37 you do seem to be rather confused. What is it you are actually trying to do that you can't do by pressing 'Start' and typing 'Jupyter'?

Hi, I was indeed confused.

Just to clarify and close the loop in this topic (you have helped enough), it all started when I created a new virtual environment and launched jupyter notebook from it. A new notebook opened but I could not import the virtual environment modules inside of the notebook...unless I first created a jupyter kernel inside that same new environment. After that, my notebook could use that new kernel and have access to the resources in the virtual environment without a problem.

I did not fully understand the role of the jupyter kernel in relation to the Python interpreter inside the virtual environment and the jupyter server and jupyter client. This is origin of my questions.