Understanding where pip saves the downloaded Python modules

[fog37]

TL;DR Summary

Understanding which in folder pip saves the downloaded Python modules.

Hello,

I have been trying to figure out where things are stored in Windows. For example, I have two versions of Python, one is stored in the PYthon37 folder and one in the Anaconda3 folder. See below:

I have been using pip at the command prompt to install packages/modules and when typing at the command prompt pip list -v, I see that ALL currently installed packages are stored in a subfolder of the anaconda3 folder...I didn't expect because I thought they would we be stored is a subfolder of the Python37 folder instead...
Also, when using the IDLE, which uses the Python3.7 interpreter stored in the python37 folder, I can import numpy and other modules with no problem even if they are stored in the anaconda3 folder...Why? And why did pip store the downloaded modules in a a subfolder of the anaconda3 folder?

Thanks!

 

[jbunniii]

Do you also have two pips installed? If so, each is probably configured to install packages into a directory associated with its corresponding Python installation.

If both Pythons are the same version (3.7) then they'll be able to import each other's packages if visible in your PYTHONPATH.

 

[fog37]

at the command prompt, I only see one pip installed in a sub-folder of the anaconda3 folder. So that explains why all the modules are saved in a subfolder of the anaconda3 folder.

However, when I am using the IDLE, which was installed with the other Python (not the Anaconda one), the modules are available from the IDLE anyway even if it uses not the Anaconda Python interpreter...

Does it mean that my Python3.7 installation did not come with pip? I thought it would...Should install another pip?

 

[fog37]

There is an IDLE also in anaconda apparently:

But when I use the IDLE I launch it from here

which is not the IDLE part of anaconda...

 

[PeterDonis]

I see that ALL currently installed packages are stored in a subfolder of the anaconda3 folder...

I would assume that's because you used the pip that came with the anaconda installation of Python to install those packages. Pip is just a Python program, so whichever Python interpreter is running it will be the one it uses to determine where packages should go.

when using the IDLE, which uses the Python3.7 interpreter stored in the python37 folder, I can import numpy and other modules with no problem even if they are stored in the anaconda3 folder...Why?

Try looking at sys.path, since that gives the folders that the interpreter will look in for modules and packages.

This is one of those times where "try it!" is a good way of figuring out what is going on.

Btw, is there any particular reason why you have two Python installations? On Linux that's common, because pretty much any Linux install will install whichever version of Python the distro is using for system management scripts, but often that's not the version the user wants to run for their own programs, so they'll install a different one. But Windows doesn't use Python for its own system management, so the only way you'll have a Python installation at all is if you do it yourself.

 

[jedishrfu]

A lot of folks tend to use Anaconda in the Machine Learning field and its corresponding conda multi-purpose command to install packages from the Anaconda repos.

But when you're using the installed pyhtin or just not using the Anaconda distro then folks will use pip to get third party modules. However, Anaconda folks realized new folks coming to Ananconda may still want to use pip and so they provided it in their distro as well.

 

[fog37]

I would assume that's because you used the pip that came with the anaconda installation of Python to install those packages. Pip is just a Python program, so whichever Python interpreter is running it will be the one it uses to determine where packages should go.
Try looking at sys.path, since that gives the folders that the interpreter will look in for modules and packages.

This is one of those times where "try it!" is a good way of figuring out what is going on.

Btw, is there any particular reason why you have two Python installations? On Linux that's common, because pretty much any Linux install will install whichever version of Python the distro is using for system management scripts, but often that's not the version the user wants to run for their own programs, so they'll install a different one. But Windows doesn't use Python for its own system management, so the only way you'll have a Python installation at all is if you do it yourself.

Thanks.

No, having two Python installations is not intentional at all. When I started, I first downloaded the vanilla Python installation from the python.org. Later, I downloaded Anaconda, to try it and because it has Jupiter notebooks and all the packages already installed, and ended up in this place.

I will check sys.path and see what is there.

What would you recommend doing? Uninstalling Anaconda? Should I have not downloaded Anaconda having the regular Python installation?

 

[fog37]

A lot of folks tend to use Anaconda in the Machine Learning field and its corresponding conda multi-purpose command to install packages from the Anaconda repos.

But when you're using the installed pyhtin or just not using the Anaconda distro then folks will use pip to get third party modules. However, Anaconda folks realized new folks coming to Ananconda may still want to use pip and so they provided it in their distro as well.

Thanks. I thought I was using pip that came with the regular Python installation before I downloaded Anaconda...Now I have two pips and two Python interpreters.. It sounds messy...

 

[PeterDonis]

What would you recommend doing?

If you only want one Python installation you should probably uninstall whichever one you don't want. Since you seem to be using the extra features in the Anaconda installation, I would say that's the one you should keep. From what you've said, it seems like all of the packages you've installed are installed with the Anaconda Python anyway, so you won't be losing anything if you just uninstall the vanilla python.org Python.

 

[fog37]

Indeed, at the command line, when typing python, it is the Anaconda interpreter that gets launched, not the regular Python one. However, it talks about a conda environment not being activated and I am not there yet understanding what that does and means. I need to make sure I don't uninstall the correct Python :)

Also, I followed your advice and typed sys.path after that (see below). You mentioned that those are the folders that the interpreter look into for modules and packages. I see a lot of anaconda3 subfolders but there is one subfolder called site-packages which is located in the Python37 subfolder. Why would the Anaconda python interpreter look into a folder that pertains to the other python interpreter?

 

[PeterDonis]

there is one subfolder called site-packages which is located in the Python37 subfolder.

In the Python37 subfolder of your user account. That's not the same as the system-wide site-packages for the vanilla Python from python.org that you installed. Pretty much any Python installation is going to give you some way of installing packages specific to a particular user, and those will get installed in that user's account somewhere.

 

[fog37]

Thanks.

I guess it was a mistake to install both, the regular Python interpreter and Anaconda later on to try it out.
I should have removed the first Python interpreter before installing Anaconda. Anaconda has everything: the IDLE, pip, tons of modules, everything essentially.

Keeping both interpreters does not make sense.

Maybe using virtual environments would be a solutions. Those are folders containing modules, a pip, and a particular Python interpreter and they isolate their content from everything else. I can create virtual environments from either Anaconda or the regular vanilla Python...

 

[PeterDonis]

Maybe using virtual environments would be a solutions.

A solution to what? You don't appear to want to have multiple different interpreters around.

In any case, virtual environments don't eliminate the confusion you appear to have been having; you still have to know which of the multiple Python interpreters you have installed is the one that the virtual environment was made from, and nothing in the virtual environment itself tells you that.

Virtual environments are typically used when different Python programs people are running (or developing) require different versions of particular modules or packages, either from each other or from the ones that come with the global Python interpreter that is installed on the system.

 

[fog37]

A solution to what? You don't appear to want to have multiple different interpreters around.

In any case, virtual environments don't eliminate the confusion you appear to have been having; you still have to know which of the multiple Python interpreters you have installed is the one that the virtual environment was made from, and nothing in the virtual environment itself tells you that.

Virtual environments are typically used when different Python programs people are running (or developing) require different versions of particular modules or packages, either from each other or from the ones that come with the global Python interpreter that is installed on the system.

Ok, I see how virtual environments are not the solution but just a way to keep specific packages/modules aside for a specific project.

I guess what I call the problem is to have both a python.exe interpreter version 3.7.3 from the Anaconda installation and another Python interpreter version 3.7.9 from the regular Python installation on the same machine.
When typing python at the command line, not matter the current directory, the anaconda python interpreter is the one launched, not the other. Both python interpreters are stored in folders that are on the Windows PATH variable...Not sure why the Anaconda Python is the one starting and not the other..

I just wonder if having the other Python version 3.7.9 interpreter could create issues with modules, etc.

Not sure how to launch the version 3.7.9 Python interpreter directly from the command line yet. I can use the 3.7.9 version if I start the IDLE from the desktop menu:

Overall, I noticed, that working inside the IDLE that uses the 3.7.9 Python interpreter, I can still import and use
the modules located in the Anaconda subfolders. Those modules were installed using the pip coming with Anaconda. I guess 3.7.9 Python interpreter can have access to all those modules too...

 

[PeterDonis]

what I call the problem is to have both a python.exe interpreter version 3.7.3 from the Anaconda installation and another Python interpreter version 3.7.9 from the regular Python installation on the same machine.

Why would you want to?

Sometimes people who are doing software development (i.e., development of software they are going to ship to others) do want different Python interpreter versions on their machine so they can test software they are developing with different versions; but even then those will be different 3.x versions, such as 3.6 vs. 3.7 vs. 3.8, not different releases of the same 3.x version.

But if all you want to do is run Python programs for yourself (ones that you write or ones that others give you), I'm not sure why you would need more than one Python interpreter installed.

Both python interpreters are stored in folders that are on the Windows PATH variable...Not sure why the Anaconda Python is the one starting and not the other..

I would assume it's because the folder it is in appears first on the Windows PATH. Windows checks those folders in the order given.

working inside the IDLE that uses the 3.7.9 Python interpreter, I can still import and use
the modules located in the Anaconda subfolders. Those modules were installed using the pip coming with Anaconda. I guess 3.7.9 Python interpreter can have access to all those modules too...

Again, the way to check this is to look at sys.path. You should be able to do that in IDLE.

 

[fog37]

Why would you want to?

thanks.

I really did not want to do that. I have no desire to have two interpreters at all.

When I started, I just downloaded Python from python.org and got the version 3.7.9 interpreter. So far so good.
Later, I decided I wanted to try Anaconda because of all its modules, etc. so I downloaded it. And that installation apparently gave me the other interpreter, the version 3.7.3 interpreter. I don't recall if I made some wrong selection during the Anaconda installation process.

I guess a beginner like me, wanting to try the Anaconda distro, should have first uninstalled the regular Python interpreter and then downloaded Anaconda with its interpreter and everything else...

 

[jbunniii]

I guess a beginner like me, wanting to try the Anaconda distro, should have first uninstalled the regular Python interpreter and then downloaded Anaconda with its interpreter and everything else...

I don't see any reason why you can't uninstall the one you don't want now. If there is some technical reason, then I would suggest editing your PATH and PYTHONPATH to remove any directories associated with the Python that you don't want to use.

At least Windows users have the luxury of being able to have only one Python installation. Unix/Linux/MacOS users have to deal with a pre-installed Python that might not be the version they want (e.g. MacOS shipped with Python2 and not Python3 until last year) and which they might not want to modify by installing packages, for fear of screwing up whatever the OS is doing with Python.