Monday, November 23, 2015

Run Pip Against Your Local Pypi Server

A couple of weeks ago I posted about creating your own pypi mirror.  Being the anal type and wanting a complete mirror (in case I needed a slightly older version of any module), I gave a quick tutorial on how to do this using the bandersnatch module.

Hopefully you have had time to download everything and have sitting nicely in a directory.  I say hopefully as it was 180gb when I downloaded it and I know that not everyone has a fast connection.

As a note, anything done from this point is done inside of a virtual environment.  If you haven't used virtual environments and are not familiar with them, I highly suggest you read about them as they have worthy benefits.

Now that you have your mirror created you need to put it to good use.  To do this there are a couple of things you need to do:
  • install a module for running a pypi server
  • edit the appropriate config file(s) to point to the new repo
  • start the pypi server
  • install a module while configured to point to your new repo
 For the first piece, you will need to install a module that provides you a pypi server.  There are a few such modules out there, so feel free to research them and find one that works for you.  For our basic purposes (as I don't currently need much beyond serving the mirror) I am using the 'pypiserver' module.  You can install that with:

    $ pip install pypiserver

Once the server is installed, you will want to start it with the following command:

    $ pypi-server -p 7777 ~/pypi/packages

That will start the server on:  http://localhost:7777/

You now need to make some config changes.  At a minimum you need to change  ~/.pip/pip.conf
You will want to mofify 'index-url' in the 'global' section of the config file to point to 'http://localhost:7777/'.   (I do suggest you remember where your config file was pointing to before you change it.  Its always a good idea to make a backup copy of the config file before you make your edits).

Now that you have that configured things and have the server running, its time to turn off your internet connection and do a 'pip install' of a module and see if it works.  



Friday, November 20, 2015

Creating A Rubygems Mirror On Ubuntu 14.10

So I guess it is apparent that I am on a bit of a kick creating mirrors.  This is because they are creating mirrors internally at work and I want to have a better understanding of how things work (and what is needed to puppetize things). 

Before we can get down to the nitty-gritty of creating the mirror, we have to do a bit of prerequisite work first.   To start with, I will tell you that I am working on a fresh installation of Ubuntu 14.10.  That disclosed, ruby comes installed by default.  Please know that I will be only referencing the

The first thing that we need to get installed is the ruby-dev package.  You can install that with the following command:
$ sudo apt-get install ruby-dev
After you install that, make sure that your system is up to date:
$ sudo apt-get update
$ sudo apt-get upgrade
Now, we need to get a couple of gems installed:
$ sudo gem install net-http-persistent
Next we need to install a whole slew of things (inluding git):
sudo apt-get install git-core curl zlib1g-dev build-essential libssl-dev libreadline-dev libyaml-dev libsqlite3-dev sqlite3 libxml2-dev libxslt1-dev libcurl4-openssl-dev python-software-properties libffi-dev
Now it is time to install rbenv.  This tool will provide you the ability to install different versions of ruby (other than the default).  Its handy, so I am including it here for giggles.   Here are the steps to install rbenv (these should be done as you, not sudo/root):
$ git clone git://github.com/sstephenson/rbenv.git ~/.rbenv
- Add $HOME/.rbenv/bin to your PATH variable
- Add eval "$(rbenv init -)" to your .bashrc file
 Now you need to install the rubygems-mirror gem.  This is what is used to create the mirror:
$ sudo gem install rubygems-mirror
After that finishes, you will need to create the the '.mirrorrc' file:
$ vi ~/.gem/.mirrorrc
 In this file, you are going to specify where to grab the gems from and where to put them on your system:
---
- from: http://rubygems.org
  to: ~/path/to/put/the/files
You will need to make sure that you create the directory you specify in the "to:" portion.
After you save the .mirrorrc file and create the directory, you are ready to start mirroring:
$ gem mirror
This is going to take a while.  I ran it and it took several hours to download (and I have a fiber internet connection with 30Mb down).  So, once you start the download, you might as well go do something (watch a movie, read a book, hack on a Raspberry Pi, whatever you feel like.  Just monitor it once in a while to ensure it keeps going successfully.

Warning:  The download is many gigabytes and you are going to need some space for it.

That's it, you now have a mirror of Rubygems that you can reference in offline situations.  Stay tuned and I will put up a post on how to run a server that points to this mirror so that you can put it to good use.
 



Tuesday, November 10, 2015

Creating A Pypi Mirror

One of the things I have read a lot of people like to do, is to create a local pypi mirror.  That way, when the real pypi is on the fritz or you don't have internet access, you can still install modules and work on your pet projects.

I worked through a bunch of different modules for creating a pypi mirror, and most of them seemed to make a mirror of the most recent versions of modules.  Which would be fine, except I am anal and wanted as complete of a mirror as I could get. 

So, after playing around with a number of different modules I discovered Bandersnatch.  At first it looked promising (claiming that the mirror would be about 120 Gb).  Considering that the module documentation was probably written (and not necessarily updated) a couple of years ago, I could only imagine what that number is now. 

I followed the installation from the above link and installed the module (in a virtual environment) and got it running and let it run until completion (which was about 3-4 hours later).  I did a df of the directory and BLAM!, a little over 180Gb of moudules.  Just WOW!!  Now that is the mirror I was looking for. 

I still want to play with some of the other methods (as having a mirror of the most recent set of modules is also handy), but this is definitely what I was looking for.  Hopefully this information allows others to create their own Pypi mirror as well.  I would say quickly, but that will depend on the internet connection you are using.

Monday, November 09, 2015

I Have A New Domain!!!!

At long last, I have set up a new domain name.  The previous domain (www.parsed.co) ended up being too expensive to renew.  I now have a new domain name for the blog, something that goes better with the blogs name:   www.parsedcontent.com

I hope that all the readers that I had previously will return.  I will put this out there asap so that people are aware of the new domain name.  Thank you to all for your patience during this arduous transition.
 
Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.