Wednesday, March 01, 2017

I give you.... PyBlueprint

About 3 years ago (or so), I created a bash script for starting a new Python project.  I was tired of setting up each new project by hand and wanted a quicker way to do it as a time saver.   That endeavor yielded a script that I have used for the last 3 years.

While the script works fine from a bash perspective, I was not yet happy with it in that state.  So, I embarked on re-writing it in Python.  I didn't get the chance to work full force on it right away, but over the last week or so, I have made an immense amount of progress on it.  So much, that I am ready to announce the project for people to download, play with and hopefully enjoy.


PyBlueprint's whole purpose is to create a base project directory for new Python projects.  Some of the features that the script has are:

    - Creates project directory for you and populates it with a base set of files (script, README, etc)
    - Initializes a new git repo (or not, if that is your choice)
    - Create a new Github or Bitbucket repository for you to push your code to

I could be over estimating, but I would say that this project saves me about 5-10 minutes of initial project setup so that I can get to coding my projects quicker.

I know that not everyone works in the same manner and we each have our own requirements.  I just ask that if you are a python developer, please give this a shot and see if it works for you.  If there are suggestions, I am open to them, please just open an issue in the project.

Project Link:

UPDATE: I realized a short time ago that I still had the repository set to be private.  I have changed it to be a public repo.  Sorry about that!

Wednesday, September 14, 2016

An Update On Combining PDF Files

Back on January 19th, 2009, I wrote a blog post about combining pdf files into one.  This is something I do now and again with one of my hobbies.
When I wrote the aforementioned article, I was only doing this on a Linux server.  Since then I have become an avid Mac user, as well as continuing my love and infatuation with Linux.  On Mac, you have a plenty of the Linux utilities you are used to, but some take some searching, and even then they may not exist or be ported.
In my quest to find an easy way on Mac OSx to combine PDF files, I did run across one.  It is called, believe it or not, "Combine PDFs".  I know, simple name, but believe me when I say that this GUI tool is pretty awesome!  You simply select the files you want to combine and it then not only shows you the order it is putting the in, but it also allows you to preview each page.
The tool is installed using brew on Mac.  If you don't have brew, you'll have to either install it.  If you don't use brew, you can always try to find it for download.  But, I recommend using brew as its pretty handy for obtaining and installing software packages you need and that aren't available through the app store.
If you do use brew, you can install it with the following command:
  brew install Caskroom/cask/combine-pdfs
After that finishes, you will see it in your applications list, as it is a GUI based application.


Friday, September 09, 2016

Python pip Broken After Upgrade To OSx El Capitan

Wow its been a long time since I blogged.  I have no excuse other than things have been a touch crazy.  My work recently started allowing us to upgrade our Mac's from Yosemite to El Capitan.  For me the upgrade only took about 45 minutes, which is pretty quick in the grand scheme of things.  The download is a little over 6 gb and went pretty quickly.  

I was going along fine until today when I tried to install a new module using pip.  About 1/2 way through the installation I was presented with the following output:

    In file included from src/lxml/lxml.etree.c:321:
    src/lxml/includes/etree_defs.h:14:10: fatal error: 'libxml/xmlversion.h' file not found
    #include "libxml/xmlversion.h"
    1 error generated.
    Compile failed: command 'cc' failed with exit status 1
    creating tmp
    cc -I/usr/include/libxml2 -I/usr/include/libxml2 -c /tmp/xmlXPathInitf3OxN6.c -o tmp/xmlXPathInitf3OxN6.o
    /tmp/xmlXPathInitf3OxN6.c:1:10: fatal error: 'libxml/xpath.h' file not found
    #include "libxml/xpath.h"
    1 error generated.
    Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
    Perhaps try: xcode-select --install
    error: command 'cc' failed with exit status 1
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/private/tmp/pip-build-9IduXA/lxml/';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-RgavmY-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/tmp/pip-build-9IduXA/lxml/

I had installed other modules last week, but then I realized that that was prior to my upgrading.  I looked at the above output and noticed the line:

     Perhaps try: xcode-select --install

OF COURSE!!!  Although I upgraded, and XCode had been upgraded to, I still needed to do some things with XCode to get things going again.  First, I opened XCode and went through its initial startup after an upgrade.  A bunch of clicking and things were good there.  I then opened up my command line and, prefixing the command with 'sudo', I ran the 'xcode-select --install' command from above.  This presented me a pop-up window to install tools.  So, I accepted and allowed it to install.  Once finished, I tried my pip module installation again and VOILA!!!!  It worked!!!

So, in this lesson, always make sure to update your XCode and xcode-select before moving on after an upgrade.

Wednesday, February 10, 2016

Where Is All My iPhone Space???

So recently my work upgraded my iPhone to an iPhone 6.  In preparation for this, I plugged in my old phone and offloaded all of my photos.  I made sure to delete them so the phone was cleaned off (as I always did.   To my amazement though, when going into iTunes, I noticed that there was still about 2Gb of space used up by photos.

I navigated over to Photos on my phone and there was nothing.  I had emptied out all my stuff.  I did a search and something said to turn off photo synching in iTunes.  I remember I had done that a while ago, but checked it and sure enough, it was off.

So, a bit more digging and I discovered that apparently, there is an 'issue' (I call it that, but Apple may call it a feature), where there are deleted files that are hidden and not visible, remaining on your phone.  You cannot see them or find them..... so you think.    Time to have some fun.

If you would like to clean these hidden pictures off of your phone, do the following:

- go to Settings-> General -> Date & Time
- Uncheck "Set Automatically" and then set the date to August 1st, 2014 (I don't know why, its a magical date apparently.

After you have done that, get out of settings and open Photos.  You should now see a bunch of items in the deleted folder. Voila!! Your hidden photos that you have probably already offloaded to your laptop.

I don't know why Apple hides them, but I really wish one of their developers would look at this issue and make the phone do the right thing.  Our phones need space, not things we deleted taking up said space.

All you need to do with the deleted items is go into the folder, touch "Select" and select all items and then click 'Delete' and they will be gone...... Space recovered.  After that, go back to the Date & Time above and turn back on Set Automatically and your date should fix itself.

Enjoy the added space to your phone!

Thursday, January 07, 2016

Keep Your SSH Session ALIVE!!

Most of us have had those days where we have a bunch of windows open, ssh'd into multiple servers.  You get caught up working on one of those servers and forget about your other ssh sessions.  Or, if you are like me, you are using something like screen/tmux and have many sessions going at the same time that you want to keep alive.

Well, with the default ssh setup, there is a setting in your ssh_config file called "ServerAliveInterval".  Initially (at least on my system) it is commented out and also has a value of zero (0).  

In order to keep your sessions alive indefinitely, you will need to uncomment that variable in the ssh_config file and set it to a value other than zero.  I have mine set to 60. 

What this variable does is tell ssh that it needs to send a packet across the wire to the connected server so that there is traffic, keeping the connection active, not letting it time out. 

So, by to keep your sessions alive, you will need to make the changes as above, but also, do not forget that this change will only effect new connections made after you save the change.  If you have active connections, you will need to exit them out and restart them.  If you don't, they will still be subject to the same issue you just corrected.

Go now, keep those connections alive! 

Wednesday, January 06, 2016

Online Learning - Open Source Society (Computer Science)

Almost a year ago (1/31/2015 to be exact), I posted an article about online learning.  I reviewed a number of online places to take courses and learn a new skill and provided the list. 

In my research around the internet I stumbled across another resource which I have found very valuable.  I am referring to the Open Source Society page on Github.  The OSS has the tag line "Path to a free self-taught education in Computer Science".  This is great for someone similar to myself, who his completely self-taught, or is looking for an inexpensive way to get into the field, starting with Computer Science. 

The courses listed are sourced from several of the places that I reviewed in my previous article (MIT, Stanford, Coursera, Udacity, Edx, etc) and are all Free for the taking/learning.  The nice thing is, you don't have to go searching through and navigating each of the individual sites, the links are all gathered for you there on the OSS page. 

Hopefully this comes in as handy for everyone as it has for me.

Monday, November 23, 2015

Run Pip Against Your Local Pypi Server

A couple of weeks ago I posted about creating your own pypi mirror.  Being the anal type and wanting a complete mirror (in case I needed a slightly older version of any module), I gave a quick tutorial on how to do this using the bandersnatch module.

Hopefully you have had time to download everything and have sitting nicely in a directory.  I say hopefully as it was 180gb when I downloaded it and I know that not everyone has a fast connection.

As a note, anything done from this point is done inside of a virtual environment.  If you haven't used virtual environments and are not familiar with them, I highly suggest you read about them as they have worthy benefits.

Now that you have your mirror created you need to put it to good use.  To do this there are a couple of things you need to do:
  • install a module for running a pypi server
  • edit the appropriate config file(s) to point to the new repo
  • start the pypi server
  • install a module while configured to point to your new repo
 For the first piece, you will need to install a module that provides you a pypi server.  There are a few such modules out there, so feel free to research them and find one that works for you.  For our basic purposes (as I don't currently need much beyond serving the mirror) I am using the 'pypiserver' module.  You can install that with:

    $ pip install pypiserver

Once the server is installed, you will want to start it with the following command:

    $ pypi-server -p 7777 ~/pypi/packages

That will start the server on:  http://localhost:7777/

You now need to make some config changes.  At a minimum you need to change  ~/.pip/pip.conf
You will want to mofify 'index-url' in the 'global' section of the config file to point to 'http://localhost:7777/'.   (I do suggest you remember where your config file was pointing to before you change it.  Its always a good idea to make a backup copy of the config file before you make your edits).

Now that you have that configured things and have the server running, its time to turn off your internet connection and do a 'pip install' of a module and see if it works.  

Friday, November 20, 2015

Creating A Rubygems Mirror On Ubuntu 14.10

So I guess it is apparent that I am on a bit of a kick creating mirrors.  This is because they are creating mirrors internally at work and I want to have a better understanding of how things work (and what is needed to puppetize things). 

Before we can get down to the nitty-gritty of creating the mirror, we have to do a bit of prerequisite work first.   To start with, I will tell you that I am working on a fresh installation of Ubuntu 14.10.  That disclosed, ruby comes installed by default.  Please know that I will be only referencing the

The first thing that we need to get installed is the ruby-dev package.  You can install that with the following command:
$ sudo apt-get install ruby-dev
After you install that, make sure that your system is up to date:
$ sudo apt-get update
$ sudo apt-get upgrade
Now, we need to get a couple of gems installed:
$ sudo gem install net-http-persistent
Next we need to install a whole slew of things (inluding git):
sudo apt-get install git-core curl zlib1g-dev build-essential libssl-dev libreadline-dev libyaml-dev libsqlite3-dev sqlite3 libxml2-dev libxslt1-dev libcurl4-openssl-dev python-software-properties libffi-dev
Now it is time to install rbenv.  This tool will provide you the ability to install different versions of ruby (other than the default).  Its handy, so I am including it here for giggles.   Here are the steps to install rbenv (these should be done as you, not sudo/root):
$ git clone git:// ~/.rbenv
- Add $HOME/.rbenv/bin to your PATH variable
- Add eval "$(rbenv init -)" to your .bashrc file
 Now you need to install the rubygems-mirror gem.  This is what is used to create the mirror:
$ sudo gem install rubygems-mirror
After that finishes, you will need to create the the '.mirrorrc' file:
$ vi ~/.gem/.mirrorrc
 In this file, you are going to specify where to grab the gems from and where to put them on your system:
- from:
  to: ~/path/to/put/the/files
You will need to make sure that you create the directory you specify in the "to:" portion.
After you save the .mirrorrc file and create the directory, you are ready to start mirroring:
$ gem mirror
This is going to take a while.  I ran it and it took several hours to download (and I have a fiber internet connection with 30Mb down).  So, once you start the download, you might as well go do something (watch a movie, read a book, hack on a Raspberry Pi, whatever you feel like.  Just monitor it once in a while to ensure it keeps going successfully.

Warning:  The download is many gigabytes and you are going to need some space for it.

That's it, you now have a mirror of Rubygems that you can reference in offline situations.  Stay tuned and I will put up a post on how to run a server that points to this mirror so that you can put it to good use.

Tuesday, November 10, 2015

Creating A Pypi Mirror

One of the things I have read a lot of people like to do, is to create a local pypi mirror.  That way, when the real pypi is on the fritz or you don't have internet access, you can still install modules and work on your pet projects.

I worked through a bunch of different modules for creating a pypi mirror, and most of them seemed to make a mirror of the most recent versions of modules.  Which would be fine, except I am anal and wanted as complete of a mirror as I could get. 

So, after playing around with a number of different modules I discovered Bandersnatch.  At first it looked promising (claiming that the mirror would be about 120 Gb).  Considering that the module documentation was probably written (and not necessarily updated) a couple of years ago, I could only imagine what that number is now. 

I followed the installation from the above link and installed the module (in a virtual environment) and got it running and let it run until completion (which was about 3-4 hours later).  I did a df of the directory and BLAM!, a little over 180Gb of moudules.  Just WOW!!  Now that is the mirror I was looking for. 

I still want to play with some of the other methods (as having a mirror of the most recent set of modules is also handy), but this is definitely what I was looking for.  Hopefully this information allows others to create their own Pypi mirror as well.  I would say quickly, but that will depend on the internet connection you are using.

Monday, November 09, 2015

I Have A New Domain!!!!

At long last, I have set up a new domain name.  The previous domain ( ended up being too expensive to renew.  I now have a new domain name for the blog, something that goes better with the blogs name:

I hope that all the readers that I had previously will return.  I will put this out there asap so that people are aware of the new domain name.  Thank you to all for your patience during this arduous transition.
Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.