Friday, January 13, 2023

Setting Up Your Own Remote Git Server

Introduction

 I don't know about anyone else, but I love having my GitHub / GitLab / BitBucket account, where I can store all of my code (whichever combination of those, or others you may use).  But, I find it handy to have my own local, self-hosted git server, where I can store my code, until I am at a point where I want it published up to one of those other sites.  Don't get me wrong, its not that I am not about sharing, but more about getting things to a usable point first.  Development takes time.   

Anywho, that is the 'gist' of this post..... what is involved in setting up your own, self-hosted git server, to store your code.  I could also note that this is super handy if you are also a bit paranoid.  Paranoia, to some degree, can be healthy.  Just don't let it get out of hand.  


My Setup

As I mentioned, I am running it on a Raspberry PI, but here are some further details on my setup:

  • Raspberry Pi 3 B+
  • A Raspberry Pi 3 mSata SSD Shield
  • A 256 Gb mSata ssd stick
  • A 16 Gb microSD card
  • A 7" touch screen display (all hooked up with the pi so I don't need a separate monitor)
I am using a 256 Gb drive, as I want to have enough room for whatever projects I decide to work on.  Also, its expandable, so not limited to just that size.  Depending on how much you plan on working on, adjust your ssd stick size.  


Setting Up Your Git Server

Considerations

You will need to work out where you are going to run this system.  It could be a spare system (desktop or laptop).... it could be in the cloud somewhere....  or, like me, you could set it up on a Raspberry Pi on your local network.  I find this handy, as its portable, should I need to take it with me on a trip, and it also uses a minute amount of power to run.   No matter what, the first thing you'll need to do is work out where to run it, before you can go to the install and setup. 

Another consideration is, that if you are using an mSata ssd, you'll need to set it up, format it and get it setup so it mounts at boot.   

Install the necessary software

So the first thing you will need to do is install git on your system.  This varies from system to system, so I am not going to get into that, but just make sure that git is installed.

Setup The Git User

If there is not a git user setup after the software install (there wasn't on my system), then you will want to do the following:
sudo adduser git   (provide whatever password you want, but you have to set something)
sudo -u git -i
mkdir .ssh && chmod 700 .ssh
touch .ssh/authorized_keys && chmod 600 .ssh/authorized_keys

 

Add Developer SSH Key(s) To The Git User's authorized_keys File

For each developer that you have using your system, you are going to have to add their public SSH key to the git users's ~/.ssh/authorized_keys file.  If you copy their key up to the /tmp directory, then you can do something like this, as root:

cat /tmp/userkey.pub >> ~git/.ssh/authorized_keys

Setting Up A New Repository

For each new repository that you wish to host on the server, you'll  need to create an empty repository to host the code that developer(s) will push.  You'll need to have a base directory that will house all of your repositories.  For instance, I have my ssd drive mounted at /data on my raspberry pi, and have a directory in there called git.  So I host all my repositories at /data/git.

To setup and empty repository for your code, do the following (I will give examplease in /data/git, where I host mine.  Feel free to change to whatever you are using):

cd /data/git
mkdir projectName.git
cd projectName.git
git init --bare
You will see something similar to this output after the git command is run:

Initialized empty Git repository in /srv/git/project.git/

Developer Setup To Push To Your New Repository

Now that you have your repository created and ready to receive code, all you need to do is have your developer point at it and push their code.  Here is how to point your repository at your new repo:

git remote add origin git@gitserver:/data/git/projectName.git

After they set that up, they just need to run:

git push origin master

 

Securing Your Setup

After you have gone through the trouble of setting up all of this, you are probably going to want to secure it, especially if you are sharing it with others.

Change The Shell For git User

NOTE:  Please know that once you do this, you can no longer just switch to the git user.  Anything you do withing the git user home directory will need to be as the root user.  So ensure you are ready to commit to this

The first thing you need to do is change the shell that is used by the git user to be git-shell.  git-shell is a special shell that comes with git, that only allows the git user to act for git related activities, and does not allow normal account shell access.  This is needed, as developer's ssh keys are in the git users' authorized_keys file, and they would otherwise be able to connect as git user, to your system.  This will prevent that. 

To change the shell for git user, run this:

sudo chsh git -s $(which git-shell)

Remove Port-Forwarding Ability For git User

The second thing you need to do to secure the server, is to remove the ability of users to get port-forwarded access, which would give them access to any host that can connect to the git server.  To remove this ability, you need to prepend the following text before the ssh-rsa portion of each and every developer key in the authorized_keys file:

no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty

 

After performing both of the above steps for security purposes, users will still be able to use git commands, but will be unable to get any shell on the system (or other systems).  


Some Notes

After you put the security precautions above into place, you will not be able to become the git user in order to create the new repositories and such.  If this is YOUR server, and nobody else will be on it, then you don't need to use the security precautions above, as you most likely aren't going to pwn your own data.   To be honest, if you're going to share with others, it would most likely be via one of the aforementioned services and not your own, self-manage git servers.  But if you do set something like this up and share with others at some point, make sure you trust them, or put the measures into place.  

It also worthwhile to note that scripting the repo creation and its modifications is probably a good idea.  Once I get that created, I will share them in another post.  

You'll notice that I did not get into any GUI interfaces, like Gitea, Gitlab, etc.  The reason is, I just wanted to provide guidance on setting up a plain git server for your code.  As such, no gui is needed for that.  If you want one of those, and decide to use one, there are a plethora of tutorials out there for installing and setting them up.   



Saturday, May 07, 2022

Thoughts On Ubuntu 22.04 (UPDATED)

 So here I am a few days ago, sitting at my computer, working on a project, and a pop-up appears asking me that Ubuntu 22.04 had been release, asking if I wanted to upgrade to it.  (Yes, I know that I can limit what I am notified of, I actually choose to know when the latest is available).  

I think, "Sure, why not." and clicked through the plethora of buttons to get it up upgrade (agreeing to the upgrade no less than 3 or 4 times). So my machine goes into upgrading and I decide to let it do its thing.  An hour or so later, I return and its at a login prompt, as it had rebooted.  Only, the login page is now dark themed.  Very nice!!!  The upgrade results were starting off on a good note.  

While its just a lot of packages that were upgraded, as well as the underlying software, I will say that there was an extremely annoying side-effect of this upgrade.  I have some projects that I have setup, including having done pip installs of packages and such for the software.  It seems that during the upgrade, all of those packages and settings went away.  Seriously?  Just *POOF*?  What the heck, Ubuntu?  Way to blow away my installs and configs.  

So, I had to spend some time re-setting up the stuff I regularly used (and some I was working on).  All of the items, by the way, were already using Python 3, so nothing except a newer version being installed.  

I just wanted to share this annoyance, in case you are planning on upgrading.  You will need to plan on some extra time, re-setting up some things once the upgrade completes.


UPDATE:  After some research, and noting issues seen on other machines (non-ubuntu) and work systems, this seems to be something related to a python3 upgrade.  On my work Mac, it seems that Python3 upgraded, and put itself in a different location, but did not take on what was previously installed.  It caused a ton of confusion in my system.  There were two different versions of pip3 and two different versions of python3.  It took using 'which -a' to figure this out and zero in on the issue.  In the end, I had to use an alias for each on my Mac to set things right.  

Here on my Ubuntu system though, the python3 upgrade simply wiped out everything that was installed with the previous python3 installation, that wasn't part of the standard library.  Thanks for that.  Way to seriously cause broken software that is used constantly.

Friday, April 29, 2022

So Long.... Youtube-DL

 A couple of years ago, I decided to write a quick script called ytdl, short for 'youtube downloader'.  I wrote it as a wrapper for the youtube-dl application, as I was tired of always trying to remember the options that I used when downloading either a video or a song (in audio format only) from youtube. 

Recently, I revisited the application as I needed to grab a couple of things off of youtube, and found that when running it, the downloads were CRAZY slow!  By CRAZY slow, I mean in the area of only getting between 25k and 50k max as a download speed.  I was looking at 1-2 hours to download each of the videos I was wanting to grab.  Considering in the past it would only have taken me a few minutes to download the videos I wanted (in total), I decided to use my Google Foo and see what was up. 

Sure enough, it appears that:

1. youtube.com was throttling the youtube-dl application.  This was confirmed by the plethora of complaints from the community that uses(used) it.  

2. I discovered that youtube-dl was no longer being maintained, and that the last version (from december of last year), was the final version. 

Well, that's not good.  But, what is good, is that there was a replacement...... yt-dlp!!!  Excellent, a successor.  So, I proceeded to download yt-dlp and incorporate it into my script, utterly replacing the old, useless youtube-dl.  

After some tweaking to get the downloaded files the way I like them, I can happily say that I am now seeing download speeds in the megabytes!!!  (7-13Mb, just on the dl's I did today).  

So, I have updated all the download types in the script and pushed up a new version.  If you haven't played with it, please feel free to give it a go.  And, if you have any download sites that you'd like added to the script (that are supported by it), please feel free to open an issue for an improvement.

Here is a link to my ytdl project.  Enjoy!

Wednesday, April 27, 2022

Removing Tag(s) From A Git Repository

 If you work on a team that is constantly developing their code, then you are probably dealing with at least one repository that gets periodically tagged (such as when a code release is performed).  If that is the case, then you have probably (or not) had to deal with setting the tags that get applied to the repository.  

If so, then you would be able to see the tags on a repository with the following command:

   $ git tag

If you are looking for a tag, but you only know part of it, you can certainly use grep to weed through the potentially large list of tags that will be presented to you.  Now, if you happen to set a tag incorrectly, or apply it to the wrong repository (or branch of that repository, for instance), you are not stuck.  You can remove it.  Here are the steps:

  1. First, ensure you are in the repository in question and in the correct branch, then do a 'git pull' to ensure you are at the latest change
  2. Run 'git tag | grep <tag>' to get the tag name you need to remove
  3. Run 'git tag -d <tag_name>'        <--- This will remove the tag in your local repository
  4. Run 'git push --delete origin <tag_name>'    <--- This will remove the tag from the remote repo
  5. Run 'git pull'
From here, you will need to run the 'git tag' command in step 2 and verify that the tag has been removed locally.  You can also then go up to GitHub, Gitlab, etc, and check the repository and ensure that the tag has been removed there as well.   

Wednesday, April 13, 2022

Read Medium Articles For Free

 There is nothing more annoying to someone who is just trying to read about something they are trying to learn about, than to have a site say, "Hey, you've reached your 'free' limit!".  That is exactly what medium.com does with its articles.  You need to log in to read further (so it says), and then, when you do, they STILL won't let you go any further until you pay for the right to read their articles.  Call me cheap, but I believe in the freedom of information.  

So, a little Google-foo and I discovered that there is a workaround for this that works in Chrome/Brave:

- Load up the article you want to read

- In the URL bar at the top, click the little padlock (the one that indicates that the site is secured)

- In there, it will tell you how many cookies there are.  Click on "Cookies".  It will bring up a window similar to this:


- Click on 'medium.com' and click 'Block' down below, and then Done.


Chrome/Brave will then ask you to reload the page.  Once you do, you should be able to read the article without the annoying message telling you that you haven't paid them for the privilege yet.  

Enjoy! 


**UPDATE:  I know its been quite a while since I posted this, but it is worth noting that this fix is truly only a temporary fix.  I tend to browse using a VPN, and find that the fix only works for a little while.   Once it runs out, I just switch to a different vpn location and voila, I can browse a little while longer.  Its not perfect, but the Scottish in me doesn't want to pay for Medium at the moment.  

Also, if you have this cookie block in place and you try to sign up for membership on Medium, the link they send you will hang indefinitely.  You'll have to remove the block, retry the link, and things should work.  Just noting.  

Sunday, April 10, 2022

Installing Dependencies for dpkg Software Installation

Let's say you download any of the plethora of '.deb' packages that exist out there, and want to install it on your Ubuntu system (or the like).  You would use something similar to the following to do that installation:

    sudo dpkg -i imager_1.7.2_amd64.deb

Now, let's say you run that, but are then presented with output stating that there are a bunch of unmet dependencies.  Disconcerting, sure, but its not the end of the world, for sure.  For example:

$ sudo dpkg -i imager_1.7.2_amd64.deb 

Selecting previously unselected package rpi-imager.

(Reading database ... 195449 files and directories currently installed.)

Preparing to unpack imager_1.7.2_amd64.deb ...

Unpacking rpi-imager (1.7.2) ...

dpkg: dependency problems prevent configuration of rpi-imager:

 rpi-imager depends on libqt5qml5 (>= 5.10.0); however:

  Package libqt5qml5 is not installed.

 rpi-imager depends on qml-module-qtquick2; however:

  Package qml-module-qtquick2 is not installed.

 rpi-imager depends on qml-module-qtquick-controls2; however:

  Package qml-module-qtquick-controls2 is not installed.

 rpi-imager depends on qml-module-qtquick-layouts; however:

  Package qml-module-qtquick-layouts is not installed.

 rpi-imager depends on qml-module-qtquick-templates2; however:

  Package qml-module-qtquick-templates2 is not installed.

 rpi-imager depends on qml-module-qtquick-window2; however:

  Package qml-module-qtquick-window2 is not installed.

 rpi-imager depends on qml-module-qtgraphicaleffects; however:

  Package qml-module-qtgraphicaleffects is not installed.


dpkg: error processing package rpi-imager (--install):

 dependency problems - leaving unconfigured

Processing triggers for mailcap (3.69ubuntu1) ...

Processing triggers for gnome-menus (3.36.0-1ubuntu1) ...

Processing triggers for desktop-file-utils (0.26-1ubuntu2) ...

Processing triggers for hicolor-icon-theme (0.17-2) ...

Errors were encountered while processing:

 rpi-imager


I know that is a lot of output, sorry about that.  But I left it that way for effect and to give a full example.  Now, what you will need to do in this case, is run the following:

        sudo apt -f install -y

That will take and install any dependencies that were found during the previous software installation attempt, and install them.  After that is done, you can re-run the dpkg command to install your software, and it "should" just work and install your software.  ( I say should in quotes, as nothing is guaranteed. )


Thursday, November 04, 2021

PyBlueprint update

 Since I have started this project, I have had an issue where the "checks" option, to verify that everything was correctly installed, has been failing to detect virtualenvwrapper as installed, even though it is in the 'pip list' output.  

After a bunch of digging and research, I have discovered a way to do the checks that seems to be working without issue.  I have installed it on a couple of different systems, both Linux and Mac, and things seem to be functional.  That is a breath of fresh air from me, as I have hated the error happening (and subsequently being ignored by me as I knew it was installed). 

So what is next?  I plan on adding support for more languages (shell, perl, and fix the ruby implementation to be more automated).  I also want to give an option for setting up directories for code that has been checked out from a repo, so that it can be added and setup more easily than trying to remember the commands to add it.  

We shall see how those options come along and hopefully I can advance this software's language support.  If anyone has any requests, by all means, please leave a comment and I will see about adding to my list of items.  Heck, the best way would be to open an enhancement in the repo.  You can do this by creating an issue and title it as an enhancement.  

Happy Geeking!

Friday, October 30, 2020

Firefox and Your Bookmarks

 Recovering your bookmarks from firefox

I am writing this as an alternate method of retrieving your bookmarks, other than using Firefox itself.  I had an issue where the main drive in one of my servers decided it didn't want to boot anymore.  I could still access the data on it via an external cable, but the system became unbootable.  

In searching, I found that Firefox stores its book marks in a file called 'places.sqlite'.  That's correct, it is an sqlite database, which means bookmarks are not the only things in this file.  The file is stored in ~/.mozilla/firefox/<blah>.default-release.  Now, i am on Firefox 82 and my full path is:  ~/.mozilla/firefox/6pcydzru.default-release.  

Now, the first thing to do is to go to that directory and copy that file to some place else (don't move it, copy it).

    cp places.sqlite ~   <- Yes, I simply copied it to my home directory

The reason for this is that Firefox locks this file and working around that lock is as simple as copying it to another location.  And you can keep the same name, if you wish, its not an issue.

Now, if you were to load up that file into sqlite3 and take a look, you would see that the bookmarks are certainly there, in a table called moz_bookmarks, but the urls are hashed and not readable.  Here is what it looks like when I loaded it up and took a look:

$ sqlite3 places.sqlite
SQLite version 3.31.1 2020-01-27 19:55:54
Enter ".help" for usage hints.
sqlite> .tables
moz_anno_attributes    moz_historyvisits      moz_meta             
moz_annos              moz_inputhistory       moz_origins          
moz_bookmarks          moz_items_annos        moz_places           
moz_bookmarks_deleted  moz_keywords         
sqlite> select * from moz_bookmarks;
1|2||0|0||||1602647046341000|1604087203227000|root________|1|1
2|2||1|0|menu|||1602647046341000|1602647046960000|menu________|1|3
3|2||1|1|toolbar|||1602647046341000|1602647046988000|toolbar_____|1|3
4|2||1|2|tags|||1602647046341000|1602647046341000|tags________|1|1
5|2||1|3|unfiled|||1602647046341000|1604087203227000|unfiled_____|1|8
6|2||1|4|mobile|||1602647046545000|1602647046943000|mobile______|1|2
7|2||2|0|Mozilla Firefox|||1602647046960000|1602647046960000|hk7fo0BVoSNm|0|1
8|1|3|7|0|Help and Tutorials|||1602647046960000|1602647046960000|mtMmQ-E9oI58|0|1
9|1|4|7|1|Customize Firefox|||1602647046960000|1602647046960000|DrTu2wYZmIj-|0|1
10|1|5|7|2|Get Involved|||1602647046960000|1602647046960000|-sWrRJ5NeO8R|0|1
11|1|6|7|3|About Us|||1602647046960000|1602647046960000|LCI5sKvqimve|0|1
12|2||2|1|Ubuntu and Free Software links|||1602647046960000|1602647046960000|wvi_2QDOZLm9|0|1
13|1|7|12|0|Ubuntu|||1602647046960000|1602647046960000|Zfpq8U7qHTzz|0|1
14|1|8|12|1|Ubuntu Wiki (community-edited website)|||1602647046960000|1602647046960000|6A8e4nK1328m|0|1

This isn't really usable, by any stretch, at the moment.  But a little googling, and I found a site called purposeful.co.uk, whom posted a nice shell script which would do the job for you.  
The link to this is: http://www.purposeful.co.uk/software/places2bookmarks/

The script is on there for you to copy to your system, I am not going to repost it here.  But, for readability, I will say that I modified a single line in the script.  Here is the modified line:

echo "$3  <A HREF=\"$url\">$title</A><br />"

As you can see, I added a '<br />' so each link would be on its own line, making things much nicer, and one entry per line.


Once you have the script in place (and the modification, if you so choose), you can run it like this: 

 ./places2bookmarks.sh ./places.sqlite > bookmarks.html

You will need to ensure that you replace the './' with the correct path to the files, if that is incorrect.  For me, I had them in the same directory

This doesn't seem huge thing, and sure, you could use the gui, but that is not me.  I prefer to do things on the command line and figure them out. Besides, I had fun doing it and learning about it, and thought I would share with everyone in case anyone else found it fun and fascinating.

Enjoy!

Thursday, February 28, 2019

Untracking files in git

In one of my current projects, pyblueprint-py3, I have a config file.  This file gets edited locally after cloning the repository, and those changes get seen by git.  That is a problem.  This is a file that is distributed with default values that the user edits, and git does not need to know about these changes.  But, how do we tell git not to care about changes to a file?  Good question, and here is the answer.

For your config file (or whatever file it is that you want to prevent git from tracking), create it in your repository, and then issue the following command:

    git update-index --skip-worktree file_name

The file_name should include the path if it isn't in the current directory (ie: its in a subdirectory).

If the file that you don't want to track is already on git's radar, then you can do the following:

    git rm 
    git rm -r --cached 
    git commit -m'- Cleaning up and removing file to be ignored'
    git push

Then, add the file and edit the conf file with the generic placeholder and do this:

    git add 
    git update-index --skip-worktree 
    git commit -m'- Committing new, untracked file'
    git push

One thing to remember, is that when someone clones your repository, before they do anything, they will need to reissue the 'git update-index' command above for the specific file you untracked.  That should be noted in your documentation.



pyblueprint-py3 Now Exists

What a crazy 3/4 of a year its been.  The last company I worked for kept going down hill for the employees with a ton of management change-over, which was in no way good.  Eventually, things had to change and I needed to get back to a happy place.  At the turn of the year I started a new job, and BOY, am I happier.  While I am still ramping up at my new place of employment, I am certainly having a lot more fun here than I did the last couple years at my old job.  Considering what is going on over there, I do not envy my old colleagues.

Enough about the craziness.  I recently sat down and took the time to convert the pyblueprint project over to using Python3.  Not only that, there is a new option for specifying a language.  It defaults to python, but it will create a directory structure for whatever language you choose.  But, python and Ruby have support for the virtualenviroment that this project sets up (as does Go and NodeJS, but I haven't yet put together docs and such for that.  That is coming though.  I migrated the project to a new name, call pyblueprint-py3.  Feel free to download it and play with it.  Hopefully it works for you.  If there are issues, please open a ticket in gitlab under the project.  Also, I am open to feature requests, should anyone have any. 

Hopefully I can get more additions on to this project in the near future. 



 
Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.