Friday, October 06, 2023

Pointing CPAN To Your Local Mirror

 I have been out of the Perl development arena for quite a while (about 10 years or so, to be brutally honest).  That's what happens when your career takes you in a different direction.  

Recently I re-found my wanting to code in Perl after reading some postings by a friend, which made me realize that I was kind of missing it.  

Anywho, that brings me to the reason for this article.  I am re-learning some things, and one of those things is the setting up of your own, local CPAN repository.   There are times when one may not have an internet connection, but wants to work, none-the-less.  This is where having a mirror of CPAN on your local machine is a good thing.  

I am not going to go into how to create the mirror, because there is quite the plethora of documentation and write-ups on that topic, but instead, I am going to write a quick bit on actually pointing to your local mirror. 

Now, it matters not where you have it, but for the sake of this article, let's say that you have the mirror located at /tmp/cpan_mirror  (yes, I am assuming that you are on Linux.  If you are on Windows, I hate to say it but you are on your own as I don't touch the thing).

The first thing you need to do is drop into the cpan cli.  You can do this simply by typing cpan at your command line and hitting enter.  You'll be dropped to the cpan prompt:

 cpan[?]>

Once there, you'll need to do the following things.  I will create a list of tasks first, and then put the commands after. 

  • check the urllist and see what other mirrors are listed
  • remove all other mirrors (saving the url's aside if you wish to reset them later)
  • add your local repo path
  • commit your changes
  • reload cpan

Ok, so here are the commands:

$ cpan       (to drop into the cpan prompt)
cpan[?]> o conf urllist
cpan[?]> o conf urllist pop   <--- run this till any urls are out of the list
cpan[?]> o conf urllist "file:///tmp/cpan_mirror/"
cpan[?]> o conf commit
cpan[?]> reload cpan

After that, you should be able to, at the cpan prompt or using the cli, install modules.  You can test its pointing to your local repo by turning off your wifi and ensuring you cannot get to the internet.

To reset it back to internet based mirrors, just follow the same procedure, adding the mirror you like to use in place of your local path, and without the double quotes.

Enjoy!  And TMTOWTDI!!

 

 

Tuesday, February 07, 2023

Removing Unwanted Lines From A File

Intro

 This is a bit of a "back to basics" post.  I find it is good to revisit, now and then, as it keeps you sharp.  It also helps as a reminder when you find yourself needing to perform this task.  One never know when they need to remove lines from something like a log file. ( for say, hiding one's interactions on a server )

So, lets say you have a file ( we will call it logfile.txt) on a system, and you need to remove all the lines that contain the text foobar (I know, pretty standard, but you understand the usage).  I will preface this with the fact that this is being done on Linux.  And as with anything on Linux, there are a plethora of ways to do things.  So long as the end result is what you require, then the method was correct, even if there are quicker or more efficient ways.    These are just a couple of the quick ways that you could achieve this goal.  So without further ado, let's jump right in.

NOTE: Please keep in mind that all of these methods will achieve the same exact thing, just in different ways.



Method 1

So the first method is more my preferred method.  Why my preferred?  Well, because its a one-liner and it doesn't require me to do any moving of files to different names.  Its just quick and direct.  But also, you need to make sure you are absolutely positive that it is doing what you expect, as it is immediately effecting of your file and not reversible.

 

The Method:

$ sed -i '/sometext/d' logfile.txt

 In the above, the '-i' tells sed to edit the file in place (that's what makes this a more dangerous and immediately effecting version).  The /sometext/ is the text that you want to match on each line in the file.  (yes, the file will be read, line by line, matching against 'sometext' to check for a match).  The 'd' option says to delete that line if a match is found.  'logfile.txt' is the file to search in. 

This tends to work quick and is efficient.  It would help your situation to search the file, say with grep, first, and ensure of what you will be matching.  Caution is always a good thing to err on the side of, but that's just me. 


A Note Before Continuing:  The above method is the only immediate method I am presenting.  The other two methods I show, will involve the use of temporary files.  That said, they are the safer options, unless of course, you decide to script them, in which case, you make them more immediate.  Your choice.



Method 2

This method involves using grep and using the '-v' option, which will ignore the lines that match.  It will take the lines that do NOT match, and output them to the temporary file.  You will then take the temporary file, after grep runs over your file, and move it back to the original filename.  Or, you could take and name it something different, the option is yours.

 

The Method:

$ grep -v "sometext" logfile.txt > tempfile && mv tempfile logfile.txt

 

Again, you could move the tempfile to another filename.  Or, even better, after purging into the tempfile, rename the original file to a backup name, and then rename the tempfile to the original name.

 

 

Method 3

This is the last method I will cover here.  This involves using the awk utility.  Just as the others, it will do a match, but its method is to use a '!', which means DON'T match, which tells awk you want to match all lines that do not contain the matching text.  So any lines that do not have the 'sometext' matching text, will be output to the tempfile.  

 

The Method:

$ awk '!/sometext/' logfile.txt > tempfile && mv tempfile logfile.txt

 

And as noted previously, you can rename as you wish.  Either back to the original file name, or backing up the original name first, and then renaming.  Its totally up to you.  And also, totally scriptable.

 

Conclusion

I hope that this helps you manage this basic of tasks.  Enjoy!

 

 


Friday, January 13, 2023

Setting Up Your Own Remote Git Server

Introduction

 I don't know about anyone else, but I love having my GitHub / GitLab / BitBucket account, where I can store all of my code (whichever combination of those, or others you may use).  But, I find it handy to have my own local, self-hosted git server, where I can store my code, until I am at a point where I want it published up to one of those other sites.  Don't get me wrong, its not that I am not about sharing, but more about getting things to a usable point first.  Development takes time.   

Anywho, that is the 'gist' of this post..... what is involved in setting up your own, self-hosted git server, to store your code.  I could also note that this is super handy if you are also a bit paranoid.  Paranoia, to some degree, can be healthy.  Just don't let it get out of hand.  


My Setup

As I mentioned, I am running it on a Raspberry PI, but here are some further details on my setup:

  • Raspberry Pi 3 B+
  • A Raspberry Pi 3 mSata SSD Shield
  • A 256 Gb mSata ssd stick
  • A 16 Gb microSD card
  • A 7" touch screen display (all hooked up with the pi so I don't need a separate monitor)
I am using a 256 Gb drive, as I want to have enough room for whatever projects I decide to work on.  Also, its expandable, so not limited to just that size.  Depending on how much you plan on working on, adjust your ssd stick size.  


Setting Up Your Git Server

Considerations

You will need to work out where you are going to run this system.  It could be a spare system (desktop or laptop).... it could be in the cloud somewhere....  or, like me, you could set it up on a Raspberry Pi on your local network.  I find this handy, as its portable, should I need to take it with me on a trip, and it also uses a minute amount of power to run.   No matter what, the first thing you'll need to do is work out where to run it, before you can go to the install and setup. 

Another consideration is, that if you are using an mSata ssd, you'll need to set it up, format it and get it setup so it mounts at boot.   

Install the necessary software

So the first thing you will need to do is install git on your system.  This varies from system to system, so I am not going to get into that, but just make sure that git is installed.

Setup The Git User

If there is not a git user setup after the software install (there wasn't on my system), then you will want to do the following:
sudo adduser git   (provide whatever password you want, but you have to set something)
sudo -u git -i
mkdir .ssh && chmod 700 .ssh
touch .ssh/authorized_keys && chmod 600 .ssh/authorized_keys

 

Add Developer SSH Key(s) To The Git User's authorized_keys File

For each developer that you have using your system, you are going to have to add their public SSH key to the git users's ~/.ssh/authorized_keys file.  If you copy their key up to the /tmp directory, then you can do something like this, as root:

cat /tmp/userkey.pub >> ~git/.ssh/authorized_keys

Setting Up A New Repository

For each new repository that you wish to host on the server, you'll  need to create an empty repository to host the code that developer(s) will push.  You'll need to have a base directory that will house all of your repositories.  For instance, I have my ssd drive mounted at /data on my raspberry pi, and have a directory in there called git.  So I host all my repositories at /data/git.

To setup and empty repository for your code, do the following (I will give examplease in /data/git, where I host mine.  Feel free to change to whatever you are using):

cd /data/git
mkdir projectName.git
cd projectName.git
git init --bare
You will see something similar to this output after the git command is run:

Initialized empty Git repository in /srv/git/project.git/

Developer Setup To Push To Your New Repository

Now that you have your repository created and ready to receive code, all you need to do is have your developer point at it and push their code.  Here is how to point your repository at your new repo:

git remote add origin git@gitserver:/data/git/projectName.git

After they set that up, they just need to run:

git push origin master

 

Securing Your Setup

After you have gone through the trouble of setting up all of this, you are probably going to want to secure it, especially if you are sharing it with others.

Change The Shell For git User

NOTE:  Please know that once you do this, you can no longer just switch to the git user.  Anything you do withing the git user home directory will need to be as the root user.  So ensure you are ready to commit to this

The first thing you need to do is change the shell that is used by the git user to be git-shell.  git-shell is a special shell that comes with git, that only allows the git user to act for git related activities, and does not allow normal account shell access.  This is needed, as developer's ssh keys are in the git users' authorized_keys file, and they would otherwise be able to connect as git user, to your system.  This will prevent that. 

To change the shell for git user, run this:

sudo chsh git -s $(which git-shell)

Remove Port-Forwarding Ability For git User

The second thing you need to do to secure the server, is to remove the ability of users to get port-forwarded access, which would give them access to any host that can connect to the git server.  To remove this ability, you need to prepend the following text before the ssh-rsa portion of each and every developer key in the authorized_keys file:

no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty

 

After performing both of the above steps for security purposes, users will still be able to use git commands, but will be unable to get any shell on the system (or other systems).  


Some Notes

After you put the security precautions above into place, you will not be able to become the git user in order to create the new repositories and such.  If this is YOUR server, and nobody else will be on it, then you don't need to use the security precautions above, as you most likely aren't going to pwn your own data.   To be honest, if you're going to share with others, it would most likely be via one of the aforementioned services and not your own, self-manage git servers.  But if you do set something like this up and share with others at some point, make sure you trust them, or put the measures into place.  

It also worthwhile to note that scripting the repo creation and its modifications is probably a good idea.  Once I get that created, I will share them in another post.  

You'll notice that I did not get into any GUI interfaces, like Gitea, Gitlab, etc.  The reason is, I just wanted to provide guidance on setting up a plain git server for your code.  As such, no gui is needed for that.  If you want one of those, and decide to use one, there are a plethora of tutorials out there for installing and setting them up.   



Saturday, May 07, 2022

Thoughts On Ubuntu 22.04 (UPDATED)

 So here I am a few days ago, sitting at my computer, working on a project, and a pop-up appears asking me that Ubuntu 22.04 had been release, asking if I wanted to upgrade to it.  (Yes, I know that I can limit what I am notified of, I actually choose to know when the latest is available).  

I think, "Sure, why not." and clicked through the plethora of buttons to get it up upgrade (agreeing to the upgrade no less than 3 or 4 times). So my machine goes into upgrading and I decide to let it do its thing.  An hour or so later, I return and its at a login prompt, as it had rebooted.  Only, the login page is now dark themed.  Very nice!!!  The upgrade results were starting off on a good note.  

While its just a lot of packages that were upgraded, as well as the underlying software, I will say that there was an extremely annoying side-effect of this upgrade.  I have some projects that I have setup, including having done pip installs of packages and such for the software.  It seems that during the upgrade, all of those packages and settings went away.  Seriously?  Just *POOF*?  What the heck, Ubuntu?  Way to blow away my installs and configs.  

So, I had to spend some time re-setting up the stuff I regularly used (and some I was working on).  All of the items, by the way, were already using Python 3, so nothing except a newer version being installed.  

I just wanted to share this annoyance, in case you are planning on upgrading.  You will need to plan on some extra time, re-setting up some things once the upgrade completes.


UPDATE:  After some research, and noting issues seen on other machines (non-ubuntu) and work systems, this seems to be something related to a python3 upgrade.  On my work Mac, it seems that Python3 upgraded, and put itself in a different location, but did not take on what was previously installed.  It caused a ton of confusion in my system.  There were two different versions of pip3 and two different versions of python3.  It took using 'which -a' to figure this out and zero in on the issue.  In the end, I had to use an alias for each on my Mac to set things right.  

Here on my Ubuntu system though, the python3 upgrade simply wiped out everything that was installed with the previous python3 installation, that wasn't part of the standard library.  Thanks for that.  Way to seriously cause broken software that is used constantly.

Friday, April 29, 2022

So Long.... Youtube-DL

 A couple of years ago, I decided to write a quick script called ytdl, short for 'youtube downloader'.  I wrote it as a wrapper for the youtube-dl application, as I was tired of always trying to remember the options that I used when downloading either a video or a song (in audio format only) from youtube. 

Recently, I revisited the application as I needed to grab a couple of things off of youtube, and found that when running it, the downloads were CRAZY slow!  By CRAZY slow, I mean in the area of only getting between 25k and 50k max as a download speed.  I was looking at 1-2 hours to download each of the videos I was wanting to grab.  Considering in the past it would only have taken me a few minutes to download the videos I wanted (in total), I decided to use my Google Foo and see what was up. 

Sure enough, it appears that:

1. youtube.com was throttling the youtube-dl application.  This was confirmed by the plethora of complaints from the community that uses(used) it.  

2. I discovered that youtube-dl was no longer being maintained, and that the last version (from december of last year), was the final version. 

Well, that's not good.  But, what is good, is that there was a replacement...... yt-dlp!!!  Excellent, a successor.  So, I proceeded to download yt-dlp and incorporate it into my script, utterly replacing the old, useless youtube-dl.  

After some tweaking to get the downloaded files the way I like them, I can happily say that I am now seeing download speeds in the megabytes!!!  (7-13Mb, just on the dl's I did today).  

So, I have updated all the download types in the script and pushed up a new version.  If you haven't played with it, please feel free to give it a go.  And, if you have any download sites that you'd like added to the script (that are supported by it), please feel free to open an issue for an improvement.

Here is a link to my ytdl project.  Enjoy!

Wednesday, April 27, 2022

Removing Tag(s) From A Git Repository

 If you work on a team that is constantly developing their code, then you are probably dealing with at least one repository that gets periodically tagged (such as when a code release is performed).  If that is the case, then you have probably (or not) had to deal with setting the tags that get applied to the repository.  

If so, then you would be able to see the tags on a repository with the following command:

   $ git tag

If you are looking for a tag, but you only know part of it, you can certainly use grep to weed through the potentially large list of tags that will be presented to you.  Now, if you happen to set a tag incorrectly, or apply it to the wrong repository (or branch of that repository, for instance), you are not stuck.  You can remove it.  Here are the steps:

  1. First, ensure you are in the repository in question and in the correct branch, then do a 'git pull' to ensure you are at the latest change
  2. Run 'git tag | grep <tag>' to get the tag name you need to remove
  3. Run 'git tag -d <tag_name>'        <--- This will remove the tag in your local repository
  4. Run 'git push --delete origin <tag_name>'    <--- This will remove the tag from the remote repo
  5. Run 'git pull'
From here, you will need to run the 'git tag' command in step 2 and verify that the tag has been removed locally.  You can also then go up to GitHub, Gitlab, etc, and check the repository and ensure that the tag has been removed there as well.   

Wednesday, April 13, 2022

Read Medium Articles For Free

 There is nothing more annoying to someone who is just trying to read about something they are trying to learn about, than to have a site say, "Hey, you've reached your 'free' limit!".  That is exactly what medium.com does with its articles.  You need to log in to read further (so it says), and then, when you do, they STILL won't let you go any further until you pay for the right to read their articles.  Call me cheap, but I believe in the freedom of information.  

So, a little Google-foo and I discovered that there is a workaround for this that works in Chrome/Brave:

- Load up the article you want to read

- In the URL bar at the top, click the little padlock (the one that indicates that the site is secured)

- In there, it will tell you how many cookies there are.  Click on "Cookies".  It will bring up a window similar to this:


- Click on 'medium.com' and click 'Block' down below, and then Done.


Chrome/Brave will then ask you to reload the page.  Once you do, you should be able to read the article without the annoying message telling you that you haven't paid them for the privilege yet.  

Enjoy! 


**UPDATE:  I know its been quite a while since I posted this, but it is worth noting that this fix is truly only a temporary fix.  I tend to browse using a VPN, and find that the fix only works for a little while.   Once it runs out, I just switch to a different vpn location and voila, I can browse a little while longer.  Its not perfect, but the Scottish in me doesn't want to pay for Medium at the moment.  

Also, if you have this cookie block in place and you try to sign up for membership on Medium, the link they send you will hang indefinitely.  You'll have to remove the block, retry the link, and things should work.  Just noting.  

Sunday, April 10, 2022

Installing Dependencies for dpkg Software Installation

Let's say you download any of the plethora of '.deb' packages that exist out there, and want to install it on your Ubuntu system (or the like).  You would use something similar to the following to do that installation:

    sudo dpkg -i imager_1.7.2_amd64.deb

Now, let's say you run that, but are then presented with output stating that there are a bunch of unmet dependencies.  Disconcerting, sure, but its not the end of the world, for sure.  For example:

$ sudo dpkg -i imager_1.7.2_amd64.deb 

Selecting previously unselected package rpi-imager.

(Reading database ... 195449 files and directories currently installed.)

Preparing to unpack imager_1.7.2_amd64.deb ...

Unpacking rpi-imager (1.7.2) ...

dpkg: dependency problems prevent configuration of rpi-imager:

 rpi-imager depends on libqt5qml5 (>= 5.10.0); however:

  Package libqt5qml5 is not installed.

 rpi-imager depends on qml-module-qtquick2; however:

  Package qml-module-qtquick2 is not installed.

 rpi-imager depends on qml-module-qtquick-controls2; however:

  Package qml-module-qtquick-controls2 is not installed.

 rpi-imager depends on qml-module-qtquick-layouts; however:

  Package qml-module-qtquick-layouts is not installed.

 rpi-imager depends on qml-module-qtquick-templates2; however:

  Package qml-module-qtquick-templates2 is not installed.

 rpi-imager depends on qml-module-qtquick-window2; however:

  Package qml-module-qtquick-window2 is not installed.

 rpi-imager depends on qml-module-qtgraphicaleffects; however:

  Package qml-module-qtgraphicaleffects is not installed.


dpkg: error processing package rpi-imager (--install):

 dependency problems - leaving unconfigured

Processing triggers for mailcap (3.69ubuntu1) ...

Processing triggers for gnome-menus (3.36.0-1ubuntu1) ...

Processing triggers for desktop-file-utils (0.26-1ubuntu2) ...

Processing triggers for hicolor-icon-theme (0.17-2) ...

Errors were encountered while processing:

 rpi-imager


I know that is a lot of output, sorry about that.  But I left it that way for effect and to give a full example.  Now, what you will need to do in this case, is run the following:

        sudo apt -f install -y

That will take and install any dependencies that were found during the previous software installation attempt, and install them.  After that is done, you can re-run the dpkg command to install your software, and it "should" just work and install your software.  ( I say should in quotes, as nothing is guaranteed. )


Thursday, November 04, 2021

PyBlueprint update

 Since I have started this project, I have had an issue where the "checks" option, to verify that everything was correctly installed, has been failing to detect virtualenvwrapper as installed, even though it is in the 'pip list' output.  

After a bunch of digging and research, I have discovered a way to do the checks that seems to be working without issue.  I have installed it on a couple of different systems, both Linux and Mac, and things seem to be functional.  That is a breath of fresh air from me, as I have hated the error happening (and subsequently being ignored by me as I knew it was installed). 

So what is next?  I plan on adding support for more languages (shell, perl, and fix the ruby implementation to be more automated).  I also want to give an option for setting up directories for code that has been checked out from a repo, so that it can be added and setup more easily than trying to remember the commands to add it.  

We shall see how those options come along and hopefully I can advance this software's language support.  If anyone has any requests, by all means, please leave a comment and I will see about adding to my list of items.  Heck, the best way would be to open an enhancement in the repo.  You can do this by creating an issue and title it as an enhancement.  

Happy Geeking!

Friday, October 30, 2020

Firefox and Your Bookmarks

 Recovering your bookmarks from firefox

I am writing this as an alternate method of retrieving your bookmarks, other than using Firefox itself.  I had an issue where the main drive in one of my servers decided it didn't want to boot anymore.  I could still access the data on it via an external cable, but the system became unbootable.  

In searching, I found that Firefox stores its book marks in a file called 'places.sqlite'.  That's correct, it is an sqlite database, which means bookmarks are not the only things in this file.  The file is stored in ~/.mozilla/firefox/<blah>.default-release.  Now, i am on Firefox 82 and my full path is:  ~/.mozilla/firefox/6pcydzru.default-release.  

Now, the first thing to do is to go to that directory and copy that file to some place else (don't move it, copy it).

    cp places.sqlite ~   <- Yes, I simply copied it to my home directory

The reason for this is that Firefox locks this file and working around that lock is as simple as copying it to another location.  And you can keep the same name, if you wish, its not an issue.

Now, if you were to load up that file into sqlite3 and take a look, you would see that the bookmarks are certainly there, in a table called moz_bookmarks, but the urls are hashed and not readable.  Here is what it looks like when I loaded it up and took a look:

$ sqlite3 places.sqlite
SQLite version 3.31.1 2020-01-27 19:55:54
Enter ".help" for usage hints.
sqlite> .tables
moz_anno_attributes    moz_historyvisits      moz_meta             
moz_annos              moz_inputhistory       moz_origins          
moz_bookmarks          moz_items_annos        moz_places           
moz_bookmarks_deleted  moz_keywords         
sqlite> select * from moz_bookmarks;
1|2||0|0||||1602647046341000|1604087203227000|root________|1|1
2|2||1|0|menu|||1602647046341000|1602647046960000|menu________|1|3
3|2||1|1|toolbar|||1602647046341000|1602647046988000|toolbar_____|1|3
4|2||1|2|tags|||1602647046341000|1602647046341000|tags________|1|1
5|2||1|3|unfiled|||1602647046341000|1604087203227000|unfiled_____|1|8
6|2||1|4|mobile|||1602647046545000|1602647046943000|mobile______|1|2
7|2||2|0|Mozilla Firefox|||1602647046960000|1602647046960000|hk7fo0BVoSNm|0|1
8|1|3|7|0|Help and Tutorials|||1602647046960000|1602647046960000|mtMmQ-E9oI58|0|1
9|1|4|7|1|Customize Firefox|||1602647046960000|1602647046960000|DrTu2wYZmIj-|0|1
10|1|5|7|2|Get Involved|||1602647046960000|1602647046960000|-sWrRJ5NeO8R|0|1
11|1|6|7|3|About Us|||1602647046960000|1602647046960000|LCI5sKvqimve|0|1
12|2||2|1|Ubuntu and Free Software links|||1602647046960000|1602647046960000|wvi_2QDOZLm9|0|1
13|1|7|12|0|Ubuntu|||1602647046960000|1602647046960000|Zfpq8U7qHTzz|0|1
14|1|8|12|1|Ubuntu Wiki (community-edited website)|||1602647046960000|1602647046960000|6A8e4nK1328m|0|1

This isn't really usable, by any stretch, at the moment.  But a little googling, and I found a site called purposeful.co.uk, whom posted a nice shell script which would do the job for you.  
The link to this is: http://www.purposeful.co.uk/software/places2bookmarks/

The script is on there for you to copy to your system, I am not going to repost it here.  But, for readability, I will say that I modified a single line in the script.  Here is the modified line:

echo "$3  <A HREF=\"$url\">$title</A><br />"

As you can see, I added a '<br />' so each link would be on its own line, making things much nicer, and one entry per line.


Once you have the script in place (and the modification, if you so choose), you can run it like this: 

 ./places2bookmarks.sh ./places.sqlite > bookmarks.html

You will need to ensure that you replace the './' with the correct path to the files, if that is incorrect.  For me, I had them in the same directory

This doesn't seem huge thing, and sure, you could use the gui, but that is not me.  I prefer to do things on the command line and figure them out. Besides, I had fun doing it and learning about it, and thought I would share with everyone in case anyone else found it fun and fascinating.

Enjoy!

 
Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.