Tuesday, February 07, 2023

Removing Unwanted Lines From A File


 This is a bit of a "back to basics" post.  I find it is good to revisit, now and then, as it keeps you sharp.  It also helps as a reminder when you find yourself needing to perform this task.  One never know when they need to remove lines from something like a log file. ( for say, hiding one's interactions on a server )

So, lets say you have a file ( we will call it logfile.txt) on a system, and you need to remove all the lines that contain the text foobar (I know, pretty standard, but you understand the usage).  I will preface this with the fact that this is being done on Linux.  And as with anything on Linux, there are a plethora of ways to do things.  So long as the end result is what you require, then the method was correct, even if there are quicker or more efficient ways.    These are just a couple of the quick ways that you could achieve this goal.  So without further ado, let's jump right in.

NOTE: Please keep in mind that all of these methods will achieve the same exact thing, just in different ways.

Method 1

So the first method is more my preferred method.  Why my preferred?  Well, because its a one-liner and it doesn't require me to do any moving of files to different names.  Its just quick and direct.  But also, you need to make sure you are absolutely positive that it is doing what you expect, as it is immediately effecting of your file and not reversible.


The Method:

$ sed -i '/sometext/d' logfile.txt

 In the above, the '-i' tells sed to edit the file in place (that's what makes this a more dangerous and immediately effecting version).  The /sometext/ is the text that you want to match on each line in the file.  (yes, the file will be read, line by line, matching against 'sometext' to check for a match).  The 'd' option says to delete that line if a match is found.  'logfile.txt' is the file to search in. 

This tends to work quick and is efficient.  It would help your situation to search the file, say with grep, first, and ensure of what you will be matching.  Caution is always a good thing to err on the side of, but that's just me. 

A Note Before Continuing:  The above method is the only immediate method I am presenting.  The other two methods I show, will involve the use of temporary files.  That said, they are the safer options, unless of course, you decide to script them, in which case, you make them more immediate.  Your choice.

Method 2

This method involves using grep and using the '-v' option, which will ignore the lines that match.  It will take the lines that do NOT match, and output them to the temporary file.  You will then take the temporary file, after grep runs over your file, and move it back to the original filename.  Or, you could take and name it something different, the option is yours.


The Method:

$ grep -v "sometext" logfile.txt > tempfile && mv tempfile logfile.txt


Again, you could move the tempfile to another filename.  Or, even better, after purging into the tempfile, rename the original file to a backup name, and then rename the tempfile to the original name.



Method 3

This is the last method I will cover here.  This involves using the awk utility.  Just as the others, it will do a match, but its method is to use a '!', which means DON'T match, which tells awk you want to match all lines that do not contain the matching text.  So any lines that do not have the 'sometext' matching text, will be output to the tempfile.  


The Method:

$ awk '!/sometext/' logfile.txt > tempfile && mv tempfile logfile.txt


And as noted previously, you can rename as you wish.  Either back to the original file name, or backing up the original name first, and then renaming.  Its totally up to you.  And also, totally scriptable.



I hope that this helps you manage this basic of tasks.  Enjoy!



Friday, January 13, 2023

Setting Up Your Own Remote Git Server


 I don't know about anyone else, but I love having my GitHub / GitLab / BitBucket account, where I can store all of my code (whichever combination of those, or others you may use).  But, I find it handy to have my own local, self-hosted git server, where I can store my code, until I am at a point where I want it published up to one of those other sites.  Don't get me wrong, its not that I am not about sharing, but more about getting things to a usable point first.  Development takes time.   

Anywho, that is the 'gist' of this post..... what is involved in setting up your own, self-hosted git server, to store your code.  I could also note that this is super handy if you are also a bit paranoid.  Paranoia, to some degree, can be healthy.  Just don't let it get out of hand.  

My Setup

As I mentioned, I am running it on a Raspberry PI, but here are some further details on my setup:

  • Raspberry Pi 3 B+
  • A Raspberry Pi 3 mSata SSD Shield
  • A 256 Gb mSata ssd stick
  • A 16 Gb microSD card
  • A 7" touch screen display (all hooked up with the pi so I don't need a separate monitor)
I am using a 256 Gb drive, as I want to have enough room for whatever projects I decide to work on.  Also, its expandable, so not limited to just that size.  Depending on how much you plan on working on, adjust your ssd stick size.  

Setting Up Your Git Server


You will need to work out where you are going to run this system.  It could be a spare system (desktop or laptop).... it could be in the cloud somewhere....  or, like me, you could set it up on a Raspberry Pi on your local network.  I find this handy, as its portable, should I need to take it with me on a trip, and it also uses a minute amount of power to run.   No matter what, the first thing you'll need to do is work out where to run it, before you can go to the install and setup. 

Another consideration is, that if you are using an mSata ssd, you'll need to set it up, format it and get it setup so it mounts at boot.   

Install the necessary software

So the first thing you will need to do is install git on your system.  This varies from system to system, so I am not going to get into that, but just make sure that git is installed.

Setup The Git User

If there is not a git user setup after the software install (there wasn't on my system), then you will want to do the following:
sudo adduser git   (provide whatever password you want, but you have to set something)
sudo -u git -i
mkdir .ssh && chmod 700 .ssh
touch .ssh/authorized_keys && chmod 600 .ssh/authorized_keys


Add Developer SSH Key(s) To The Git User's authorized_keys File

For each developer that you have using your system, you are going to have to add their public SSH key to the git users's ~/.ssh/authorized_keys file.  If you copy their key up to the /tmp directory, then you can do something like this, as root:

cat /tmp/userkey.pub >> ~git/.ssh/authorized_keys

Setting Up A New Repository

For each new repository that you wish to host on the server, you'll  need to create an empty repository to host the code that developer(s) will push.  You'll need to have a base directory that will house all of your repositories.  For instance, I have my ssd drive mounted at /data on my raspberry pi, and have a directory in there called git.  So I host all my repositories at /data/git.

To setup and empty repository for your code, do the following (I will give examplease in /data/git, where I host mine.  Feel free to change to whatever you are using):

cd /data/git
mkdir projectName.git
cd projectName.git
git init --bare
You will see something similar to this output after the git command is run:

Initialized empty Git repository in /srv/git/project.git/

Developer Setup To Push To Your New Repository

Now that you have your repository created and ready to receive code, all you need to do is have your developer point at it and push their code.  Here is how to point your repository at your new repo:

git remote add origin git@gitserver:/data/git/projectName.git

After they set that up, they just need to run:

git push origin master


Securing Your Setup

After you have gone through the trouble of setting up all of this, you are probably going to want to secure it, especially if you are sharing it with others.

Change The Shell For git User

NOTE:  Please know that once you do this, you can no longer just switch to the git user.  Anything you do withing the git user home directory will need to be as the root user.  So ensure you are ready to commit to this

The first thing you need to do is change the shell that is used by the git user to be git-shell.  git-shell is a special shell that comes with git, that only allows the git user to act for git related activities, and does not allow normal account shell access.  This is needed, as developer's ssh keys are in the git users' authorized_keys file, and they would otherwise be able to connect as git user, to your system.  This will prevent that. 

To change the shell for git user, run this:

sudo chsh git -s $(which git-shell)

Remove Port-Forwarding Ability For git User

The second thing you need to do to secure the server, is to remove the ability of users to get port-forwarded access, which would give them access to any host that can connect to the git server.  To remove this ability, you need to prepend the following text before the ssh-rsa portion of each and every developer key in the authorized_keys file:



After performing both of the above steps for security purposes, users will still be able to use git commands, but will be unable to get any shell on the system (or other systems).  

Some Notes

After you put the security precautions above into place, you will not be able to become the git user in order to create the new repositories and such.  If this is YOUR server, and nobody else will be on it, then you don't need to use the security precautions above, as you most likely aren't going to pwn your own data.   To be honest, if you're going to share with others, it would most likely be via one of the aforementioned services and not your own, self-manage git servers.  But if you do set something like this up and share with others at some point, make sure you trust them, or put the measures into place.  

It also worthwhile to note that scripting the repo creation and its modifications is probably a good idea.  Once I get that created, I will share them in another post.  

You'll notice that I did not get into any GUI interfaces, like Gitea, Gitlab, etc.  The reason is, I just wanted to provide guidance on setting up a plain git server for your code.  As such, no gui is needed for that.  If you want one of those, and decide to use one, there are a plethora of tutorials out there for installing and setting them up.   

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.