Learning the Command Line

Commands your mom should have taught you.

Do you think the command line is dead?! You are wrong, so wrong my fair-weathered friend! Even in the age of cloud computing, the command line is still going strong, stronger than ever before I wager. If you would only learn how to use the command line effectively, you can accomplish more faster, of this I am sure.

The command line is helpful in the cloud on AWS, GCP, or whichever cloud provider you work with. The reason? You will still need to remote into these services occasionally to work on your server. Not to mention the plethora of CLI tools they provide you.

You'll never look back once you learn the command line because it can be much faster than digging through the never-ending cloud UI.

So, that’s what we will do today, roll over the rock you’ve been living under, drag you out into the sunlight, dust the moss off you, and teach you some command line stuff.

You should check out Prefect, the sponsor of the newsletter this week! Prefect is a workflow orchestration tool that gives you observability across all of your data pipelines. Deploy your Python code in minutes with Prefect Cloud.

Benefits of learning the command line

Why learn the command line? Everyone is using tablets or phones to do their work, right? Not if you're an engineer! Engineers, unlike the teeming masses, use desktop and laptop computers to do their work. Don't put yourself into a cage where you only use the UI that is provided to you, there is nothing more stunting to an engineer’s career.

Some companies provide amazing, intuitive user interfaces, and take the overly complex but at the same time wonderful AWS Console. It’s almost too easy to become reliant on a GUI or web browser to do a job that could be automated via the command line.

But even the best user interface is slower than using a command line.

Here are some of the benefits you'll get from using the command line:

• Enables a faster development lifecycle

• The command line is much easier to automate

• If you can automate the command line, then you can scale it

• You'll beat the pants off your peers!

But wait? Doesn't each operating system work differently? Let's talk about that.

Linux vs Windows vs Mac OSX

Each operating system has its own command line utility. Let's talk about each of these tools. First, before I talk sides (*cough* Linux), I started out in Windows. Guess what?

Learning to be comfortable on the command line doesn’t really have that much to do with Windows, Mac OSX, or Linux. It’s more about knowing the general concepts and what the possibilities are, the rest is just Googling.

Options, Options.

You can choose between cmd.exe or Powershell or Windows Terminal on Windows. You should pick Powershell or Windows Terminal because it can do a lot more and you can use Linux-like commands in it that you cannot use in cmd.exe.

Mac comes with an iTerm tool that you use as a command line or terminal. You should upgrade and install iTerm2 because it looks much nicer and more feature-rich. You'll also be using Zsh in your terminal on Mac, which is a more feature-rich version of Bash

Linux's terminal usually defaults to something called Bash although you can also install zsh (AKA Z-shell) or other types of shells in your Linux terminal.

If you want to get fancy with your terminal, you can install custom prompts, such as oh-my-zsh or Starship.

What should we say to all those folks who say they don’t need to know the command line just to copy files around? There is a lot more that happens on the command line on the average day of a Data Engineer.

The command line assists using popular tools

What are some cloud-related tools that you can use from the command line? Here are just a few:

• Docker

• Kubernetes

• AWS

The reality is if you are going to be a senior engineer and work on high-performance teams where you can contribute, you are going to have to be comfortable around the command line.

Using Docker on the command line

You may not know it, but docker Desktop didn't always exist. When docker was first released, it was a command-line-only tool. Even now, you can use docker to do anything you can do in Docker Desktop and more!

For example, if you want to build a new image, you can do the following

Want to see what containers are running or stopped? Try the ps command:

If you don't know any docker commands, you can run docker all by itself or execute docker help to get a full listing.

Using Kubernetes on the command line

There are few infrastructure tools you are more likely to run across than Kubernetes. It’s used by many teams to deploy every sort of service and app you can think of.

The chances of you skating by your whole career never working with Kubernetes on the command line are slim. Consider yourself uncommonly lucky if you manage to get away scot-free.

Using AWS on the command line

I dare to say it’s hard to call yourself an engineer or developer in the age of cloud if you haven’t worked with cloud providers, especially AWS, via the command line.

If and when you work on AWS, more tasks can be listed that are done via the AWS CLI. It’s just another reason to learn the command line.

Introduction to command line basics and concepts

You've probably heard enough about the theory and are ready for more concrete examples. You've come to the right place!

In this section, you will learn the basics of the command line. Don’t worry about Mac vs Windows vs Linux. Focus on just understanding the basic command line options we are going to present here and learn how to do them on whatever system you’re working on.

Moving around directory structures

When you open a command line, you are in a directory. Usually, you'll start out in your user folder. You can type pwd to get your current location:

If you want to change your location, you'll use the cd command, which stands for change directory.

Here's an example:

Changing directories is all about navigating your directory structure, kind of like how you would do it in Windows Explorer or Mac's Finder application.

But you can do more than change directories!

Listing files

Once you have navigated to the directory you wish to be in, you'll probably want to check and see if there's anything in that directory.

On Windows, the most common command to use is dir. When you run dir, the command line will print out all the files and directories in your current location.

If you are on Mac or Linux, you can use the ls command instead to get the same functionality. You can also use ls on Windows if you use Powershell or Windows Terminal. However, ls is not supported by cmd.exe, so you have to use dir there.

Here's an example of running ls or dir in Powershell:

Directory listing is only the tip of the iceberg. Let's find out what else you can do!

Moving and copying files

Let's say you've found a file using the list command you learned about and you want to move it. You can do that too!

To move a file, use the mv command:

If you pass in the fully qualified path to the file, you can move it from anywhere:

Sometimes, you don't want to move the original. You want to copy it instead. You can copy a file with the cp command:

Note that cp and mv use the same syntax. Handy!

Creating and Deleting files

You can create or delete a file on the command line as well.

Linux and Mac both use touch to create an empty file. Windows has to be different. On Windows, you would use New-Item in Powershell.

Let's look at an example for Linux or Mac:

The command above creates a file in your current directory and names it my_file_name.txt. You can also pass in a fully qualified path to have touch create the file in some other location, as long as it's valid.

On Windows, you can simply replace touch with this command-let:

Now that you know how to create a file, it's time to learn about destroying them! In this case, Windows, Mac, and Linux all use the same command: rm.

Here's an example:

This command will delete the file if it exists. You'll receive an error message if you try to delete something that isn't there!

Creating and Deleting directories

All the operating systems provide ways to create and delete directories. Fortunately, the commands are almost the same across all the major platforms.

To create a directory, use mkdir:

The command you use will change if you need to create a nested set of directories (i.e., a directory tree). On Windows, you can use mkdir to create the directory tree the same way you create a single directory.

On the other hand, if you are using Linux or Mac, then you'll need to add a -p flag:

The -p tells mkdir to create all the directories in the directory tree.

Want to remove an empty directory? Then you can use the same command across all platforms:

If you want to remove a directory tree, you'll need to do something different. In Windows PowerShell, you can use the following command:

On Mac and Linux, the usual command is:

Be careful with these deletion commands. If you point them at the wrong folder on your machine, you will mess up your operating system. Fortunately, most operating systems won't let you delete important files unless you're running as administrator or root, but be cautious regardless.

Viewing files

The command line allows you to view text files too. You won't want to open a PDF or a Microsoft Word document here, though. Binary files aren't human-readable, so even if you did open them, they would be incomprehensible!

However, the command line provides a tool called cat, allowing you to view any text file, whether it's a log, a Python file, or any other text-based file.

Here's an example that will display what's in a Rust file in one of my projects:

If the file is longer than can be shown on a single screen, it will whip on by, and you'll have to scroll back up to read through it.

For large files, you can use more instead:

The more command will show a page's worth of data. You can then scroll down one line at a time using the Enter or Return key. For the most control, you should use the less command, but that one isn't available on Windows. You would need to install a tool for that or use Linux on Windows.

All of these commands are available on Mac and Linux.

Editing files with vim

Linux and Mac come with command-line text editors built-in. The most common ones you'll find pre-installed are:

• vi

• vim

• nano

These command-line editors allow you to create or edit text files in your command line.

Windows does not come with a command-line text editor, but you can install them. For example, you can install a Windows-compatible version of Vim too.

To launch vim, you can type vim and it will launch an empty editor window in your command line where you can type or copy and paste your text into.

If you have a pre-existing file, do this instead:

This time the editor launches with your file open inside of it. Hit the i button to write in the file. The i button turns on INSERT mode.

When you're done editing, hit the ESC key to enter NORMAL mode and then hit the colon key (:). Next, type wq! or x! to save your changes and exit.

The colon tells vim that you want to do some type of command while the wq means "write and quit".

If you don't want to save your changes, do q! instead.

There are tons of other built-in commands. You should Google for a vim cheatsheet and find one that you like.

SSH into machines

Secure Socket Shell or SSH is a popular command line tool to connect to a remote machine. You will almost always use [OpenSSH] on Linux and Mac. Windows 10 and 11 come with an SSH client built-in that is based on OpenSSH. However, you may need to enable the client.

Open up your command line tool of choice and get your credentials ready for one of your servers.

Let's pretend that you have a server named sample.ssh.com. Here's how you could use SSH to login to it using your default credentials:

Pretty simple, right? However, you'll often want to specify which user to login as.

That's easy, too:

Here you are logging in as the user "john".

You can also send along some of the commands you've learned about in this tutorial and run them on the remote machine.

For example, you might want to log in as "john" and list the contents of a directory. To do that, you could run the following:

If you don't want to enter your credentials every time you log in to your server, consider creating an SSH keychain.

Here's an example using ssh-keygen:

This will create two files, one with a .pub extension and one without an extension but with the same name in ~.ssh. You'll need to copy the public key over to your server to use your SSH keys.

Just copy the contents of the .pub file over to your remote machine and put it in ~/.ssh/authorized_keys and you should be good to go.

Move remote files back and forth with SCP

You'll find that you often have files on your local machine that you want to copy to your server. Yes, there's a command-line tool for that tool

It's called scp (secure copy) and it's available on all the operating systems.

Let's see scp in action. To copy a file from your machine to a remote machine, you would do something like this:

This command will copy the specified file, login to the remote machine, and place that file in the /remote/directory location.

If you'd rather copy a file from the remote machine to your local computer, you would use this command:

Note the period at the end. That tells scp to copy the remote file to the current location on your command line (remember pwd?). You can provide a full path instead of a period.

Closing Thoughts

Well, I could go on but I think you’ve probably had your fill. I don’t think I can overstate the importance of being able to work on the command line in a variety of environments, with the ability to do basic commands.

You know what’s hard? Being a new engineer, or onboarding a new engineer onto your team, you give a them a simple task … ssh into this AWS EC2 box and do x,y,z. You get blank stare.

They don’t know how to SSH, setup keys, or even move around and use basic commands. Not good.

The reality is that the command is easy. Unless you’ve never done it much.

Work in your IDE all day long? Consider switching to vim for a week or so, just to freshen up your skills. Again, it’s not like you have to memorize every command line operation on Windows and Linux … it’s more about being comfortable on the command line so you can do what you need to do when the need arises.