Intelligently Arranged

McFly upgrades Bash with artificial intelligence

Article from Issue 228/2019
Author(s):

When it comes to working at the command line, using Bash history effectively can save you time. McFly extends the Bash history's features and helps you find past commands more quickly.

The most popular Linux shell by far is the GNU Bourne-Again Shell (Bash for short). One of its many outstanding features is its integrated command history, where Bash saves previously run command lines so that you can recall them later on without a lot of typing.

Listing 1 shows the output you'll see if you enter the history command at the terminal prompt. Each line begins with a line number in ascending order. The last command line to have been entered is at the end of the history list. You can use the arrow keys to scroll backward or forward one step in the history in the terminal. Pressing the Enter key copies the displayed line and runs it again.

Listing 1

Running the history Command

§§nonumer
01 $ history
02 [...]
03   446  nano .config/GIMP/2.10/scripts/image-subdivide.scm
04   447  nano .config/GIMP/2.10/scripts/slice-and-join.scm
05 [...]
06   901  yay -Syu
07   902  trizen -Syu
08   903  yay -Syu
09 [...]
10  1044  ++
11  1045  ++ /tmp/
12  1046  mv /tmp/*png . -v
13  1047  ++
14  1048  mv /tmp/*png . -v
15  1049  ++
16  1050  mv /tmp/linify.png .
17 $ history | grep slice
18   447  nano .config/GIMP/2.10/scripts/slice-and-join.scm
19  1051  history | grep slice

Bash supports two approaches to searching for specific strings directly in the history: Ctrl+R searches backwards from the current cursor position; Ctrl+S searches forwards from the current cursor position. Ctrl+S only makes sense if you have already advanced further back in the history and are no longer at the end. In addition, entering Ctrl+S only works in an appropriately configured terminal; normally, this keyboard shortcut blocks the terminal.

To indicate reverse searching in the history, Bash uses Ctrl+R with a special prompt:

(reverse-i-search)`':

i-search stands for an incremental search, which refines or enhances the search by entering a new character. The function finds the last line in the history that matches the search criteria; each time Ctrl+R is pressed, the search jumps one match further back into the past.

In practice, this form of search in the history proves to be as simple as it is effective. A few cleverly selected entries are usually sufficient to find the desired line. However, you need to know what you are looking for – and, above all, you need to know a good pattern that will quickly take you to the desired command. The pattern can be anywhere on the command line, so you don't have to type it from the beginning. (See also the "Search Patterns" box.)

Search Patterns

You can combine history with grep to find all command lines in the history where a given search pattern occurs (Listing 1, line 17). Using agrep -B instead of grep gives you a fuzzy search.

For instance, to retrieve the command from Listing 1 (line 16)

mv /tmp/linify.png .

you would just need to type Ctrl+R I F Y. Command lines such as trizen -Syu or yay -Syu can be quickly found with Ctrl+R, Shift+S, or Ctrl+R - Shift+S.

As shown in Listing 1, the history usually contains command lines that are very similar to each other and can be described well with patterns, but cannot be easily distinguished. A remedy for this requires a new approach.

Modern Times

Bash history lacks the ability to search in a context-based way. For example, on many modern systems based on systemd, /var/log/journal/ contains large numbers of logfiles. Deleting outdated logfiles can often free up a large amount of storage space.

To see if, and how many, old logfiles are stored there, change to the directory. The du -sh command then determines the space used by the existing files. find -mtime 7 lets you search for all files that have not been changed in over one week; you can then delete them using rm [...].

You could now write a script or – preferably – a shell function for this task in a few moments, but the problem remains the same: You have to actively develop a solution for every situation. Wouldn't you prefer that the shell guessed what you were going to do based on the directory and offered you a corresponding action?

McFly [1] does this by reading the previous history, analyzing it in a synthetic neural network, and attempting to recognize correlations in the command lines by machine learning. These relationships then serve as the basis for the suggestions offered by the shell's search functions.

In contrast to the neural network's quite complicated structure, McFly's last step is very easy to understand: The previous-history function normally mapped to Ctrl+R is replaced by the new mcfly search function.

The following parameters serve as the basis for the neuronal evaluation of the history and the McFly suggestions for the search function, in the specified order:

  • The current directory
  • The previously entered command lines
  • The frequency and order of the call
  • Whether a command line was previously found by McFly
  • Whether the command ran without errors

After setting up McFly (see the box "Installing McFly"), the shell behavior changes. Ctrl+R calls the tool directly; the titlebar changes to reflect this, and the new key bindings become active (Figure 1).

Figure 1: McFly extends the search for commands in the shell's command history.

Installing McFly

McFly [1] is not yet found in popular distribution's package sources. Only Arch Linux offers the program in the Arch User Repository (AUR). On Arch, you can install the program with an AUR helper, for example, using

yay -S mcfly

This will add a whole series of additional packages with a total volume of several hundred megabytes due to the dependencies.

In Ubuntu, and many other distributions, you need to install the Brew program, which was actually developed as a package manager for Mac OS X and later ported to Linux. Depending on the distribution, the package in question goes by the name of brew, linuxbrew, homebrew, homebrew-bundle, or – like in Ubuntu – linuxbrew-wrapper. Using the commands from the first six lines of Listing 2, you import Brew in Ubuntu and configure the program. Now set up the tap with Brew (line 7), which retrieves several hundred megabytes of additional software. Following this, install by typing the command on line 8, which you then have to initialize (line 9).

Finally, expand your $PATH to include the executable file's folder. To do this, transfer the code in Listing 3 to the ~/.bashrc file. After a restart, you will be able to call the mcfly command from the terminal.

Everything went smoothly on my Ubuntu test system, but some things can go wrong with this procedure: The /home/linuxbrew/ directory and its .linuxbrew/ subfolder may not be created automatically. When running brew install mcfly, you will then see an error message. In this case, you need to create the directory yourself or create a symbolic link to it. (See also the "Bug Report" box.)

The McFly developer has also posted ready-to-go 32- and 64-bit binaries of the program on GitHub [2].

Bug Report

If McFly crashes on your system with a cryptic message similar to:

Unable to add cmd_tpl to commands: SqliteFailure([...]])',   src/libcore/result.rs:1009:5

you are struggling with a bug that has affected many other users [3]. Commands with special characters will disrupt the program. In the test this happened when I accidentally entered the command:

^@^@^[...]^@^@sudo

Until the developers fix this problem, you'll have to make do with simply deleting the line from ~/.bash_history with an editor.

Listing 2

Configuring and Installing Brew

§§nonumer
01 $ sudo apt install linuxbrew-wrapper
02 $ brew
03 $ echo 'PATH="/home/linuxbrew/.linuxbrew/bin:$PATH"' >> ~/.profile
04 $ echo 'MANPATH="/home/linuxbrew/.linuxbrew/share/man:$MANPATH"' >> ~/.profile
05 $ echo 'INFOPATH="/home/linuxbrew/.linuxbrew/share/info:$INFOPATH"' >> ~/.profile
06 $ PATH="/home/linuxbrew/.linuxbrew/bin:$PATH"
07 $ brew tap cantino/mcfly https://github.com/cantino/mcfly
08 $ brew install mcfly
09 $ source "$(brew --prefix)/opt/mcfly/mcfly.bash"

Listing 3

Expanding $PATH

§§nonumer
if [ -f $(brew --prefix)/opt/mcfly/mcfly.bash ]; then
. $(brew --prefix)/opt/mcfly/mcfly.bash
fi

While initializing, McFly reads the existing history and evaluates it, which takes some time. You can press Esc to exit this mode and return directly to the prompt. Then use the up and down arrows to navigate the recommendations offered by McFly. When you enter a search term, McFly retrieves matching hits from the history.

Press Enter to accept the currently highlighted line from the history and execute the command immediately. The Tab key, on the other hand, transfers the highlighted history line to the input prompt without executing the command immediately. You can then edit the command and press Enter to execute it. Press F2 to delete individual lines from the list of suggestions after a prompt.

Artificial Intelligence

The interesting thing about McFly is its underlying artificial intelligence, the neural network. What used to require a huge amount of power and entire data centers, can now (with some limitations) even be performed by a Raspberry Pi. Today, such small neural networks are useful for all kinds of tasks, from image processing to character and pattern recognition.

These systems' missing logic often proves to be problematic. Neural networks learn coherencies without it being possible to clearly understand how they do it in each case. With McFly, however, you can still see the learning behavior, or at least the rudiments of it. If you call a command from the list of suggestions, McFly moves the command further up the list the next time it is called. However, it is less clear which commands appear (and in which order) initially in the proposal list. The rules used already seem so complex that some effort is needed to predict their impact.

You actually encounter a very similar system every day when searching on the Internet. Here, too, it is often impossible to identify how, where, and why certain results appear in the hit list while others do not. This problem is attributable to the principle of neural networks and is not completely harmless. For this reason, IT researchers repeatedly urge cautious use of the technology [4].

Conclusions

In addition to autocompletion, the history is an important shell feature for interactive use. McFly adds value to the history command. Fortunately, the developer implemented the tool as an independent program instead of adding it to the already complex Bash or developing a plugin. This minimizes the side effects and allows the tool to be tried out quickly and safely.

As far as McFly's AI performance is concerned, a typical problem of self-learning systems applies here. It is virtually impossible to make reliable statements about the software, since using the software changes the system itself at run time. In my test, McFly delivered very good results; most of the command lines I searched for were found in the first list of suggestions.

If you are just looking for an upgrade of history, there are simpler alternatives in the open source universe, such as hstr [5] and cdhist [6], which do without neural networks and take up far less disk space.

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

comments powered by Disqus
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters

Support Our Work

Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.

Learn More

News