Beyond the Web

Anyone who has used the World Wide Web (WWW) lately knows that something bad is happening to it. It does not resemble the WWW of the early years, with enthusiastic amateurs freely sharing ideas and information. These things still exist, and the web is still an indispensable medium connecting the world. But the web experience is now encumbered with advertising, invasions of privacy in the form of pervasive tracking, enormous file sizes, CPU straining JavaScript, the danger of exploits, and door slams asking you to subscribe to a newsletter before viewing a site.

This unpleasant environment has led to a backlash. There are now some communities of developers and computer users who still desire a connected information system, but who seek a refuge from the noise, danger, and increasingly resource-hungry WWW. They feel that web technology does too much, and that since it makes various forms of abuse too easy, no lasting reform is possible.

The solution is to use or create a separate protocol that is simply not capable of supporting the technologies that enable advertising networks, user fingerprinting, or the myriad of other things that exploit users rather than helping them. This small movement has approached the problem from two directions that in practice are often merged: the revival of the Gopher protocol and the creation of a new protocol called Gemini.

A Little History

Looking back at computer networks before the emergence of the Internet in the late 1980s, and even the Internet itself in its early years, there were only a few applications of interest to most users: email, logging on to timesharing systems, and the transfer of documents through the file transfer protocol (FTP).

FTP began as a command-line program, with graphical clients eventually arriving for every platform. To get a file, you had to know its specific address on the FTP server where it resided. Eventually, it got a bit easier to find things with the arrival in 1990 of Archie, the first public search engine on the Internet, which indexed FTP files.

A command in the terminal would download the file to your machine. You could navigate through directories if they were exposed on the server, but you could not navigate outside the server, discover content, or interact with the files before downloading them. FTP was essentially a networked version of the cp command. It was, for years, the only way to share content, aside from passing around tapes and floppy disks; FTP was the way the first versions of the Linux kernel were distributed [1].

Shortly after the Internet's predecessors evolved into the modern global network, other means of sharing content emerged that offered more than FTP's spare file copying. The three most significant participants in this information landscape were the Wide Area Information Server (WAIS), Gopher, and the WWW.

WAIS was a sophisticated software system [2] that translated the user's natural language query into search commands for databases. From a networked computer, you could search the contents of databases all over the world and retrieve a variety of documents. There were textual and graphical clients for major operating systems, with varying degrees of polish.

WAIS was developed by Thinking Machines Corporation [3], a visionary company that created the first massively parallel supercomputers. WAIS is completely dead now. Even if a client could be found that ran under the current version of any OS, there is no content and no servers to interact with.

Gopher, on the other hand, is still being used. As Cameron Kaiser points out [4], "Gopher takes the strict hierarchical nature of a file tree or FTP and turns it into a friendlier format that still gives the fast and predictable responses that they would get by simply browsing their hard drive."

And that serves to explain the nature of Gopher: Like FTP, it was a way to display directories of files that could be downloaded, or in the case of text files, displayed directly by the client.

Gopher clients went beyond simply downloading files, because they had some notion of file type, and could respond appropriately, in a limited way. While FTP servers only presented lists of files and directories local to that server, Gophermasters could create directories that appeared as pages of links and text. The links could lead to other servers, creating the first easily accessible global network of information. These directories were not hypertext documents, however; they contained isolated links and descriptions of content.

Around 1990-91, the first WWW browsers appeared on computers all over the world, and in another year or two the web had surpassed FTP, WAIS, and Gopher as the medium for finding information and entertainment through the Internet.

At that time, I was working in a government laboratory, enjoying the luxury of the kind of fast Internet connection that most of the developed world takes for granted now. I had several FTP clients and at least one Gopher and WAIS client on my laboratory computer. The reasons for the rapid ascendency of the web are not mysterious to me, because I distinctly remember watching it happen in real time.

As soon as people saw hypertext, it captured their imaginations. That is the magic of the web: to click on a word in a document and be taken somewhere else for further enlightenment. And these were not merely text documents; they could be styled to look like magazines. When inline images and other media appeared, it sealed the deal. In the lab, we started making our own pages to share information with other departments. This was the original purpose of the web, for scientists to share research with each other, and in our lab it was a reality.

Everyone I knew with a computer, including laypeople who had never heard of FTP or WAIS, was soon browsing the web. The WWW had replaced, more or less, the existing protocols and became the standard way to share information.

In 1992-93, as the WWW was just beginning to surpass WAIS and Gopher in network traffic, the WWW's inventor, Tim Berners-Lee, wrote a note explaining the differences among the three systems [5]. Berners-Lee pointed out the advantages of hypertext over the other methods of distributing information and emphasized an additional crucial virtue of the WWW – it could be used as an interface to Gopher or WAIS and was therefore a superset containing the other two protocols. Thus the web rendered Gopher and WAIS obsolete in a technical sense in addition to replacing them as a preference on the part of users. Fast-forward to today, and even Tim Berners-Lee thinks the web has gone too far. Berners-Lee is one of many experts who have spoken out against the perverse incentives and lack of privacy that characterize today's web [6], and he has even launched an initiative to try to fix it [7].

The first murmurings of a revival of old protocols as reaction to the excesses of the web mentioned Gopher [8] as a plausible candidate for a refuge from the storm. (See the box entitled "URLs and Protocols.") Some advocates for the Gopher revival speak of a "community outside the chaos of the modern Internet."

This is a relatively easy way to build an alternative Internet community, because the Gopher universe never completely disappeared. There are clients that still work, or can be updated to work, on current operating systems.

Now Gopher is growing again, specifically because it is not the web. The number of Gopher servers is again increasing, a development that would have seemed highly unlikely as recently as five years ago. Because it does not support any kind of scripting, much of the abuse of the modern web can not occur. The lack of embedded media removes the possibility of most forms of advertising or other irritations. And its very retro nature gives the use of Gopher an inherent cool quality.

Enter Gemini

Project Gemini [9] has no interest in replacing either the web or Gopher but seeks to live alongside them as another option for disseminating and consuming content. It carefully positions itself between Gopher and the WWW in its complexity and capabilities. The creators of Gemini realized that Gopher lacks certain features that are desirable for dissemination of even simple documents over the web, such as encryption and the inline inclusion of images along with text. But they deliberately excluded from the specification [10] web features such as scripting and complex document models that are more likely to enable abuse of the information consumer. To this end the protocol is designed to be difficult to extend, so that its protections against abuse will endure. One example of Gemini's emphasis on privacy is that connections are required to be encrypted. Another is that the protocol has no provision for User-Agent or Referer headers, and it is specifically designed not to be extensible to include them.

It has several advantages over the older Gopher protocol, chiefly in the use of non-ASCII character sets, using MIME types to handle non-text content, redirects, and virtual hosting; the requirement to use TLS encryption; and links within documents.

The type of link implemented by Gemini is not freely mixed in with the document as in HTML; instead it sits on a separate line. The Gemini designers see this as an advantage, as it helps in the organization and outlining of documents. We'll take a closer look at the native Gemini file format in Listing 1, when we try out one of the Gemini browsers.

    # GUS - Gemini Universal Search
    => / Home
    => /search Search GUS
    => /backlinks Query backlinks
    ## Search
    => /v/search/1?cornea Enter verbose mode
    "cornea"
    => gemini://gemini.circumlunar.space/
    users/lel/blog/anp/q4/ gemini.
    circumlunar.space/users/lel/blog/anp/q4/
    (text/gemini, 14K)
    => gemini://gemini.circumlunar.space/
    ~lel/blog/anp/q4/ gemini.circumlunar.
    space/~lel/blog/anp/q4/
    (text/gemini, 14K)
    Page 1 of 1 (2 results)
    > "Nobody ever figures out what life is
    all about, and it doesn't matter. Explore
    the world. Nearly everything is really
    interesting if you go into it deeply
    enough." --- Richard Feynman
    => /add-seed See any missing results?
    Let GUS know your Gemini URL exists.
    Index updated on: 2020-12-06

This article will not go into setting up content servers, but those interested in serving Gemini content will find some free, hosted avenues for doing so at the Gemini FAQ.

The Gemini space is still small, but growing rapidly. For historical interest, the project exposes a list of the first 50 known Gemini hosts at (naturally) a Gemini address. Up-to-date information, including an automatically generated list of Gemini hosts, is available at the Gemini Universal Search (GUS) search engine.

Still, there are, according to GUS, about 200,000 Gemini pages served from 422 domains, compared with over a billion pages on the World Wide Web. Consequently, Gemini is not a place to find information or do research, nor even to browse interesting content in depth. Today it is still a place to explore a new idea in information sharing and perhaps to enjoy wandering through a small but enthusiastic networked community while knowing that your privacy and security are being respected and without the irritations of the modern web. Gemini fills a niche, is well executed, and has been thoughtfully designed. Its limitations are very much deliberate choices. I believe it will continue to grow and stands a reasonable chance of becoming a real second place to find and share information on the Internet.

The new Gemini protocol, along with the revival of Gopher, and the attitudes and community that go with them, are sometimes called the "small Internet," in contrast to the WWW, which is obviously anything but small. This nascent movement also embraces the venerable Unix Finger protocol. The finger command in Unix machines returns a small amount of text indicating the status of a user or a machine. Despite its simplicity, it has been used by creative individuals as a minimalist communications medium.

Part of the appeal of Gemini is the same thing that makes Gopher attractive to some people: The very fact that it is still small and new makes browsing the available content seem like exploring a new universe, somewhat like the feeling of exploring the web in the early years.

Linux Gopher Clients

VF-1's motto [11] is "High speed, low drag." It is a basic terminal client, similar in operation to the command-line FTP clients of olden times. VF-1 is written in Python and can be installed with:

<C>pip3 install VF-1<C>

After the user types vf1 in a terminal, a brief welcome and a simple prompt appear. If the user guesses correctly and types help, he or she will be rewarded with a list of the available commands (Figure 1). One of these is veronica, which submits a search term to the Gopher search engine of that name. Navigating to results is accomplished by typing the number displayed next to the file name and hitting return, after which the file contents are dumped to the screen. Alternatively, the user can view the file in the less pager or save it to disk.

Figure 1: A list of the available commands for the VF-1 Gopher client.

If you know where you're going, the go command will take you right there. In Figure 2, I've used this command to navigate to a Gopher mirror of the Hacker News front page. Gophermasters often adorn their directory pages with ASCII art to personalize them in the absence of inline images.

Figure 2: The Hacker News Gopher mirror in VF-1.

The next step up in convenience from a pure command-line program like VF-1 is a curses-style interface. These programs are still lean and fast, and they run in the terminal, but through random access to the screen area they allow a richer mode of interaction and display.

Phetch is a Gopher client in this mode written by Chris West. Binaries and source are available on GitHub [12]. After typing phetch in the terminal, the program presents the startup screen shown in Figure 3. Navigation is accomplished through the keyboard, with links and text being distinguished by color. As this is Gopher, text is all there is, so the terminal remains a reasonable environment for consuming the content, as shown in Figure 4. Phetch does what it was designed to do with an intuitive, efficient interface. I found it a pleasant way to explore the Gophersphere and encountered no bugs or rough edges of consequence.

Figure 3: The startup screen for phetch, a Gopher client that runs in the terminal.

Figure 4: A page from the Gopherpedia, the Gopher interface to Wikipedia, shown in phetch.

There appear to be no existing graphical Gopher clients for Linux that are not broken or too ridden with bugs to be useful.

FORG [13] is a graphical Gopher client written in Python. The GUI uses tkinter. I was able to start FORG by executing the main Python file after downloading the repository from GitHub, but first I had to use pip to install a dependency, Pmw. When I tried to use it, I found that FORG displayed its configured home page with no problem, but any link I followed resulted in a blank page and error messages written to the console. I hope that this program gets these issues resolved, because it has some valuable features.

Another GUI client too buggy to use in my testing on Linux, but apparently popular on macOS, is called Little Gopher Client.

Finally, some web browsers, including text-mode browsers such as Lynx and ELinks, have support for the Gopher protocol, so it's likely you can browse Gopherspace without downloading any new software. There is also a growing number of Gopher extensions for browsers that don't come with support out of the box.