Seal of Approval Still Missing

Google then asks the user logged in with their Google account whether they want to grant the app read access to their Google Drive data. Since the Go program is a homemade app and does not yet bear the Google seal of approval, the dialog in Figure 7 warns the user against granting access. Brave readers of this column will want to press Advanced and grant the Go program read access to their Google Drive anyway on the following page.

Figure 7: Google warns you against using the unverified app, but Advanced lets you proceed at your own risk.

After confirming again, the Google OAuth 2 flow finally outputs a hex code (Figure 8), which you need to copy to the standard input of the Go program that has been waiting at the command line. When you type Enter, the program devours the code and continues to run. It contacts the Google server with the code and receives an access and refresh token from the server. Listing 1 then bundles these credentials into a local JSON file, which it stores as token.json in the same directory.

Figure 8: After further confirmation, Google creates an access token, which the app can use to access the drive.

Now that it has persistent credentials, Listing 1 no longer sends the user through the OAuth 2 flowwhen next called, but can use the access token in the JSON file to read the user's data on Google Drive. It is important to protect this JSON file against unauthorized access: Anyone in possession of the token can gain access to your Google Drive data. However, write access is not possible, because the scope was previously defined as read-only during the OAuth 2 flow.

Hop, Skip, and Google

Listing 2 implements the oauth2Client function to let the program collect and manage the token when run for the first time. When done, it returns an HTTP client to the main program, which handles user authentication under the hood when communicating with the Google Drive web server.

01 package main
02
03 import (
04   "encoding/json"
05   "fmt"
06   "golang.org/x/net/context"
07   "golang.org/x/oauth2"
08   "log"
09   "net/http"
10   "os"
11 )
12
13 func oauth2Client(config *oauth2.Config) *http.Client {
14   tokFile := "token.json"
15   tok, err := readCachedToken(tokFile)
16   if err != nil {
17     tok = fetchAccessToken(config)
18     cacheToken(tokFile, tok)
19   }
20   return config.Client(context.Background(), tok)
21 }
22
23 func fetchAccessToken(config *oauth2.Config) *oauth2.Token {
24   url := config.AuthCodeURL("state-token", oauth2.AccessTypeOffline)
25   fmt.Printf("Point browser to %v, follow the flow and then paste "+
26     "the code here.\n", url)
27
28   var authCode string
29   if _, err := fmt.Scan(&authCode); err != nil {
30     log.Fatalf("Error reading auth code %v", err)
31   }
32
33   tok, err := config.Exchange(context.TODO(), authCode)
34   if err != nil {
35     log.Fatalf("Error getting access token: %v", err)
36   }
37   return tok
38 }
39
40 func readCachedToken(file string) (*oauth2.Token, error) {
41   f, err := os.Open(file)
42   if err != nil {
43     return nil, err
44   }
45   defer f.Close()
46   tok := &oauth2.Token{}
47   err = json.NewDecoder(f).Decode(tok)
48   return tok, err
49 }
50
51 func cacheToken(path string, token *oauth2.Token) {
52   fmt.Printf("Saving credential file to: %s\n", path)
53   f, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0600)
54   if err != nil {
55     log.Fatalf("Can't write token to %s: %v", path, err)
56   }
57   defer f.Close()
58   json.NewEncoder(f).Encode(token)
59 }

When the program is first called, there is no token in the token.json file yet, so calling readCachedToken() in line 15 returns an error. This is then remedied by calling the fetchAccessToken() function in line 17. Starting in line 23, the code prints out a Google URL, which you need to paste into your web browser; from there, you then go through the OAuth 2 flow.

At the end, the flow shows a hex code that the user then copies into the main program waiting at the command line. Line 29 grabs the code from the standard input; line 33 exchanges it on the Google server for an access token, which line 18 stores in JSON format on the local filesystem. On subsequent invocations of the program, the code instructs oauth2Client() to retrieve the token from the local file without the previously necessary rigamarole. No more extra Google hops needed at this point.

The json package from the Go core collection makes reading and writing token data a breeze. Juggling the tokens to gain access to a protected resource isn't as convenient; someone might want to sit down and provide a standardized token handling package on GitHub for everyone else to use.

The pickNGet() function in Listing 3 sends the search query to Google Drive. The service stores files in a freely definable hierarchy of folders. However, users often simply search for file content or names, and the search engine giant returns a unique ID for matching files.

01 package main
02
03 import (
04   "bufio"
05   "fmt"
06   pb "github.com/schollz/progressbar/v3"
07   "google.golang.org/api/drive/v3"
08   "log"
09   "os"
10   "strings"
11 )
12
13 func pickNGet(srv *drive.Service, query string) error {
14   q := fmt.Sprintf("name contains '%s'", query)
15   r, err := srv.Files.List().Q(q).PageSize(100).
16     Fields("nextPageToken, files(id, name, size)").Do()
17   if err != nil {
18     log.Fatalf("Error retrieving files: %v", err)
19   }
20
21   if len(r.Files) == 0 {
22     fmt.Println("No files found.")
23     return nil
24   }
25
26   reader := bufio.NewReader(os.Stdin)
27
28   for _, file := range r.Files {
29     fmt.Printf("Download %s (y/[n])? ", file.Name)
30     text, _ := reader.ReadString('\n')
31     if !strings.Contains(text, "y") {
32       continue
33     }
34     bar := pb.DefaultBytes(file.Size, "downloading")
35
36     fmt.Printf("Downloading %s (%d) ...\n", file.Name, file.Size)
37     dwn, err := srv.Files.Get(file.Id).Download()
38     if err != nil {
39       log.Fatal("Unable to get download link %v", err)
40     }
41     defer dwn.Body.Close()
42
43     reader := bufio.NewReader(dwn.Body)
44     err = download(reader, file.Name, bar)
45     if err != nil {
46       log.Fatal("Download of %s failed: %v", file.Name, err)
47     }
48   }
49
50   return nil
51 }

My file names are usually unique, which is why line 14 in Listing 3 uses the search query name contains x. If you prefer to let Google search for text chunks in the content, just replace the string in line 14 with fullText contains x. Further search queries are explained in the API document [4].

Machine-Generated SDK

It turns out that the Go SDK for the Google API provided by Google is simply a machine-generated wrapper around the web API.

The API endpoint for listing files is addressed by srv.Files.List() (line 15). The concatenated calls to Q(q).PageSize(100) append the search query and set the maximum number of matches delivered for each request to 100. If so desired, the user can pick up the next batch of matches on the next call, but Listing 3 does not do this, because a command-line client would be unsuitable for processing more than 100 matches anyway.

The concatenated Fields() function limits the fields returned per match to the unique document ID, the file name, and the size of the file in bytes. pickNGet() can therefore offer the user a compact list for selection. Pressing Y starts the download.

The for loop starting in line 28 iterates over all the matches in the r.Files array and prompts the user on os.Stdin at each pass to press Y if they want to download the current match to their local drive. To allow this to happen, the ReadString() function in line 30 waits for keyboard input from the user, which is sent after the Enter key gets pressed. If the user did not type y (e.g. by accepting the default "n"), line 32 uses continue to go to the next round, asking the user what they want to do with the next match.

If the user wants to download a matching file, line 34 sets the progress bar to zero percent and the maximum length to the size of the file. The Google Drive call srv.Files.Get() in line 37 selects the desired file by referencing its ID and calls Download() to initiate the download. In this case, the Google Drive server sends back a download URL, which the client docks onto, and the download process starts via HTTPS.

Line 43 defines a buffered reader from the standard bufio package for the incoming data stream. In line 44, the program passes the reader to the download() function from Listing 4, together with a reference to the progress bar bar and the file name to store it under later locally.

01 package main
02
03 import (
04   "bufio"
05   pb "github.com/schollz/progressbar/v3"
06   "io"
07   "os"
08 )
09
10 func download(r io.Reader, lpath string, bar *pb.ProgressBar) error {
11   outf, err := os.OpenFile(lpath, os.O_WRONLY|os.O_CREATE, 0644)
12   if err != nil {
13     return err
14   }
15   writer := bufio.NewWriter(outf)
16   defer outf.Close()
17
18   total := 0
19   data := make([]byte, 1024*1024)
20
21   for {
22     count, rerr := r.Read(data)
23     if rerr != io.EOF && rerr != nil {
24       return err
25     }
26     total += count
27     bar.Add(count)
28     realdata = data[:count]
29
30     _, werr := writer.Write(realdata)
31     if werr != nil {
32       return werr
33     }
34
35     if rerr == io.EOF {
36       break
37     }
38   }
39   writer.Flush()
40   return nil
41 }