Subscribe to the Ardan Labs Insider

You’ll get our FREE Video Series & special offers on upcoming training events along with notifications on our latest blog posts.

Included in your subscription
  • Access to our free video previews
  • Updates on our latest blog posts
  • Discounts on upcoming events

Valid email required.

Submit failed. Try again or message us directly at hello@ardanlabs.com.

Thank You for Subscribing

Check your email for confirmation.

Data Science in Go: How Much To Tip

Author image

Miki Tebeka

The Question

When you work on data science problems, you always start with a question you’re trying to answer. This question will affect the data you pick, your exploration process, and how you interpret the results.

The question for this article is: How much (in percentage) should you tip your taxi driver?

To answer the question, we’ll use a portion of the NYC Taxi dataset. The data file we’ll be using is taxi-01-2020-sample.csv.bz2

Note: CSV is a horrible, horrible format. There’s no standard, no schema and everything is unmarshalled to text (unlike JSON where you have types). If you can, pick another format. My preferred data storage format is SQLite.

Exploration Code

We’re searching for the answer to our question, we’ll be focused on productivity. If at some point this code will get to production, go ahead and refactor it.

To simplify working with the input, we’ll pass the input file in the standard input. We’ll have several stages of exploring the data and each will have a corresponding command line switch. In the main function we have the following line:

r := bzip2.NewReader(os.Stdin)

and then we’ll call each exploration step with r.

First Contact

Before you start working with data, it’s important to have a quick look at it to see that it matches your expectations. You should also check that the data fits in memory.

Listing 1: First Look

19 func firstLook(r io.Reader) error {
20     var numLines, numBytes int
21     s := bufio.NewScanner(r)
22     for s.Scan() {
23         if numLines < 5 {
24             fmt.Println(s.Text())
25         }
26         numBytes += len(s.Text())
27         numLines++
28     }
29 
30     if err := s.Err(); err != nil {
31         return err
32     }
33 
34     fmt.Printf("size: %.2fMB\n", float64(numBytes)/1_000_000)
35     fmt.Printf("lines: %d\n", numLines)
36     return nil
37 }

Listing 1 shows the first look on the data. On line 21, we create a bufio.Scanner to scan line by line. On lines 23-25, we print the first 5 lines from the file. On line 34, we print the file size and on line 35, we print the number of lines.

Listing 2: Running First Look

$ go run taxi.go -first_look < taxi-01-2020-sample.csv.bz2
VendorID,tpep_pickup_datetime,tpep_dropoff_datetime,passenger_count,trip_distance,RatecodeID,store_and_fwd_flag,PULocationID,DOLocationID,payment_type,fare_amount,extra,mta_tax,tip_amount,tolls_amount,improvement_surcharge,total_amount,congestion_surcharge
2,2003-01-01 00:07:17,2003-01-01 14:16:59,1.0,0.0,1.0,N,193,193,2.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
2,2008-12-31 23:35:00,2008-12-31 23:36:53,1.0,0.42,1.0,N,263,263,2.0,3.5,0.5,0.5,0.0,0.0,0.3,7.3,2.5
2,2009-01-01 00:06:19,2009-01-01 00:10:22,1.0,0.85,1.0,N,107,137,2.0,5.0,0.0,0.5,0.0,0.0,0.3,8.3,2.5
2,2009-01-01 00:48:28,2009-01-01 00:57:48,1.0,0.93,1.0,N,100,186,2.0,7.5,0.0,0.5,0.0,0.0,0.3,10.8,2.5
size: 101.68MB
lines: 1000001

Listing 2 shows how to run the first step. We run the program with go run. In the output we see the first 5 lines and the uncompressed file size and the number of lines.

The file is a CSV file and small enough to fit in memory. To calculate the tip percentage we need only two columns: tip_amount and total_amount. If you’re curious about the data schema, see here.

Loading the Data

Once the initial look aligns with your assumptions, you can load the data. We’re going to use github.com/jszwec/csvutil to parse the CSV and gonum to calculate some statistics.

Listing 3: Imports

14     "github.com/jszwec/csvutil"
15     "gonum.org/v1/gonum/floats"
16     "gonum.org/v1/gonum/stat"

Listing 3 shows the external imports in our code. On line 14, we import csvutil and on lines 15-16, we import floats and stat from gonum.

Listing 4: Load Data

62 type Row struct {
63     Tip   float64 `csv:"tip_amount"`
64     Total float64 `csv:"total_amount"`
65 }
66 
67 func loadData(r io.Reader) ([]float64, []float64, error) {
68     var tip, total []float64
69     dec, err := csvutil.NewDecoder(csv.NewReader(r))
70     if err != nil {
71         return nil, nil, err
72     }
73 
74     for {
75         var row Row
76         err := dec.Decode(&row)
77 
78         if err == io.EOF {
79             break
80         }
81 
82         if err != nil {
83             return nil, nil, err
84         }
85 
86         tip = append(tip, row.Tip)
87         total = append(total, row.Total)
88     }
89 
90     return tip, total, nil
91 }

Listing 4 shows how we load the data. On lines 62-65, we define a Row struct with the fields that interest us. On line 68, we define the tip and amount slices to hold the values from the tip_amount and total_amount fields in the CSV. On lines 74-88, we run a for loop to upload the data and finally on line 90, we return the data.

Listing 5: Statistics

39 func statistics(r io.Reader) error {
40     tip, total, err := loadData(r)
41     if err != nil {
42         return err
43     }
44 
45     fmt.Printf(
46         "tip: min=%.2f, mean=%.2f, max=%.2f\n",
47         floats.Min(tip),
48         stat.Mean(tip, nil),
49         floats.Max(tip),
50     )
51 
52     fmt.Printf(
53         "total: min=%.2f, mean=%.2f, max=%.2f\n",
54         floats.Min(total),
55         stat.Mean(total, nil),
56         floats.Max(total),
57     )
58 
59     return nil
60 }

Listing 5 shows the statistics step of our data exploration. On line 40, we load the date. On lines 45-50, we print the minimal, mean (average) and maximal values for the tip. On lines 52-57, we do the same for the total.

Listing 6: Running Statistics

$ go run taxi.go -stats < taxi-01-2020-sample.csv.bz2 
tip: min=-11.80, mean=2.21, max=333.50
total: min=-333.30, mean=18.47, max=4268.30

Listing 6 shows how to run the statistics step. We can see there are some bad values. Both minimum values are negative and the maximal value for the total amount is more than $4,000.

In any real life dataset you will have bad values, and you need to decide what to do with them. We’re going to take the easy approach and ignore them. We will filter out negative values. Also, since we’re not planning on taxi rides that cost more than $100, we’ll filter out rows with total_amount bigger than 100.

Tip

Listing 7: Load Filtered Data

114 func loadDataFiltered(r io.Reader) ([]float64, []float64, error) {
115     var tip, total []float64
116     dec, err := csvutil.NewDecoder(csv.NewReader(r))
117     if err != nil {
118         return nil, nil, err
119     }
120 
121     for {
122         var row Row
123 
124         err := dec.Decode(&row)
125         if err == io.EOF {
126             break
127         }
128 
129         if err != nil {
130             return nil, nil, err
131         }
132 
133         if row.Total <= 0 || row.Tip <= 0 || row.Total > 100 {
134             continue
135         }
136 
137         tip = append(tip, row.Tip)
138         total = append(total, row.Total)
139     }
140 
141     return tip, total, nil
142 }

Listing 7 shows loading of filtered data. The only difference from loadData are lines 133-135 where we filter out rows.

Now we can calculate the tip we want to pay. We’d like to be on the generous side so we’ll use the 75% quantile value. The 75% quantile (or percentile) is a number that 75% of the values are below it.

Listing 8: Desired Tip

93  func desiredTip(r io.Reader) error {
94      tip, total, err := loadDataFiltered(r)
95      if err != nil {
96          return err
97      }
98  
99      fmt.Printf("%d filtered values\n", len(tip))
100 
101     pct := make([]float64, len(tip))
102     for i, t := range tip {
103         pct[i] = t / (total[i] - t)
104     }
105 
106     // stat.Quantile required sorted values
107     sort.Float64s(pct)
108     q := 0.75
109     val := stat.Quantile(q, stat.Empirical, pct, nil)
110     fmt.Printf("%.2f quantile tip: %.2f\n", q, val)
111     return nil
112 }

Listing 8 shows the desiredTip function. On line 94, we load the filtered data. On line 99, we print how many filtered values we have to check so we don’t filter out too many rows. On lines 101-104, we create a slice of percentages. Finally on lines 107-110, we calculate the 75% percentile and on line 107, we print it.

Listing 9: Running Desired Tip

$ go run taxi.go -tip < taxi-01-2020-sample.csv.bz2 
716422 filtered values
0.75 quantile tip: 0.20

Listing 9 shows the tip step output. We see we filtered out about 30% of the rows. And finally, we see that the 75% quantile is 20%.

But wait! Maybe we should tip more on the weekend? Let’s have a look:

Listing 10: Loading Data With Time

145 func unmarshalTime(data []byte, t *time.Time) error {
146     var err error
147     *t, err = time.Parse("2006-01-02 15:04:05", string(data))
148     return err
149 }
150 
151 type TimeRow struct {
152     Tip   float64   `csv:"tip_amount"`
153     Total float64   `csv:"total_amount"`
154     Time  time.Time `csv:"tpep_pickup_datetime"`
155 }
156 
157 func loadDataWithTime(r io.Reader) ([]time.Time, []float64, []float64, error) {
158     var tip, total []float64
159     var times []time.Time
160     dec, err := csvutil.NewDecoder(csv.NewReader(r))
161     dec.Register(unmarshalTime)
162     if err != nil {
163         return nil, nil, nil, err
164     }
165 
166     for {
167         var row TimeRow
168 
169         err := dec.Decode(&row)
170         if err == io.EOF {
171             break
172         }
173 
174         if err != nil {
175             return nil, nil, nil, err
176         }
177 
178         if row.Total <= 0 || row.Tip <= 0 || row.Total > 100 {
179             continue
180         }
181 
182         tip = append(tip, row.Tip)
183         total = append(total, row.Total)
184         times = append(times, row.Time)
185     }
186 
187     return times, tip, total, nil
188 }

Listing 10 shows how to load the data with time. On lines 145-149, we write an unmarshalTime function to parse time from a []byte. On lines 151-155, we define TimeRow which is a row that contains a Time field. On line 159, we define the times slice and on line 160, we register unmarshalTime to handle time.Time fields. Finally on line 187, we return the times, tip and total.

Listing 11: Tip by Weekday

190 func weekdayTip(r io.Reader) error {
191     times, tip, total, err := loadDataWithTime(r)
192     if err != nil {
193         return err
194     }
195 
196     pct := make(map[time.Weekday][]float64)
197     for i, t := range tip {
198         wday := times[i].Weekday()
199         p := t / (total[i] - t)
200         pct[wday] = append(pct[wday], p)
201     }
202 
203     for wday := time.Sunday; wday < time.Saturday; wday += 1 {
204         // stat.Quantile required sorted values
205         p := pct[wday]
206         sort.Float64s(p)
207         q := 0.75
208         val := stat.Quantile(q, stat.Empirical, p, nil)
209         fmt.Printf("%-10s: %.2f quantile tip: %.2f (%6d samples)\n", wday, q, val, len(p))
210     }
211 
212     return nil
213 }

Listing 11 shows the “tip by weekday” calculation. On line 196, we use a map to hold percentages per weekday. On lines 197 to 201, we populate the percentages for each week day, this is the equivalent of a “GROUP BY” operation in a database. On lines 203-209, we go over each weekday, calculate the .75 quantile and print it out.

On line 209, we use the -10s% verb to have all weekdays take at least 10 characters and aligned to the line. This might seem like a trivial detail, but aligned output is much easier to compare for us humans - as you can see in the output below. We also align the number of samples for the same reason.

Listing 12: Running Weekday Tip

$ go run taxi.go -daily < taxi-01-2020-sample.csv.bz2
Sunday    : 0.75 quantile tip: 0.20 ( 77942 samples)
Monday    : 0.75 quantile tip: 0.20 ( 82561 samples)
Tuesday   : 0.75 quantile tip: 0.20 ( 97634 samples)
Wednesday : 0.75 quantile tip: 0.20 (118497 samples)
Thursday  : 0.75 quantile tip: 0.20 (125692 samples)
Friday    : 0.75 quantile tip: 0.20 (125743 samples)

Listing 12 shows how to run the daily step. We can see that there’s no difference in the tip percentage over the weekend.

Conclusion

A little curiosity and a little Go will get you pretty far in your data science journey. You don’t have to use deep learning, decision trees, support vector machines and other algorithms to get useful answers.

How are you using Go for data science? I’d love to hear, ping me at miki@ardanlabs.com.

Go Training

We have taught Go to thousands of developers all around the world since 2014. There is no other company that has been doing it longer and our material has proven to help jump start developers 6 to 12 months ahead of their knowledge of Go. We know what knowledge developers need in order to be productive and efficient when writing software in Go.

Our classes are perfect for both experienced and beginning engineers. We start every class from the beginning and get very detailed about the internals, mechanics, specification, guidelines, best practices and design philosophies. We cover a lot about "if performance matters" with a focus on mechanical sympathy, data oriented design, decoupling and writing production software.

Capital One
Cisco
Visa
Teradata
Red Ventures

Interested in Ultimate Go Corporate Training and special pricing?

Let’s Talk Corporate Training!

Join Our Online
Education Program

Our courses have been designed from training over 4,000 engineers since 2013 and they go beyond just being a language course. Our goal is to challenge every student to think about what they are doing and why.