General Transit Feed Specification and Clojure | Hacker News Reader

General Transit Feed Specification and Clojure

grison.me

78 pointssimonpure8 comments

Seems like they could cut out a lot of that code by using macros. Here's a quick and dirty Common Lisp script that reads the CSVs from the .zip file and generates classes at runtime using the headers in each CSV for the slot names. It also defines a 'select' function to make queries easy to write. It's a little too long to post here:

https://gist.github.com/jl2/fd324fa3faa919803152465fade14b6b

And here's what it looks like to use it:

    CL-USER> (defparameter *transit-data* (read-transit-data "~/Downloads/gtfs_current.zip"))
    
    CL-USER> (select (table *transit-data* :stops) 'stop-name "PIERNE")
    
    ((STOPS  "PIERNE01"  ""  "PIERNE"  ""  "49.102273"  "6.179863"  ""  "http://lemet.fr/screen/index2.php?stop=257"  "0"  ""  "1")
     (STOPS  "PIERNE02"  ""  "PIERNE"  ""  "49.102558"  "6.179825"  ""  "http://lemet.fr/screen/index2.php?stop=234"  "0"  ""  "1"))
    
    CL-USER> (select
      (select (table *transit-data* :trips) 'trip-headsign "L5a - MAISON NEUVE")
     'route-id "5-77")
    
    ((TRIPS  "5-77"  "SMIN_H1920-CLaS50V2-Lun-Sam-98"  "409137-SMIN_H1920-CLaS50V2-Lun-Sam-98"  "L5a - MAISON NEUVE"  "1"  "60112"  "50057")
     (TRIPS  "5-77"  "SMIN_H1920-CLaS50V2-Lun-Sam-98"  "409138-SMIN_H1920-CLaS50V2-Lun-Sam-98"  "L5a - MAISON NEUVE"  "1"  "60111"  "50057")
     ;; ...
    )

>reads the CSVs from the .zip file

Neat! I can't find anything in Clojure that does that, so I guess I'd have to use java.util.zip via Clojure ineterop. But making records and doing queries is easy:

> generate classes at runtime using the headers in each CSV

In Clojure:

    (ns gtfs.core
       (:require [clojure.data.csv :as csv]))
    
    (defn load-records [f]
      (let [csv-data (csv/read-csv (slurp f))]
        (map zipmap
             (->> (first csv-data) ;; First row is the header
                  (map keyword)
                  repeat)
             (rest csv-data))))
    
    (def agency-maps (load-records "data/agency.txt"))
    (def routes-maps (load-records "data/routes.txt"))
    ;; etc.
    
    (count routes-maps) ;; 30
    (first routes-maps)
    ;; {: route_id "1-100",
    ;;  :route_short_name "1",
    ;;  :route_long_name "Ligne 1",
    ;;  :route_desc "",
    ;;  :route_type "3",
    ;;  :route_url "",
    ;;  :route_color "#B22E75"}

> defines a 'select' function to make queries easy to write

    ;; Query for long_name and route_color 
    (def route_ln_color (map #(select-keys % [:route_long_name :route_color]) routes-maps))
    
    (take 2 route_ln_color)
    ;; ({:route_long_name "Ligne 1", :route_color "#B22E75"}
    ;; {:route_long_name "Ligne 2", :route_color "#B22E75"})

Fun project! I also used Clojure for a transit status project - calling the REST API of my local bike share system to display how many bikes were in the nearest station. Clojure is great for exploring and transforming that kind of data.

One small tip for the author: clojure.data.csv would simplify csv handling. Its readme includes a simple routine to transform the data to maps.

Can you track which bikes move to which station?

No, because that data isn’t available in the public API.

Author here, thank you for the comment, you're right in the latest code I replaced it with clojure.data.csv but for the article I kept it that way to show that you don't need dependencies for such little utilities, split, map and zipmap and we're done.

You reminded me to work on my Jump Bikes Siri Shortcut which uses General Bike Share Feed specification.

https://github.com/BrianHenryIE/Bikeshare-Siri-Shortcuts

I made it so I could ask "where's the nearest Jump bike?" in Sacramento. At some point the API stopped working, but, after checking today, it was just the URLs of the updated feeds had changed (the old ones still 200 with no actual data) so I think I've managed to fix it.

Unfortunately, Jump shut down in Sacramento because of Corona virus!

Overall, it only works OK. When Apple Shortcuts requests GPS location, it doesn't just return the GPS, it makes a HTTP call to get the street info, which isn't needed here and slows things down. Then the gbfs.uber.com API is very slow... as though they don't cache at all, whereas I thought they only refreshed once/minute: https://gbfs.uber.com/v1/sacb/station_information.json

https://github.com/NABSA/gbfs/pull/190 https://www.sacbee.com/news/business/article241326756.html

From the title, my mind was primed to think this was related to Transit, which is (primarily) a Clojure standard for sending Clojure data structures back and forth between frontend and backend: https://github.com/cognitect/transit-format

It's cool to see some practical Clojure examples on hacker news.

Reply on news.ycombinator.com