Using tcpflow to debug HTTP API libraries | Hacker News Reader

Using tcpflow to debug HTTP API libraries

woss.name

59 pointsmathie11 comments

Funny that it mentions twisted (briefly). I last used tcpflow not long ago to debug some issues in twisted proxy. Twisted has two quirks that make it slightly less compatible/forgiving with the web at large than most browsers.

1. It lowercases headers, such that Cookie: becomes cookie:. Technically, the standard says they are case insensitive.

2. When reading responses, it strictly looks for \r\n line endings. \n will not cut. Standard does say \r\n is the only acceptable ending.

Guess which web server violates both of these rules? The one in arc that powers Hacker News. Figuring out why my proxy worked with every website except HN probably wouldn't have been possible without tcpflow.

Wireshark's Follow TCP Stream capability does roughly the same thing. Wireshark isn't the prettiest on OS X, but it works, and the ability to do this after-the-fact with tcpdump-created dumps from a remote box is useful.

http://www.wireshark.org/docs/wsug_html_chunked/ChAdvFollowT...

Charles is also extraordinarily handy. I used it extensively while developing an iPhone app in the simulator, since figuring out what NSURLConnection is doing is hard.

http://www.charlesproxy.com/

I typically use tcpflow or Charles on Mac OS X. They are both useful tools that any back end engineer should know how to use. It helps diagnose any problem your app/code is facing and it's another good way to learn how existing apps interact with their back end web service. I often check how iOS app interact with the server by usually setting up a proxy on my laptop and configure on the iOS device settings to use my local proxy. Then simply fire away tcpflow or Charles and watch all the HTTP calls being made.

Your first statement is a bit ambiguous, but I have to ask if you've gotten tcpflow to work on OSX. I can never get it to output anything when trying to monitor connections to localhost.

If you're looking to monitor traffic to localhost, bind the the `lo0` interface. For example, when I'm monitoring traffic to a local Rails app in development, I'll do something along the lines of:

    sudo tcpflow -c -n lo0 src or dst post 3000

and it'll pick up the traffic.

(Thanks for the tips on alternative ways of monitoring TCP traffic flows, folks!)

Plus 1 for for Charles. In the last month I've used it to determine:

- What an OAuth-signed SOAP envelope looks like, when Google (who prefer OAuth and require SOAP for some APIs) don't yet have it documented.

- That Chrome will encode links itself if necessary, but Firefox will not, causing a download on a colleagues app to not work on FF.

- That I'd accidentally seyup up my non-desktop app as a desktop app when registering for access to the Twitter API, resulting in an XML error response that Chrome wasn't showing me.

All over HTTPS. If you don't have an app like this (works on Windows, Mac and Linux) you need it.

This is why you need to hire full stack engineers, or have at least one on your team.

How does tcpflow compare with ngrep?

tcpflow is rather simpler, but probably a lot better for examining multiple connections. You specify a tcpdump/pcap filter and it saves the traffic to files (two per connection, one for each direction). You review the data at your leisure, it's not all jumbled up in your terminal.

Can tcpflow set up fake TLS certificates like Charles?

No. It takes data from the network and puts it into files. That's all. It doesn't inject, interpret, or decrypt data.

Reply on news.ycombinator.com