Pipe To Me - Stream Data over HTTP using curl
I built a thing! It’s called pipeto.me. It allows you to stream data from the command line over HTTP in real time. It’s built in Go. It uses no client-side javascript. It has no external dependencies beyond the Go standard library. All of the code is hosted up on github. Check it out.
If you are confused what this actually is, here is a demo:
How it all started
Funny enough, the idea came out of a joke site put out by Rob Pike (one of the Go creators): robpike.io. I hadn’t seen anyone use http in this way before. After digging in to how it worked, I was amazed at how much there was to learn from such such a simple example.
The browser will start displaying the contents of a response before it has entirely downloaded. In this case the browser sees it as a really slow server that sends a bit of content every so often. The page never fully loads until the user cancels the request.
In the case of robpike.io, the site sends this unicode character (💩) back on the http connection approximately every 100ms.
I started out with the idea of using the same principles to build a command line chat program that used curl as the transport mechanism. I may still build this at some point. However, like all good yak shaving programmers, I decided to take it up a level of abstraction and build something that allowed you to stream data from unix pipelines over HTTP.
How it works - The receiver
When a receiver connects to the service, it first sends over several important headers:
Transfer-Encoding: chunked : This tells the client that it uses chunked transfer encoding. Instead of replying with a single message with a fixed Content-Length, chunked transfer is an HTTP/1.1 feature that allows you to reply with a stream of messages each prefixed with it’s own length. This is not a new feature at all, but may be overlooked. Go is smart enough to recognize this kind of content and attach this header on its own.
Connection: keep-alive : This header tells the browser to keep the connection open so that we can continue to stream data. Again, Go handles this header automatically.
X-Content-Type-Options: nosniff : This was a header that I had never seen before. Since we aren’t actually sending any content yet, this tells the a web browser not to guess at what kind of content it is. Without this header, the browser will try to download the stream as a never-ending file instead of displaying it’s contents.
After sending these headers, the receiver saves the output stream and then waits until:
- The user manually disconnects (hits the browser Stop button, kills the curl process)
- Or the pipe is closed. (by a sender sending the EOF character)
This mechanism is actually very similar to a technology called server sent events, which adds some additional metadata and has a javascript client api in most browsers. I didn’t try to be compatible with SSE, because the additional metadata might break some use cases.
How it works - The sender
The sender uses curl -T- to read input from stdin and send it to the service. The send side also uses chunked transfer encoding to stream data. In addition to Transfer-Encoding, it also sends the following header:
Expect: 100-continue : This header means that the client is going to wait before it starts sending data. If the server is in failure mode (?mode=fail) and there are no receivers connected, it will respond with status 417 Expectation Failed. Otherwise, the server will respond with 100 Continue and the client will start sending data.
The server will continue reading data from the sender until:
- The user manually kills the curl process
- Or the EOF character is sent.
If piping into curl, EOF is usually sent automatically. If you are typing to stdin manually, Ctrl-D will close the stream with EOF.
In order to copy data from the sender to the receiver, I used the Go io package directly. I made a custom writer that copied the input stream to (potentially) many receivers, wrote out to each one, and flushed the buffer back to the receiver.
Conclusion
This project actually received a lot more attention than I thought it would after I posted it to hacker news:
It was one of the github trending Go projects for 3 days straight. Maxing out at #4 on the list: