Re-Write Core Http Stream using Curb

[rweald]

Unfortunalty due to problems with ruby's streaming gzip support there is no choice other than to move off of eventmachine and towards curl. The best way to do that in ruby right now is curb.

The external API will remain the same but the underlying implementation will no longer be running on eventmachine. This is an important thing to note if you were relying on gnip-stream being non-blocking EM compatible.

[eriwen]

You might not need to do this.

I have a commit that fixes em-http-request to handle GZip streams for this issue: igrigorik/em-http-request#173 and will be submitting a pull request soon after I do more testing/cleanup.

eriwen/em-http-request@aab1b31

I've gotten this working in my app and it's way better than crub IMO. Maybe it's best to help fix em-http instead.

[rweald]

Nice if they merge it into master then that would be great. I had looked at using an implementation like this originally but it had some problems with the buffering which is why I decided not to use it.

[eriwen]

I think I got the buffering fixed, but I want to test it out a bit more in different environments. I don't necessarily like the code around buffering, but I've haven't come with any good ideas for refactoring it. If you have any ideas, please share.

[rweald]

I look forward to seeing if it gets merged into the main em-http-client. I would try testing it under high load as well. My concern would be buffer overflow and memory bloat under high load such as a high volume twitter stream.

[rweald]

This issue is pending the resolution of:

igrigorik/em-http-request#173

igrigorik/em-http-request#173