Skip to content

Intro to Torrent files

July 22, 2011

To get you started, go here http://en.wikipedia.org/wiki/Torrent_file and read until you gloss over (took me about 12 seconds). The important thing to note is that torrents have an “info hash”. This is tied directly to the contents itself, but just think of it as an id number for the torrent. When you ask a tracker for peers, you give it this id, when you look in the DHT for peers for this, you use this id, when you create a magnet link because you’ve lost the torrent file, you use this id. In addition, there is information about the files that you’re going to be downloading, things like file name, size, etc.

The main flow is this. You go to clearbits.net or somewhere else that has these .torrent files. You download the torrent file and load it into your client (uTorrent hopefully). In the torrent file is enough information to find other people that have the actual contents that the torrent file represents. It might have trackers, or it might just take the aforementioned id and search the dht for peers. Either way, you’ll find some peers and begin downloading. And that’s it, you have your file(s)!

From within your app, the code might look something like the following…

btapp.add.torrent('http://www.clearbits.net/get/1766-the-tunnel.torrent');

There is an alternative to having the entire torrent file however. You can just add a magnet link. The following shows you both how to add a magnet link in an app and what the format for the magnet link is (the info hash is 160 bits…or 40 hex characters)

var hash = '0000000000000000000000000000000000000000';
btapp.add.torrent('magnet:?xt=urn:btih:' + hash);

When you add a magnet link you’ll go through the same process of looking for peers that you would if you had the torrent. You won’t know the specifics of the files that you’re going to be downloading though, until you finally find a peer. The first step is then to download the torrent file itself, which will tell you what you’re download, and potentially other trackers to find peers on etc…

To interact with a specific torrent from within your app, you’ll need to know it’s info hash. This info hash is the same one that you specified when adding the magnet link initially (the info hash can be a bit tricky to get for torrents that you load from a web site or some other source). So lets assume that you are continuing from above.

var torrent = btapp.torrent.get(hash);

torrent.stop();

torrent.start();

var magnet_link = torrent.get_magnet_uri();

torrent.remove(0); //0 means just remove the torrent, not the content

btapp.add.torrent(magnet_link);

For every torrent file there’s a magnet link that you can use instead, and that’s what the get_magnet_uri call does. In this case it’s just returning the same magnet link that you added, but it can be useful, especially if you want to share a torrent with someone. You just have to send this string vs sending an entire torrent file.

Magnet links are of particular interest because they don’t actually have to represent files the way torrents do. If you add the same magnet link to two clients and they discover each other through tracker/dht they’ll connect to each other and stay connected. They won’t know of any other peers that have the torrent file, so they’ll stay connected hoping that the other will acquire the torrent. Before I started playing with app to app messages, there was no real reason to create magnet links that had a random hash, as you’d just be wasting bandwidth. But now you can just have all instances of your client add a magnet link with a constant hash and all of a sudden all of your apps have joined a swarm, where every peer is running your app. And there’s 2^160 different info hashes to choose from, so you can create as many communication swarms as you like and use them for anything!

Advertisements

BtApp tutorials

July 22, 2011

I will be writing a series of blogs about sections of the utorrent apps api that I think are of particular importance. I’ll provide a rough outline and then try to fill in the sections one blog at a time. If there’s a topic of particular interest, please let me know and I’ll make sure to go into detail. The topics are in no particular order and will probably be discussed in order of interest to myself 🙂

1. Downloading the sdk, and getting a hello world app running. (this is really not trivial unfortunately, and kind of terrifying to me. I’ll need a weekend to get this one sorted out)

2. App to app messaging. This is the heart of the distributed networking stuff that uchat and several future projects are based on. In my opinion this is the most exciting aspect of writing apps for utorrent as it’s completely different (and arguably much more complex) than the server/client model.

3. Basic torrenting. This is not app specific, but there’s a ton of little tricks to using torrents that should probably be addressed. This will be stuff like using trackers, starting/stopping, peers, extension messages…

4. DHT (Distributed Hash Table). I’ll refer to the wikipedia on Kademlia, but its a pretty important part of the bittorrent system, and opens possibilities for some pretty cool apps. I’ll mostly talk about using it for “fun and profit”, not necessarily how it works.

5. Privacy/Security. People want it, not matter what FB believes. I’ll touch on some of the issues that I dealt with when developing uchat. Basically just a how to on protecting your users from themselves and others. I’ll also address the permissions that your app needs to perform various operations.

6. Debugging! Its not obvious how to do it for apps inside of utorrent, but its pretty straight forward, and absolutely necessary.

µChat now open source!

July 16, 2011

I’m up in Seattle for my family’s yearly pancake eating contest/roadbowling/strawberry festival parade weekend, but I haven’t been able to set my computer down entirely. I got permission last week to go ahead and release the source for µChat, something that I’ve been pushing for ever since it was released (apps are just zip files, so it’s been essentially open the entire time). I think it’ll be a great opportunity for everyone to see what’s possible in terms of app development, and it’ll be a chance for people to teach me how to write javascript that doesn’t look like C code. Which brings me to my next point…

This is the first thing I’ve ever written in Javascript. Overall I’m pretty happy with how it performs, but I still shudder when I come across some of the code I wrote towards the beginning…as I’m sure you will. Be kind, and feel free to rewrite it! I’m hoping to do a pretty substantial rewrite to use backbone.js (something I’ve only become aware of recently…no one tells us C/C++ guys about the hot web stuff) in the future, which will really simplify the ui code.

So without further ado…I present to you, the source!

https://github.com/bittorrent/uchat/

You’re also going to want to grab a copy of the uTorrent Apps SDK if you’re planning on building the app…

http://www.utorrent.com/community/developers/appssdk

I’ll make sure to document this all much better when I’ve back to the Bay. Just wanted people to get a look at it as soon as possible. If you’re at all interested in contributing, shoot me an email/comment/etc and I’ll definitely go out of my way to get you started.

µChat: Encryption for private conversations…finally (10 day turn around ftw!)

July 4, 2011

I got the request for encrypted conversations almost immediately upon µChat’s release, but I didn’t have to time to hammer it out until this weekend (happy 4th of July everyone!). The code is pretty fresh, and I admittedly don’t have that much experience doing this in Javascript, but it feels pretty solid. I just can’t bring myself to go into the office to release on a three day weekend, so it won’t be available in the apps store until tomorrow around 10ish (+/- 3 alarm snoozes).

Friends are, as I’ve mentioned in the past, simply a common magnet link, and a couple shared RSA public keys. When you log on to µChat, you add the magnet links for each “friend”. When you connect to a peer, barring any foul play, its almost certainly your friend (Imagine two people agreeing to meet at a specific random place in an infinitely large forest). To avoid foul play, we send the peer a challenge message that we generate on the fly, which they must sign to verify their identity. We do the same for them and everyone’s happy. Its our friend after all! This has been at work since the release, but the code to do this stuff made adding per conversation symmetric encryption a lot simpler, and I don’t think I’ve thoroughly explained it yet.

The public keys that we distribute so that people can friend us, can also be used for the sake of encrypting private conversations. We encrypt half of a random AES password with their RSA public key, and send it to them with the conversation request. They also generate half of a password and send it back to us when they agree to chat. From that point both peers use the complete AES key to encrypt the conversation. It was suggested that I use Diffie–Hellman, however it was important, especially when chatting with your friends that you can identify their original identity (as I’ve mentioned in the past, you never knew precisely what their original identity was… you just know that its the same person). This means that while you don’t necessarily know your friend, you are at least sure that its them that you’re talking to.

That’s it! If you have any suggestions, please shoot them my way. Especially if you can make a compelling argument against the way that this was implemented. Otherwise, happy chatting!

 

Javascript libraries used…

Stanford Javascript Crypto Library (BSD /Gnu GPL v2+)…used for the AES encryption

http://crypto.stanford.edu/sjcl/

Tom Wu’s RSA implementation (BSD License)…rewrote the keygen to be asynchronous.

http://www-cs-students.stanford.edu/~tjw/jsbn/

Kenji Urushima’s RSA signing/verification implementation (MIT License)

http://www9.atwiki.jp/kurushima/pub/jsrsa/

µChat: Response was quite the mixed bag

July 2, 2011

So µChat’s now been out for 8 days, and there are definitely some excited users using it as we speak (just spent the morning using µChat to get feature requests from the users…awesome!). This is not to say however, that it was not a bumpy road. I’m going to try to learn something from the experience, and hopefully I’ll keep these things in mind for the next project.

Lesson 1: Test EVERYTHING.

Obvious right? But I wrote µChat and tested the functionality for months, however a small discrepancy in how the app is built when going to production vs my local build resulted in a launch day kerfuffle that really tainted some of the good press that it got.

Lesson 2: Haters going to hate.

(This is the first and last time I’ll ever use this phrase…just wanted to try it out). Man it sucks when they’re after your baby though. At times it felt like no one understood the novel technology that makes the room discovery work. And this is entirely my fault. The term “Distributed Chat” has been thrown around for a long time and while I feel like µChat is “Distributed Chat” in its purest form (you don’t rely on servers to find rooms, your friends, anything), that message has been quite difficult to convey. Slashdot user are perhaps the most dangerous, because they’re so eager to discuss new technology. If your messages isn’t perfectly articulated, you’ll get worked over. That being said, getting Slashdotted has been a life goal for as long as I can remember, so it was a very exciting moment regardless.

Lesson 3: People love their privacy.

The number one comment is related to encrypting chat conversations. This has been in the works, and the crypto is already at work in the verification of your friends list. So it shouldn’t be too long before everyone sees a little lock icon in their private chat conversations to enable encryption. This feature probably won’t be available for non-private conversations, as it doesn’t really protect you from anything other than sniffing your connection. I’ll tell people when they’re genuinely protected, but otherwise I’d prefer to not give the impression of security.

Lesson 4: µTorrent/µChat users are second to none. (Thanks Nitrate!)

uTorrent has long benefited from an incredibly bright, dedicated community of users that aggressively test alphas and betas, finding bugs and making suggestions. µChat is already benefiting for a similar community. I make an effort to sit in the welcome rooms for a couple hours a day, offering help for uTorrent and getting suggestions for µChat. And there are a lot of people that want it to succeed, and really want to be a part of making it better. I spent a good long while with a user called Nitrate this morning going over the UI and seeing what worked and didn’t work in his eyes. And its so awesome to feel like I’m working as part of a team. And these users can legitimately claim that they were integral to the success of the product.

So again, thank you to everyone who’s taken it upon themselves to help make µChat better. I’m eternally grateful, and look forward to making µChat everything you guys want it to be.

µChat: We don’t need servers, we just need each other.

July 2, 2011

this is a repost of http://blog.bittorrent.com/2011/06/30/uchat-we-just-need-each-other/…that version got editor treatment and has some nice screenshots and such.

uChat always seemed like such a simple idea. We have the world’s most passionate users, and a bunch of them were filling our forums with requests for chat functionality. Some wanted to talk to fellow downloaders to get ideas for other content, some just wanted to thank the seeders. And so when I decided I wanted to do a little side project, the need dictated the project. Except I didn’t think the world needed another server.

And we torrenters already had such awesome building blocks in the form of torrents and magnet links. So that’s what you’re using when you use uChat. Every piece of functionality probably uses magnet links in one way or another. Every room you join is a magnet link, and the peers you see in the rooms are fellow swarm peers. Except I pulled a fast one on uTorrent…so you’re not actually downloading anything (and the peers don’t disconnect because we’ve coded them to be very optimistic about their future chances of getting the torrent file. Silly peers). So that all seemed pretty good, except we still didn’t have a way to find rooms, and we still didn’t have a buddy list. Two deal breakers left.

It occured to me pretty early that we were going to need a central swarm to advertise other rooms on. So every uChat client joins a giant predetermined swarm. We’ve seen swarms in the wild that are absolutely massive, so it seemed like a safe way to get all the peers together to exchange rooms that they knew about. Great, so now uChat clients can tell each other about rooms. One deal breaker left.

The buddy list is my favorite. In the real world no one knows all your friends. Just you. And when you introduce yourself, you can say whatever you want. And I think that’s a pretty good setup. So your buddy list is known only to you, and you can call yourself whatever you’d like. When you become friends, you both add the same unique magnet link link so that you can find each other later simply by joining that swarm. We also do a bunch of crypto to verify the friend because frankly, the world’s a scary place. So instead of both your information sitting on a server, a friendship on uChat is simply a common swarm, your friend’s public key, and their original name (because sometimes friends change their name…though admittedly less often in the real world).

So there you have it. A way to make and keep relationships with people around the world with no central server. And I think that’s pretty cool. Another chat client is probably not going to rev the tech world’s engines, but the possibilities created by the underlying technology are pretty exciting. And that underlying tech that makes this all work is already finding it’s way into my next side project…