/ 
Net-BitTorrent-0.049
River stage zero No dependents
/ Net::BitTorrent::Notes

NAME

Net::BitTorrent::Notes - Annotated Guide to the Ins and Outs of Net::BitTorrent

Description

This is a first draft attempt at defining a road map for future Net::BitTorrent development and a behavioral reference for third-party client developers. A few bits for users might slip in too.

Net::BitTorrent's Way-too-Obvious Class Hierarchy

                                          .---- N::B::T::T::UDP
                                         /
                    .-------- N::B::T::Tracker
                   /                     \
                  /   .-- N::B::T::File   `--- N::B::T::T::HTTP
                 /   /
          .---- Net::BitTorrent::Torrent
         /
        /   .--- Net::BitTorrent::DHT
       /   /
Net::BitTorrent
       \
        `---- Net::BitTorrent::Peer

Installation

This distribution uses Module::Build for installation, so use the following procedure:

perl Build.PL
./Build
./Build test
./Build install

See also: Automated Test Reports

Prerequisites

Net::BitTorrent requires version and Digest::SHA. On Win32, we try to use Win32API::File and Encode to handle extended charset filenames[1]. As of perl 5.10, all of these modules are are CORE; they come bundled with the distribution.

I have listed these modules as prerequisites in Build.PL so, unless you answer 'no' when prompted, the CPAN shell should automagically install them for you.

We also use the internal utf8() functions which didn't appear until perl 5.8.1. See Portability Hacks.

How Do I...

Parts that aren't handled internally are described here (eventually) with sample code to get you started.

Get basic info from a .torrent without adding it to a client

Net::BitTorrent::Torrent objects can be created directly without a parent client. While functionally limited (obvious things like an inability to serve data, etc.) basic information is available and some 'advanced' functions still work (hashchecking, for example). See Net::BitTorrent::Torrent for more.

Pause and Resume a .torrent Session

See Net::BitTorrent::Torrent::pause( ) and Net::BitTorrent::Torrent::start( )

Stop and Resume a .torrent Session

See See Net::BitTorrent::Torrent::stop( ) and Net::BitTorrent::Torrent::start( )

Quick Resume a .torrent Session Between Client Sessions

Note: This section describes resume functionality as of v0.045.

Net::BitTorrent is capable of restoring the state of single torrents between sessions. To store resume data, use the save_resume_data( )

To resume a single torrent, use a variation of the following to save the data...

my $torrent = $bt->add_torrent( { Path=> 'some.torrent', Resume = '.resume' });

# later...

$torrent->save_resume_data();

...and unless Net::BitTorrent::Torrent decides the resume data is bad, you'll start right were you left off. I would suggest storing resume data on a regular basis while the client runs and again on exit.

To view a fully functioning example, see /tatoeba/004-resume.pl.

For more on what exactly you're saving and the structure of the data, see Resume API in the <Net::BitTorrent Internals/|"Net::BitTorrent Internals"> section.

Save and Restore Client-wide State and DHT Data

Unless you've hard coded everything, being able to restore client-wide state is a necessary feature. Besides, DHT can be very slow to boot without a good set of initial nodes and the spec states that the local nodeID should not change very often, so resume is a useful thing.

I would use a hash with the following keys:

.hash

This would be a SHA-1 hash of the bencoded data in the .t, dht, nodes, and torrents keys. On restore, I would use this to validate the data in case it was tampered with.

.t

Timestamp.

.version

To avoid problems (API changes, etc.), I would use an API version number and ignore resume data created with a newer/incompatible version. This value would not be included in the SHA-1 digest used to prevent tampering.

dht

A hash with the following keys:

id

The local node ID used in the DHT swarm. To obtain this, see node_id( ).

nodes

List of nodes in the DHT routing table.

To make life easy, each node would be a hash with the following keys:

ip

IP or hostname of the remote node.

port

UDP port number the remote port has been contacted on.

A list of nodes with this format is obtained from nodes ( ). To reload these later, use the add_node ( ) method.

settings

These would be any client-wide settings I allow users to change.

torrents

This would be a list of filenames, their current status, and some sort of verification that the .torrent file hasn't been replaced; the infohash would do.

I would save a bencoded version of this data in a file for later. For now, putting all of this into practice is an exercise for the reader.

Note: Reloading the data may require using otherwise private methods (specifically the private Net::BitTorrent::DHT->_set_node_id( ) method). Changes to private methods are not listed in the changelog found in this distribution but they are noted in the public SVN repository's log.

Set File Download Priorities

See Net::BitTorrent::Torrent::File.

Implement My Own Event Loop

[TODO]

Net::BitTorrent Internals

This section describes all the behind the scenes stuff that makes Net::BitTorrent work. Or not work. It depends.

Peer ID Specification

Please see Net::BitTorrent::Version.

Handling of Errors and Bad Data

[TODO]

.torrent Metadata

[TODO]

Incoming Protocol Data

[TODO]

Disk Access Errors

[TODO]

Piece Selection

To have any semblance of basic functionality, a BitTorrent client must have a good piece selection strategy. The 'strict priority' based algorithm used by Net::BitTorrent seems to strike a balance between efficiency and speed while keeping the pipes full of incoming data. I'll walk you through it...

  1. Count the number of unrequited $free_blocks in all working pieces.

  2. Count how many $free_slots we have open to us. To calculate this, we iterate through all peers who aren't choking us adding the difference between the total number of outgoing requests and the peer-defined maximum or our own internal 'safe' max.

  3. $max_working is set to the current number of working pieces plus the bool value of $free_blocks < $free_slots.

  4. Calculate %weights based on file priorities and our bitfield.

    4b.

    If %weights is empty, we return without making a new request.

  5. Take the sum of all the relative (priority) weights and call it $total

  6. Take a random number between 0 and 1, and multiply it by $total. Call the new number $rand_val

  7. While $rand_val is positive, iterate through each element of @weights and do the following:

    7b.

    Subtract the (priority) weight of the element from $rand_val and increment a temporary index $i

    7c.

    If $rand_val went negative or we run out of values for @weights, we exit the loop.

  8. If %weights{$i} is defined...

    8b.

    We set the working value to true and request a free block. Blocks are requested in order from the beginning of the piece to the end. Finally, we re-request any rejected or canceled blocks.

    8c.

    ...otherwise, we return an undefined value.

The language used in this section probably only makes sense to me and will require a rewrite. For the actual code, see Net::BitTorrent::Torrent::_pick_piece().

Outgoing Requests

[TODO]

Queuing

[TODO]

Slow Blocks

[TODO]

Endgame

[TODO]

Incoming Requests

[TODO]

Queuing

[TODO]

Rejections (Fast Ext)

[TODO]

Trackers

[TODO]

Multi-tracker .torrents

[TODO]

UDP Trackers

[TODO]

Fast Set Advertising

[TODO]

Socket6 does not seem to work with Win32 so... no plans for IPv6 right now.

Implemented Extensions

The BitTorrent Community Forum coordinates the development of the BitTorrent protocol suite, its reference implementation, and BitTorrent Enhancement Proposals (BEPs). For more information, see BEP 0: Index of BitTorrent Enhancement Proposals http://bittorrent.org/beps/bep_0000.html

This is the list of extensions used by this release of Net::BitTorrent sorted by their progress toward standardization.

Accepted BEPs

These BEPs describe mechanisms that have been deployed in one or more BitTorrent implementations and have proved useful. They may require minor revisions. They await the blessing of the BDFL before they can be considered Final.

  • BEP 32: Tracker Returns Compact Peer Lists - http://bittorrent.org/beps/bep_0023.html

Draft BEPs

The following BEPs are under consideration for standardization.

  • BEP 5: DHT Protocol - http://bittorrent.org/beps/bep_0005.html

  • BEP 6: Fast Extension - http://bittorrent.org/beps/bep_0006.html

    Note: the Fast Extension is only partially implemented in Net::BitTorrent.

  • BEP 10: Extension Protocol - http://bittorrent.org/beps/bep_0010.html

  • BEP 12: Multitracker Metadata Extension - http://bittorrent.org/beps/bep_0012.html

  • BEP 15: UDP Tracker Protocol - http://bittorrent.org/beps/bep_0015.html

  • BEP 27: Private Torrents - http://bittorrent.org/beps/bep_0027.html

Portability Hacks

Net::BitTorrent aims to be portable between the big three OS when possible, this should be achieved without a glut of code using easily obtained third-party modules. Core modules are considered first followed by well tested CPAN modules. Modules that prevent broad use (ie. fails to install on a majority of systems) will not be considered.

There will be times, though, that $^O-based clutter is needed. Here is a list of both stable and experimental workarounds by OS:

MSWin32
Extended charset filename support

[TODO]

Resume API

Net::BitTorrent::Torrent's resume data is bencoded and stored in a file. To restore this data, use the Resume parameter when calling Net::BitTorrent::Torrent->new( ) or Net::BitTorrent->add_torrent( ).

Note: The structure and data held in the resume data is subject to change in future versions.

Data Structure

Parsed resume data is a simple hash containing the following keys:

.format

A string describing what sort of file this is. For now, it's value is Net::BitTorrent resume.

.t

Timestamp when data was gathered.

.version

API version used to gather data. To avoid problems (API changes, etc.), Net::BitTorrent::Torrent will not load resume data created with a higher version.

bitfield

A bitvector representing the pieces we already have.

files

A list of hashes (one for each file in the .torrent) with the following keys:

mtime

The modified times for the files (or 0 if the file does not exist).

priority

The file's download priority.

peers

Compact list of peers we've found either through DHT or from a tracker.

working

List of hashes representing pieces we are currently downloading with the following keys: (Subject to change)

Block_Count

Number of blocks contained in the piece.

Block_Length

Size of blocks in piece.

Block_Length_Last

Size of final block in piece.

Blocks_Received

Bitfield representing which blocks have been received and written to disk.

Endgame

Boolean value dependent on endgame state when we began working on this piece.

Index

Zero-based index of the piece.

Length

Total size of the piece in bytes.

Priority

Priority based (partially) on the piece's containing file.

Slow

Boolean value dependent on how long ago we received a block contained within this piece.

Data Verification on Restore

TODO

Compatibility Notes

This section lists recent major changes in API or behavior between stable releases. For older news see the Changes file included with this distribution. For detail see the SVN logs.

v0.040

Entire distribution was rewritten. Both the internals and the API have broken compatibility.

Giving back

If you're interested in assisting with Net::BitTorent's development but don't know where to begin, here are a few ideas.

Joining the Project

THIS PROJECT IS ACTIVELY SEEKING DEVELOPERS. Ahem, I hate to shout but over the next few major versions, my priority is Protocol Encryption (http://wiki.vuze.com/index.php/Message_Stream_Encryption) which, like conversation with the opposite sex, requires a familiarity with cryptography.

So, if you're interested in helping with that, or anything listed on the TODO list or, maybe you have a suggestion and are willing to see it through to completion, contact me; I'm handing out SVN commit bits like (expensive) candy!

In general, N::B could use a second or third pair of eyes. So, if you're interested in BitTorrent, p2p, or just Perl in general, let me know.

Automated Test Reports

Becoming a CPAN Tester is an easy, automatic way to contribute to the quality of your favorite module and CPAN in general. If you would like to contribute automated test reports for Net::BitTorrent, install CPAN::Reporter from the CPAN shell first:

$ cpan
cpan> install CPAN::Reporter
cpan> reload cpan
cpan> o conf init test_report
  [...follow the CPAN::Reporter setup prompts...]
cpan> o conf commit
cpan> install Net::BitTorrent

For more on becoming a CPAN Tester and why this is useful, see the CPAN::Reporter documentation and http://cpantesters.org/.

Bug Reporting

Found bugs should be reported through Net::BitTorrent's Issue Tracker. Please include as much information as possible. Before creating a new report through Net::BitTorrent's Issue Tracker, please review the following list:

  1. Make sure you are using the most recent release of Net::BitTorrent. This may mean checking out the latest SVN commit. All patches should be made against the most recent revision and well tested. For a list of SVN clients, some of which make patch creation a little easier, see http://subversion.tigris.org/links.html#clients.

  2. Make sure the bug is reproducible.

  3. Please write in clear English.

  4. Include as much detail as possible when describing the bug. Provide "baby steps" to describe exactly how to replicate the bug. Sample code is welcome. Net::BitTorrent's issue tracker also allows attachments so, if relevant, also include the .torrent file.

    If the bug is related to one .torrent in particular, please attach it to your report.

  5. Search the list of open and resolved issues to make sure the flaw hasn't already been reported. If it has, you can star the issue to stay up to date. You'll know what I mean by 'star' it when you get there... I can see how many people have stars on a particular issue and popular bugs get priority.

  6. Issues are open to the public, so don't include passwords or other confidential information. Beyond that, you can never provide too much information in a bug report.

  7. One bug is one bug report. Please do not include multiple, separate issues in one report unless they are explicitly related to each other.

  8. It never hurts to do a little homework. If you run into a problem, find the place that's causing trouble and manage to fix it, please attach a patch (diff against the latest svn revision) or at least a very good description (with code) of what you did.

  9. Star the issue so you can stay up to date with my progress.

  10. Look over your report before submission to be sure you've included as much detail as possible. Seriously. Get up, have a drink of water, come back, read over it again to make sure you've included everything you intended, and then submit.

Patch Submission

Net::BitTorrent is too large for just one person to hack on. If you're Perl adept and would like to help, you can start by fixing problems reported in the Issue Tracker and bugs you find on your own. I am currently seeking co-devs and patches are a great way to get your foot in the door.

Please submit patches for review by attaching it through the Net::BitTorrent Issue Tracker. If it's a patch to fix an existing Issue, use that thread. Otherwise, create a new Issue. Minor patches get your name in the changelog. Major (security, especially) patches get your name in the Acknowledgments section. Oooo. Ahhh.

See Also

The Project's Website

For updates and info on subversion repository access and the occasional long winded rant, please visit http://sankorobinson.com/net-bittorrent/.

Receive SVN Commit and Issue Tracker Updates

The preferred way is to subscribe to one of the feeds provided by Google. ATOM feeds, Gadgets, and CSV files are provided for various data. See http://code.google.com/p/net-bittorrent/feeds for a list.

Public Mailinglist

Rather than contacting me directly (which you're welcome to do, it's just nice having a searchable, public archive), general questions and comments should be posted to the Net::BitTorrent mailing list. To subscribe to the list or view the archive, visit http://groups.google.com/group/net-bittorrent. Both ATOM 1.0 and RSS 2.0 feeds are provided; see http://groups.google.com/group/net-bittorrent/feeds for a list.

Issue Tracker

Use http://code.google.com/p/net-bittorrent/issues/list for bug tracking. Please include as much information as possible and review the list above.

Stalk Me While I Tinker

Follow Net::BitTorrent's development on Twitter: http://twitter.com/net_bitTorrent.

Ohloh

It's open source social networking. ...I think. Regardless of its use, Net::BitTorrent has an Ohloh page: http://www.ohloh.net/projects/net-bittorrent/

Wait, project cost: $65,430? Yeah, okay!

Other Recommend Open Source BitTorrent Clients

RFC 3986 (URI: Generic Syntax)

Section 2.3. "Unreserved Characters" (http://tools.ietf.org/html/rfc3986#section-2.3)

PAUSE FAQ sub-section entitled "Developer Releases"

(http://www.cpan.org/modules/04pause.html)

Author

Sanko Robinson <sanko@cpan.org> - http://sankorobinson.com/

CPAN ID: SANKO

License and Legal

Copyright (C) 2008 by Sanko Robinson <sanko@cpan.org>

This program is free software; you can redistribute it and/or modify it under the terms of The Artistic License 2.0. See the LICENSE file included with this distribution or http://www.perlfoundation.org/artistic_license_2_0. For clarification, see http://www.perlfoundation.org/artistic_2_0_notes.

When separated from the distribution, all POD documentation is covered by the Creative Commons Attribution-Share Alike 3.0 License. See http://creativecommons.org/licenses/by-sa/3.0/us/legalcode. For clarification, see http://creativecommons.org/licenses/by-sa/3.0/us/.

Neither this module nor the Author is affiliated with BitTorrent, Inc.