Extensions

From ADC Project

Extensions allow ADC to flourish and continue to evolve.

This page is only for requests and validation of new extensions to the ADC protocol, and not changing an existing command. See the Protocol requests page for changing an existing command.

See Category:Extensions for extensions that have been already added to specification.

To add a new extension, add the action, context etc to this page. To keep discussion and the actual code seperate, please conduct any talk on the talk page and only keep code here.

To be incorporated in the official specification, an extension request must meet the following requirements; §1 The extension must be used in an actual implementation. It must not only exist conceptually. §2 There must be a concensus in the talk page concerning the extension. Lack of explicit disagreement in the talk page (basically "no talk") constitutes as agreement of the command syntax and purpose. If you edit an extension's syntax etc, add a note on the talk page, explaining why.

If an extension has met §1 and there have been concensus (§2) for one month (after the last dispute), the extension's status may be changed from eg "Pending, awaiting requirement §2" to "Added as of 2008-12-30 to specification. See Samle wiki page" (or similar). As well, a note in which client and at which date when the consensus agreed. (The extension request doesn't need to list all known implementations, but at least one.)

UCMD - User commands

Status: Pending, awaiting requirement §1 and §2.

User commands are used to send hub-specific commands to the client which provide useful shortcuts for the user. These commands contain strings which must be sent back to the hub and parameter substitutions in the strings. Each user command has a display name, a string to be sent to the hub, and one or more categories where it may appear. The strings passed to the hub must first be passed through a dictionary replacement that replaces all keywords in the string and then through the equivalent of the C standard function "strftime", with the current time.

CMD

CMD name

  | 1 = Hub command, client parameters only
  | 2 = User list command, client and user parameters
  | 4 = Search result command, client, user and file parameters
  | 8 = File list command, client, user and file parameters

hubXX | One for each flag of the hub

ADCS - Secure ADC

Status: Pending, awaiting requirement §1 and §2.

Secure ADC connections can be established using a TLS tunnel, both for hub and for client connections. Certificates can be used to authenticate both hub and user, for example by making the hub the root CA, and only allow clients signed by the hub to connect. Ephemeral keys should be use to ensure forward secrecy when possible.

Client-Hub encryption

TLS client-hub connections can be initiated either by negotiating the feature "ADCS" on connection or by using the protocol adcs:// when initiating the connection. Hubs can choose to request a certificate for the user on login, and use this certificate to replace password-based login.

Client-Client encryption

TLS client-client connections can be established either by negotiating the feature "ADCS" on connection or by specifying "ADCS/1.0" in the CTM protocol field. Clients supporting encrypted connections must indicate this in the INF SU field with "ADCS".

DFAV - Distributed Favorites

Status: Pending, awaiting requirement §1 and §2.

The idea behind this extension is to generate a public hublist from the users favorite hublist. I think there should be a seperation between public and private hubs in the favorite hublist of an user, in order not to distribute private hubs where one can not connect to anyway.

GFA

Contexts: T, C

Asks all users within the same hub with the correct feature to send all publicly available hubs, in their favorite hub list to the requesting client.


RFA

Contexts: C

Response of a client.

HA | Hub address

LG | Last succesfull login time ( number of seconds since the epoch (1970), (UTC) )

PING - Pinger extension

Status: Added as of 14 Mar 2008 to specification. See PING page for details.

Actual implementation since 23rd Jan 2008 in DSHub

Actual implementation since 8th Feb 2008 in Hubtracker.com

TYPE - Typing notification

The typing notification is similar to Jabber's "Chat State Notifications" [1].

This extension adds the new command type TPN. If client supports this it must add TYPE in INF SU field.

TPN should be send to Reply-To user (for chatroom orgainzation and etc) (PM flag in MSG if any). In that case chatrooms bot can normally broadcast this messages to "joined" users.

Contexts: F, T

TPN code

Codes:

00 - gone (closed tab, minimised window, etc..)
01 - inactive (changed tab, window became unfocused (but not closed/minimised), etc)
02 - paused (paused typing)
10 - active (now in this tab, reading message, actively participating in the chat session)
11 - composing (now typing a message)

All TPN messages are sent when event occurs. 00, 01, 02 - client should send; 10, 11 - client must send.

BLOM - Bloom filters

Status: Pending, awaiting requirement §1 and §2.

Implementation: DC++, ADCH++

Description: Bloom filters allow the hub to filter certain searches using bitmap that represents the hashes of the files in the users share. For an introduction to bloom filters, see Wikipedia. When the user updates the share, it must send an INF containing the flag SF. The hub may at any time request that the client sends an updated version of its bloom filter by sending a GET command to the client. The client will then respond using SND and send the bloom filter binary data.

Let b be the number of bits used for file hashes, n the number of files in the users share, m the size of the bloom filter in bits, k the number of subhashes constructed from the file hash, h the number of bits to use for each subhash and p the probability of a false positive. The hub chooses k, h and m. k must be chosen so that k*h <= b. h must be chosen <= 64 and such that 2^h > m. m must be chosen such that m % 64 == 0. The hub then sends GET specifying "blom" as type, "/" as path, "0" as start and "m/8" as number of bytes, k in the flag "BK" and h in the flag "BH".

The client constructs the bloom filter by creating a bit array of m bits set to 0. For each file it then sets to "1" k positions constructed from the file hash. Seeing the file hash as a stream of bits (starting from the lowest bit of the first byte, ending at the highest bit of the last byte), the client should use h bits starting at the first bit of the first byte to create an integer and apply modulo m to get the position in the bit array, then redo the process k times advancing the starting position by h each time.

Once the hub has received the bloom filter bit array, for each search command it processes, if it contains a hash search term, it can discard skip broadcasting the search to a particular client if at least one of the k bits in that clients bit array is "0", calculating positions as the client does when setting bits to "1". The hub has to evaluate the filter for each client that it has a bloom filter for, for each search.

p = (1 - (1 - 1 / m)^(k * n))^k, thus p becomes smaller as m grows and larger as n grows. Larger m means more bits to transfer but also fewer false positives. The optimum value for k given m and n is (m / n) * ln 2. The largest k supported by a hash of a certain size is b / h, so if the hub wants the smallest p possible, it should choose the smallest possible h which gives the largest k, and then calculate m = k * n/ln 2, checking that the resulting m < 2^h. 2^h should much be larger than m (at least 3-4 times), because of how the modulo operator works. Also, with m grows the required bandwidth to transfer the bloom filter, so the hub may wish to cap m. In that case, it should still choose k according to m / n * ln 2, but send an h as big as possible to alleviate biasing problems.

For TTH roots, b is 192 and a reasonable value for h is 24, giving a maximum k = 8 which means that m = 8 * n / ln 2 ≈ 11.5 * n. The required bandwidth then becomes 11.5 * n / 8 bytes, so approximately 1.44 bytes per file in the users share. For 20000 files, m should then be 230016 (taking into account the modulo 64 requirement), giving a p = 0.004, in other words ~99.6% of all searches that don't match a users share would not be sent, saving valuable upload bandwidth for the hub. The client calculates i valid positions, if x is an array of bytes containing the hash of the file, on a little-endian machine, by doing pos = x[0+i*h/8] | (x[1+i*h/8] << 8) | (x[2+i*h/8] << 16) for i = [0;k). This is of course a special case where h % 8 = 0, the actual algorithm has to take into consideration values for h where the integer crosses byte boundaries.

Test Vectors

LINK - Hublink

Description: This extension allows linking hubs together. So hubs can send messages to each other, share users and forward search requests or chats.

How it works: To link two hubs, one must be a master hub and the other a slave hub. Both should support the same Hash extensions and of course the LINK extension. They have to provide a hub CID and must hide the PID to other hubs. The connecting hub is the slave hub. When a slave connects to a master, the master thinks its a normal client, sends its SUP and a SID. The slave ignores the SID and returns his INF and the master notes its a hub. Instead of sending the INFs of the connected users, the master sends the INFs of other hubs, he is connected with (both master or slave) with HUB command. The communication of the hubs works with CIDs instead of SIDs, because a CID is (should be) unique. Hubs should use tokens to decide if a message has to be forwarded or not. Its also possible to share users or forward search requests: but the hub must translate the escaped adc commands. for example internal user SIDs of every hub must be translated to own SIDs to avoid SID name clashes.

Commands:

HUB

 HUB cid hash

 Contexts: F, T

 {cid} is a Base32 encoded CID from the originator of the msg, required 
 {hash} is the hash extention used for creating {cid}, required to avoid name clashes beetween CIDs

 TO: Base32 token, at least 10 bytes long, optional. should be used to identify a message in the hub network 
     to avoid endless cycles of forwarded messages
 DH: CID of destination hub, optional. if destination is present, the hub which receives the message must send it 
     to destination or to the hub which is linked to destination; else send it to all, but not to hub where 
     message came from
 AC: escaped ADC command, optional
 TM: text message, optional

Example: 5 hubs: M(master), S(slave), C(M), D(M), E(S)

 - M-S: ISUP ADBASE ADTIGR ADLINK
 - S-M: HSUP ADBASE ADTIGR ADLINK
 - M-S: ISID ABCD
 - S-M: HINF ID{cid_S} NI{nick_S} CT32				// slave sends his INF, ignores SID
 - M-S: IINF ID{cid_M} NI{nick_M} CT32				// master sends his INF
 - M-S: IHUB {cid_M} TIGR ACIINF\sID{cid_C}\sNI{nick_C}\sCT32\n TO{token_2}	// master sends INF of other hubs
 - M-S: IHUB {cid_M} TIGR ACHINF\sID{cid_D}\sNI{nick_D}\sCT32\n TO{token_3}
 - M-S: IHUB {cid_M} TIGR ACHINF\sID{cid_S}\sNI{nick_S}\sCT32\n TO{token_4}	// INF of slave...
 - S-M: HHUB {cid_S} TIGR ACIINF\sID{cid_E}\sNI{nick_E}\sCT32\n TO{token_5}	// slave sends INF of his other hubs
 - S-E: HHUB {cid_S} TIGR ACIINF\sID{cid_M}\sNI{nick_M}\sCT32\n TO{token_2}	// slave send INFs of master to his hubs (E)
 - S-E: HHUB {cid_S} TIGR ACIINF\sID{cid_C}\sNI{nick_C}\sCT32\n TO{token_3}
 - S-E: HHUB {cid_S} TIGR ACHINF\sID{cid_D}\sNI{nick_D}\sCT32\n TO{token_4}
 - E-S: HHUB {cid_E} TIGR TMHello\sWorld DH{cid_M} TO{token_6}		// E send welcome message to M
 - S-M: HHUB {cid_E} TIGR TMHello\smy\smaster DH{cid_M} TO{token_6}	// slave forwards message to master
 - M-S: HHUB {cid_E} TIGR TMHello\sslave\s^^ TO{token_7}		// master greets his new slaves

UPQU - UPload QUeue

Status: Pending, awaiting requirement §1 and §2.

This extension's purpose is creating a queue on a client, when multiple other clients want to download from it, but they have no slots. Currently, when a slot is being freed, the first connecting client gets it. Other clients that don't have the luck of getting in time to attempt to download , have to wait again.

The extension creates a client queue based on CIDs and tokens from the connecting client.

The client who creates a queue must have a ticket number for each connecting client, which must be kept internally , and a difference between current connecting client's queue number and the currently uploading client's be provided to the connecting client, so that the clients are being deserved in the order they originally connected.

When the slot is freed up, the uploader should send a CTM/RCM to the client in question ( if not online anymore, the next in queue ) with the same token the downloader originally used in the first connection. If it doesn't respond ( in some timeout at uploader's discretion ), then it will be removed from queue.

The client could have a ticket incrementing starting from 1 for each session.

The extension adds :

STA

Contexts : C

Error code : 53 Flag QP represents the difference between current client's ticket and the one being deserved's. In other words, how many others are in front of it in the queue.

Example:

STA x53 No\sslots\savailable QP3

which means that there is no slots for the current download, and the transfer is the 3rd in the Queue. The downloading client may inform the user about his/her current position. Being 3rd in queue means that there are 3 more uploads in queue before the downloading client.