============================================================================== Introducing the X-DUPE feature by Hoopy, June 2001 ============================================================================== 1. What is it? The X-DUPE mechanism helps you saving time when uploading files by avoiding "dupes"; in this case we refer to the term "dupes" as files which already have been uploaded to the same directory by other users. For example, there's something new to upload. 10 people start uploading files. You are one of them. You are using a FTP or FXP client where you can tag a whole directory... you just select the new directory, press some hotkey for transferring it, and let the program handle the rest. Let's say there is a total of 50 files to transfer, so after you finished your first file, your program tries to upload the second file. However we got 10 people uploading, so you'll probably run into a dupe. Running into a dupe looks like this: STOR blabla.r03 553- blabla.r03: This file looks like a dupe!! 553 It was uploaded by someone ( 0m 41s ago). ... and maybe you won't get this only once, but 5 or even 10 times in a row. In short, you might lose precious time by trying to upload files which already are on the server. This is where X-DUPE comes in. X-DUPE stands for extended dupe information and makes the server send information about which files have been uploaded already to the client. The client can read this information and use it to remove files from its upload queue. It will not try to upload these dupes then and you'll spend the time on uploading instead of getting one dupe message after the other. ============================================================================== 2. How does it work? The bad news is that your ftp/fxp client has to support this feature. The good news is that pFtp (popular fxp client for Linux) already does. If you like the idea of this feature, you should encourage other FTP client authors to implement support for it in their programs. It should not be hard to implement and we're pretty confident support will be added if enough people ask for it. The idea behind it is very simple. When X-DUPE mode is turned in and you run into a dupe, you'll get an output like STOR blabla.r03 553- X-DUPE: blabla.rar blabla.r00 blabla.r01 blabla.r02 blabla.r03 553- X-DUPE: blabla.r04 blabla.r05 553- blabla.r03: This file looks like a dupe!! 553 It was uploaded by someone ( 0m 41s ago). Your FTP client checks those X-DUPE lines and uses them to remove those files from your upload queue. ============================================================================== 3. I'm author of a FTP/FXP client and want to support it. What do I have to do? To get those X-DUPE lines, you have to turn on extended dupe information mode first. This is done with the new "SITE XDUPE" ftp command. You should send this command somewhere between logging in and trying to upload a file. A server which does not have this feature will probably reply with something like "500 'SITE XDUPE 1': command not understood.". There are different modes of operation for the XDUPE command: 1) Sending "SITE XDUPE" without any parameters will show the current status. Example: > site xdupe 200 Extended dupe mode is disabled. Another example: > site xdupe 200 Extended dupe mode 1 is enabled. 2) Sending "SITE XDUPE 1" turns extended dupe checking on. Mode 1 means that in X-DUPE replies several filenames will be shown per line, up to a maximum of 80 chars total line length. Filenames longer than 66 chars will be truncated. Example: > site xdupe 1 200 Activated extended dupe mode 1. Example X-DUPE output: STOR blabla.r03 553- X-DUPE: blabla.rar blabla.r00 blabla.r01 blabla.r02 blabla.r03 blabla.r04 553- X-DUPE: blabla.r05 blabla.r06 553- blabla.r03: This file looks like a dupe!! 553 It was uploaded by someone ( 0m 41s ago). 3) Sending "SITE XDUPE 2" turns on extended dupe checking mode 2. In mode 2, the server will send only one filename per X-DUPE line, and the maximum line length is 80 chars. This means filenames longer than 66 chars are truncated as in mode 1. Example: > site xdupe 2 200 Activated extended dupe mode 2. > stor nanana03.zip 553- X-DUPE: nanana01.zip 553- X-DUPE: nanana02.zip 553- X-DUPE: nanana03.zip 553- X-DUPE: nanana04.zip 553- X-DUPE: nanana05.zip 553- X-DUPE: nanana06.zip 553- X-DUPE: nanana07.zip 553- X-DUPE: 01234567890123456789012345678901234567890123456789.zip 553- X-DUPE: 012345678901234567890123456789012345678901234567890123456789.zip 553- X-DUPE: 012345678901234567890123456789012345678901234567890123456789012345 553- X-DUPE: 012345678901234567890123456789012345678901234567890123456789012345 553 Error: You have no rights to overwrite in this directory. 4) Sending "SITE XDUPE 3" turns on extended dupe checking mode 3. In mode 3, only one filename is sent per X-DUPE line, and the filename will not be truncated. This means the line will be as long as the filename + the 553 prefix + the "X-DUPE" prefix. Use this mode only if your client is flexible enough to handle long lines. Example: > site xdupe 3 200 Activated extended dupe mode 3. > stor nanana03.zip 553- X-DUPE: nanana01.zip 553- X-DUPE: nanana02.zip 553- X-DUPE: nanana03.zip 553- X-DUPE: nanana04.zip 553- X-DUPE: nanana05.zip 553- X-DUPE: nanana06.zip 553- X-DUPE: nanana07.zip 553- X-DUPE: 01234567890123456789012345678901234567890123456789.zip 553- X-DUPE: 012345678901234567890123456789012345678901234567890123456789.zip 553- X-DUPE: 0123456789012345678901234567890123456789012345678901234567890123456789.zip 553- X-DUPE: 01234567890123456789012345678901234567890123456789012345678901234567890123456789.zip 553 Error: You have no rights to overwrite in this directory. 5) Sending "SITE XDUPE 4" turns on extended dupe checking mode 4. In mode 4, all files are listed on one long line, up to a maximum of 1024 characters. Files which are too long to fit will be skipped. 6) Sending "SITE XDUPE 0" turns extended dupe information off again. This is the default setting on server side. Thus, if you want any X-DUPE information at all, you have to set mode 1, 2 or 3 first. Basically X-DUPE lines can only be part of the possible 553 reply after a STOR command. The X-DUPE mode you choose mainly depends on which filenames you expect on the server and how flexible your client is with both parsing the X-DUPE lines and reading long server lines. All the files listed in X-DUPE lines should be removed from your upload queue. ============================================================================== 4. Acknowledgements Thanks go to ... Easy from TFCISO for demonstrating how much time can be lost when running into a lot of dupes and helping me getting this idea. HoE who contributed a lot by instantly picking up the idea and implementing it in his great Linux FXP client, pFtp. Usurper for enabling this feature for all sites by putting it into the official glFtpD release. ============================================================================== If you like the idea, ask FTP/FXP client authors to add support for it in their programs! -- Hoopy