/ 
upfiles-15
River stage zero No dependents
/ upfiles

NAME

upfiles -- upload files to FTP or SFTP server, for push mirroring

SYNOPSIS

upfiles [--options] [filename...]

DESCRIPTION

Upfiles uploads changed files from your local disk to an FTP or SFTP server, for a simple kind of "push" mirroring.

Create files locally with the same directory structure as the target, and in a ~/.upfiles.conf file give the locations,

upfiles (local  => '/my/directory',
         remote => 'ftp://fred@example.com/pub/fred');

This is actually Perl code, so you can put comment lines with #, write some conditionals, use $ENV{HOME}, etc. Then to upload run

upfiles

Or to upload just some selected files,

upfiles /my/directory/foo.txt
upfiles bar.txt ../something/quux.pl

Your username on the remote system is in the ftp:// remote URL. A password is taken from file ~/.netrc the same as for the ftp program and similar programs. See netrc(5) or Net::Netrc for the format of that file.

Note that FTP transmits username and password in clear text over the network. See "FTP Security" below for secure connections (or "SSH File Transfer (SFTP)").

upfiles records files sent in an SQLite3 database file .upfiles.sqdb in each local toplevel directory, for example /my/directory/.upfiles.sqdb. Local changes are identified by traversing the local tree and comparing file modtimes and sizes against what the database records was last sent. This is faster than asking the remote server what files etc it's got.

For convenience, some local files are always excluded from the upload. Currently these are

.upfiles.sqdb    which is upfiles itself
foo~             Emacs backups
#foo#            Emacs autosaves
.#foo            Emacs lockfiles

Files are uploaded one by one. The upload goes first to a temporary filename and is then renamed. This means an incomplete file isn't left if the connection is lost or upfiles is killed during transfer. Temporary files are noted in the database and leftovers are deleted on the next upfiles run if necessary.

File modification times are copied to an FTP server if it has the draft standard MFMT command or the common SITE UTIME extension (either 2-arg or 5-arg).

Plain RFC959 FTP doesn't have a notion of symlinks or hard links so upfiles follows any local links to actual content to upload. Perhaps in the future the SITE SYMLINK extension, if available, could be used for links within the uploaded tree. (Or SFTP below has symlinks.)

Filenames containing CR or LF characters (\r, \n) cannot be sent by FTP protocol and Upfiles will refuse to operate on them.

FTP Security

FTP can be used with SSL encryption and end-to-end destination verification in one of two ways. Either is highly recommended when supported by the server. Recent Net::FTP (version 1.28 circa 2014) and its SSL dependencies (IO::Socket::SSL) are required.

ftps is immediate SSL on both command and data channels. The server listens on a different port 990 than plain FTP (port 21). Give ftps in the remote URL,

upfiles (local  => '/my/directory',
         remote => 'ftps://fred@example.com/pub/fred');

Or, on the usual FTP connection a TLS command is given to encrypt the command channel (and data channels) from that point onwards. For this, give remote URL with plain ftp, plus use_TLS option,

upfiles (local   => '/my/directory',
         remote  => 'ftp://fred@example.com/pub/fred',
         use_TLS => 1);

Upfiles gives TLS immediately on connecting to the server, so both username and password are only sent after the connection is encrypted with SSL.

See "DEBUGGING" below on protocol tracing.

SSH File Transfer (SFTP)

The ssh "secure shell" program and protocol includes an SFTP file transfer. Upfiles can use this by setting the remote URL to sftp.

upfiles (local  => '/my/directory',
         remote => 'sftp://fred@example.com/home/fred/mydir');

This requires Perl Net::SFTP::Foreign and suitable ssh(1) program. The remote URL target directory is a full filesystem path, the same as seen in the ssh shell.

ssh has various user authentication methods. No-password operation can be had by putting your public key on the remote machine. For password operation, upfiles looks in ~/.netrc for a password by username and host similar to plain FTP. If not found there then the ssh program will query. (Not sure if querying is a good idea, might prefer to immediately abort.)

See "DEBUGGING" below on protocol tracing.

CONFIGURATION

Each upfiles call in ~/.upfiles.conf takes the following key/value parameters,

local (string)

The local directory to upload from.

remote (string)

The remote FTP or SFTP server to upload to, as a URL. The path in the URL is the target directory, and your username on the remote machine is included with "@" syntax, like

remote => 'ftp://fred@example.com/pub/fredsdir',

The "scheme" part can be ftp, ftps or sftp. A password is sought in ~/.netrc by username and machine name.

use_TLS (0 or 1, default 0)

For an ftp remote, if set to 1 then use the TLS command to give an SSL secure protocol with the server. This is highly recommended when supported by the server (or instead ftps or sftp too).

For ftps, this option is ignored, as the connection is secured before any commands at all.

exclude_regexps (arrayref of regexps)

Additional filenames to exclude. For example to exclude a local Makefile

upfiles (local => '/my/directory',
         remote => 'ftp://example.com/pub/fred',
         exclude_regexps => [ qr{/(^|/)[Mm]akefile$} ]);
copy_utime (0, 1, default "if_possible")

Whether to copy file modification times to the server. Usually this is desirable. The default "if_possible" means do so if the server supports it. 0 means don't try. 1 means must copy times.

For FTP, this uses the MFMT or SITE UTIME commands, which are widely available extensions to the basic FTP protocol. For SFTP, copying times is always possible.

COMMAND-LINE OPTIONS

The command line options are

-n, --dry-run

Show what would be uploaded or deleted on the server, but don't actually do anything.

upfiles -n
--help

Print some brief help information.

-V, --verbose, --verbose=N

Print some diagnostics about what's being done. With --verbose=2 or --verbose=3 print some technical details too.

upfiles --verbose
--version

Print the upfiles program version number. With --verbose also print the version numbers of some modules used.

DEBUGGING

The --verbose option above shows what upfiles is considering for uploads, and also enables various levels of debug or verbosity from Net::FTP (its Debug option), IO::Socket::SSL (its $DEBUG), and Net::SFTP::Foreign (-v to the ssh program). SSL and SSH diagnostics include information about certificates checked and encryption negotiated.

It can be helpful to check first that a connection to the remote works. For FTP this can be the usual ftp program, or even telnet (quit to exit).

ftp example.com
telnet example.com 21

Server support for TLS command can be checked in the usual way by the FEAT command (which in an ftp client might have to be sent by quot feat) and looking for AUTH TLS in its response. The initial text banner message often says something for human readers too. The use_TLS option should then work, or not, according to server support.

For ftps (FTP with SSL on port 990), if no server is listening on that port then the host might either immediately give "connection refused", or could ignore so that upfiles times-out. If a suitable SSL-enabled ftp program is not at hand then raw telnet equivalent though SSL can be had with either

openssl s_client -connect example.com:990
gnutls-cli --verbose -p 990 example.com

For SSH SFTP, check ssh works by running it directly (give remote username with -l). Its -v can be repeated for ever greater verbosity.

ssh -v -l fred example.com

On first connection to a new host, ssh will usually ask whether to accept the host certificate. This happens when run through upfiles too. Ideally you should have some separate independent way to validate or get a certificate so you know to whom you're speaking. Likewise SSL setups.

ENVIRONMENT VARIABLES

HOME

For ~/.upfiles.conf directory, and Net::Netrc for ~/.netrc.

FILES

~/.upfiles.conf

Configuration file.

~/.netrc

FTP password file.

/my/local/dir/.upfiles.sqdb

SQLite3 database of information about what has been sent so far from the tree /my/local/dir.

Upfiles determines the home directory ~ using File::HomeDir. Net::Netrc has something similar for the .netrc file, and looks also for _netrc.

BUGS

Changing a local file from a file to a directory or vice versa probably doesn't work very well. Remove it locally and upload to remove it at the remote, then create the new locally and upload again.

FTP and SFTP make a couple of round trip command/responses to the server for every file. When uploading many small files something streaming or parallel would be faster. The temp file and rename adds a round trip too, but is desirable so anyone looking doesn't see half a file. Perhaps an option could turn this off if not needed (such as upfiles to a remote backup). For reference, Net::SFTP::Foreign has its own atomic option in put(), but think it's still just a client side action so recording temporaries in the database is a better cleanup if interrupted.

The temporary files are named using the local $$ PID added to the target filename. This is enough to protect against simultaneous uploads from the same source machine, but potentially not if you're networked and are foolish enough to upfiles the same tree simultaneously from two different source machines. For FTP, havoc probably ensues. For SFTP, no-overwrite is usedso should make an error. FTP STOU would guarantee uniqueness, but does it have a window while the name comes back which if interrupted could leave the file created but unknown to upfiles? Net::FTP put_unique() doesn't return the name until the transfer completes too.

For FTP, there's a small chance of a 2-arg SITE UTIME attempt looking like 5-arg to a pure-ftpd pre-1.0.33 server (circa 2011), if the filename uploaded happens to be dates and "UTC". Perhaps more care could be taken to identify the SITE UTIME style the server expects. In practice such filenames should be unlikely, and there's no problem with new enough pure-ftpd which has MDTM.

There's no support for a proxy connection to the remote system. The squid CONNECT style which gives a pass-through socket wouldn't be too hard, and ssh can be made to speak through such things too. The http-only style, where http PUT must be given for FTP upload, is irritating and makes the proxy a man-in-the-middle which you have to trust. If a http PUT upload method was of interest then such proxying might follow from that easily enough.

File sizes and totals are reported in kbytes (of 1024 bytes). Probably should jump to mbytes or even gbytes at some sensible threshold.

For a given local directory, if the remote URL is changed then upfiles will want to send everything again. This is good if the remote is genuinely somewhere new, or if uploading a local directory to multiple remote places. But if it's just a change of hostname or protocol then would want no resending. Hopefully this is rare. There's a secret undocumented maybe-working --catchup for marking everything already sent. Some sort of force option to resend particular files could be useful too.

The .upfiles.sqdb database records fake timestamps for directories just to indicate whether it has been created on the remote yet. Currently this is time "1 second" in the system epoch (1 Jan 1970 for Unix, whatever 1 Jan 1904 for MacOS). Alas this makes the database contents system-dependent. Hopefully cross-system operation will be rare and the effect on moving systems no worse than some repeated MKDs. Directory stamps are only actually for whether the directory has been created yet or not, so the intention is to treat any stamp value as meaning created.

SEE ALSO

Net::FTP, Net::SFTP::Foreign, ssh(1), netrc(5), Net::Netrc, DBD::SQLite, sqlite3(1)

sitecopy(1), ftpmirror(1), ftp-upload(1), rsync(1)

HOME PAGE

http://user42.tuxfamily.org/upfiles/index.html

(Upfiles is good for uploading to tuxfamily, either FTP or SFTP.)

LICENSE

Copyright 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2020 Kevin Ryde

Upfiles is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3, or (at your option) any later version.

Upfiles is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with Upfiles. If not, see http://www.gnu.org/licenses/.