Thursday 4 December 2014

Faster vanity key mining with GnuPG

My last post described my work making GnuPG look for keys with memorable keyids. It works, but it's slow, presumably because a lot of work gets repeated between each test, and because each test is expensive. With regular expressions and system calls in the mining loop, GnuPG doesn't get to spend much time computing hashes over the generated public key.

But not anymore! I have a better patch now, that allows GnuPG to spend more time hashing and less time doing things which don't contribute to finding a desired keyid. The kernel / user space context switch is now amortized over millions of hashes, and the loop mining for hashes is tight: nearly all it does now is hashing the public key material.

diff --git a/g10/keygen.c b/g10/keygen.c index 8c3e9f6..a250914 100644 --- a/g10/keygen.c +++ b/g10/keygen.c @@ -19,6 +19,7 @@ */ #include <config.h> +#include <fcntl.h> #include <stdio.h> #include <stdlib.h> #include <string.h> @@ -120,6 +121,7 @@ struct opaque_data_usage_and_pk { PKT_public_key *pk; }; +extern int delta; static int prefs_initialized = 0; static byte sym_prefs[MAX_PREFS]; @@ -1424,6 +1453,7 @@ gen_rsa (int algo, unsigned nbits, KBNODE pub_root, KBNODE sec_root, DEK *dek, STRING2KEY *s2k, PKT_secret_key **ret_sk, u32 timestamp, u32 expireval, int is_subkey) { + extern int searchmode; int rc; PACKET *pkt; PKT_secret_key *sk; @@ -1463,7 +1493,37 @@ gen_rsa (int algo, unsigned nbits, KBNODE pub_root, KBNODE sec_root, DEK *dek, sk = xmalloc_clear( sizeof *sk ); pk = xmalloc_clear( sizeof *pk ); - sk->timestamp = pk->timestamp = timestamp; + if (getenv("GNUPG_KEYSEARCH")) { + switch (fork()) { + int status; + case -1: + /* Fall back to normal behaviour. */ + break; + case 0: + /* Set search mode and continue executing. */ + searchmode = 1; + break; + default: + wait(&status); + log_debug("Child returned %d\n", WEXITSTATUS(status)); + if (WEXITSTATUS(status)) { + /* Can't find a matching key - just continue. */ + } else { + /* Key matches - fudge timestamp. */ + int fd; + fd = open("/tmp/keysearch", O_RDONLY); + if (fd != -1) { + read(fd, &delta, sizeof (delta)); + close(fd); + log_debug("Continuing with delta=%d\n", delta); + } else { + log_debug("Couldn't open /tmp/keysearch\n"); + } + } + break; + } + } + sk->timestamp = pk->timestamp = timestamp + delta; sk->version = pk->version = 4; if (expireval) { diff --git a/g10/keyid.c b/g10/keyid.c index d7a877b..4d5d370 100644 --- a/g10/keyid.c +++ b/g10/keyid.c @@ -19,6 +19,7 @@ */ #include <config.h> +#include <fcntl.h> #include <stdio.h> #include <stdlib.h> #include <string.h> @@ -124,15 +125,63 @@ hash_public_key( gcry_md_hd_t md, PKT_public_key *pk ) } } -static gcry_md_hd_t +int searchmode = 0; + +gcry_md_hd_t do_fingerprint_md( PKT_public_key *pk ) { gcry_md_hd_t md; + int delta; + const byte *dp; + unsigned char keyid[8]; + char const *keysearch = getenv("GNUPG_KEYSEARCH"); + int keysearch_len = keysearch ? strlen(keysearch) : 0; + int spread = searchmode && keysearch ? 33554432-1 : 0; + u32 original_timestamp = pk->timestamp; - if (gcry_md_open (&md, DIGEST_ALGO_SHA1, 0)) - BUG (); - hash_public_key(md,pk); - gcry_md_final( md ); + if (searchmode && keysearch) { + int i; + + memset(keyid, 0, sizeof (keyid)); + for (i = 0; i < keysearch_len/2; i++) { + sscanf(keysearch + i*2 + keysearch_len%2, "%02hhx", keyid+i); + } + log_debug("Searching for key fingerprints ending in %02X%02X%02X%02X%02X%02X%02X%02X\n", + keyid[0], keyid[1], keyid[2], keyid[3], keyid[4], keyid[5], keyid[6], keyid[7]); + } + for (delta = -spread; delta <= 0; delta++) { + if (gcry_md_open (&md, DIGEST_ALGO_SHA1, 0)) + BUG (); + pk->timestamp = original_timestamp + delta; + hash_public_key(md,pk); + gcry_md_final( md ); + if (searchmode && keysearch) { + dp = gcry_md_read(md, 0); + if (memcmp(dp + 20 - keysearch_len/2, keyid, keysearch_len/2)) { + if (delta % 1000000 == 0) { + log_debug("Trying delta = %d\n", delta); + } + } else { + int fd; + fd = open("/tmp/keysearch", O_CREAT|O_WRONLY, 0666); + if (fd == -1) { + log_debug("Couldn't open /tmp/keysearch\n"); + continue; + } + write(fd, &delta, sizeof (delta)); + close(fd); + log_debug("Found delta = %d\n", delta); + exit(0); + } + gcry_md_close(md); + } + } + + if (searchmode) { + /* Failed to find a matching key. */ + log_debug("Never found a matching key\n"); + exit(1); + } return md; }

I now have 0x5EAF00D5 and 0x0B5E55ED, and I'm not sure what else I want to mine. Maybe I'll just stick with 0xDEC0DED1 for now.

Saturday 25 October 2014

OpenPGP vanity keys with a patched GnuPG

I'm vain, but also lazy to remember long strings of arbitrary hexadecimal digits. Generating vanity keys for GnuPG would address both needs (wants): a vanity key fingerprint would be meaningful, and therefore more memorable.

What is a memorable key fingerprint? This is not one:
pub 1024D/0DA1756824EEB426 2001-01-15 [expired: 2006-01-14] uid Bernd Jendrissek (or owner of this keypair) <Bernd.Jendrissek@mailbox.co.za> uid Bernd Jendrissek <Bernd.Jendrissek@mailbox.co.za>
That is one of my old keys, and it happens that I did memorize at least the short-form keyid - but what a nerdy thing to do. And not something I can expect other people to remember - to them it's just an arbitrary hex string, with little personal relevance.

The obvious solution is to generate a key whose fingerprint carries some meaning, that's easier to remember. A word maybe, spelled in l33tspeak, or some recognizeable number like pi or e (a shibboleth for fellow nerds).

Something like this:
pub 4096R/FAC02779DEC0DED1 2014-01-21 [expires: 2024-01-19] uid Bernd Jendrissek

That's my now-current key that should have propagated to a keyserver near you.

By what black magic did I do this? It took me a while to get to a solution that works well enough to find worthwhile vanity keys. At first I just ran gpg in a shell loop, generating keys one by one. But that's ridiculously slow - it generates a whole new key from scratch each time, which exhausts the entropy pool (arguably a bug in GnuPG - /dev/urandom is really good enough) and does a whole bunch of CPU-intensive computing for every key.

Unfortunately, it takes minutes for standard GnuPG to generate just one key (although most of that is probably in waiting for more entropy). I'd be dead 500 times over before I had a good sampling of word-like key fingerprints. No, I would have to get my hands dirty by patching GnuPG and the libraries it uses. Here's one for libgcrypt:
diff --git a/cipher/rsa.c b/cipher/rsa.c index ccc9f96..319c001 100644 --- a/cipher/rsa.c +++ b/cipher/rsa.c @@ -177,6 +177,7 @@ check_exponent (void *arg, gcry_mpi_t a) * TRANSIENT_KEY: If true, generate the primes using the standard RNG. * Returns: 2 structures filled with all needed values */ +int delta __attribute__((visibility("default"))); static gpg_err_code_t generate_std (RSA_secret_key *sk, unsigned int nbits, unsigned long use_e, int transient_key) @@ -324,6 +328,26 @@ generate_std (RSA_secret_key *sk, unsigned int nbits, unsigned long use_e, return GPG_ERR_SELFTEST_FAILED; } + if (getenv("LIBGCRYPT_MINE")) { + int spread = atoi(getenv("LIBGCRYPT_MINE")); + for (delta = -spread; delta <= 0; delta++) { + switch (fork()) { + case -1: + log_debug("failed to fork\n"); + exit(1); + break; + case 0: + /* Child. */ + return 0; + default: + /* Parent. */ + wait(NULL); + break; + } + } + exit(0); + } + return 0; } diff --git a/src/libgcrypt.vers b/src/libgcrypt.vers index 5a617cc..fd18fc2 100644 --- a/src/libgcrypt.vers +++ b/src/libgcrypt.vers @@ -104,6 +104,7 @@ GCRYPT_1.2 { gcry_mpi_set_ui; gcry_mpi_snew; gcry_mpi_sub; gcry_mpi_sub_ui; gcry_mpi_subm; gcry_mpi_swap; gcry_mpi_test_bit; gcry_mpi_lshift; +delta; local: *;
This bit of code forks the process after it has generated a precious prime pair and exposes a different fudge value (delta) which gets used later in GnuPG proper. Speaking of which:
diff --git a/g10/keygen.c b/g10/keygen.c index 8c3e9f6..64cc2d6 100644 --- a/g10/keygen.c +++ b/g10/keygen.c @@ -120,6 +120,7 @@ struct opaque_data_usage_and_pk { PKT_public_key *pk; }; +extern int delta; static int prefs_initialized = 0; static byte sym_prefs[MAX_PREFS]; @@ -1435,7 +1463,7 @@ gen_rsa (int algo, unsigned nbits, KBNODE pub_root, KBNODE sec_root, DEK *dek, if (!nbits) nbits = DEFAULT_STD_KEYSIZE; - if (nbits < 1024) + if (!getenv("GNUPG_MINE") && nbits < 1024) /* XXX I'm not sure if this is important. */ { nbits = 1024; log_info (_("keysize invalid; using %u bits\n"), nbits ); @@ -1463,7 +1491,7 @@ gen_rsa (int algo, unsigned nbits, KBNODE pub_root, KBNODE sec_root, DEK *dek, sk = xmalloc_clear( sizeof *sk ); pk = xmalloc_clear( sizeof *pk ); - sk->timestamp = pk->timestamp = timestamp; + sk->timestamp = pk->timestamp = timestamp + delta; sk->version = pk->version = 4; if (expireval) { @@ -3350,6 +3378,65 @@ start_tree(KBNODE *tree) delete_kbnode(*tree); } +static char const *check_vanity(KBNODE pub_root, char keyid[static 17]) +{ + PKT_public_key *pk; + gcry_md_hd_t md; + + gcry_md_hd_t do_fingerprint_md(PKT_public_key *pk); + + if (pub_root) { + pk = find_kbnode (pub_root, PKT_PUBLIC_KEY)->pkt->pkt.public_key; + md = do_fingerprint_md(pk); + } + if (!pub_root || md) { + const byte *dp; + char const *vanity_patterns[] = { + /* Simple repetitions and sequences. */ + "(.)\\1.*\\1\\1.*\\1\\1", + "(.)\\1\\1.*\\1.*(.)\\2\\2|(.)\\1\\1.*(.).*\\2\\2\\2", + "(.)\\1\\1\\1$", + "123456|234567|345678|456789|56789A|6789AB|789ABC|89ABCD|9ABCDE|ABCDEF", + /* Math/physics constants. */ + "314159|271828|60221|1380[67]|8314[45]|299792|6626[01]|66738|9109[34]|40490FDB|27315", + /* Biographical details. */ + "^(..)*1976|^(..)*760518|^(..)*1805|^(..)*5231080", + /* 13375p34k. */ + "^([05ABCDEF]*[0AE]){2,}[05ABCDEF]*$", + "CA5CADE|0DE55A|ACCE55|A55E55|BA0BAB|CA5CADE|DECADE|DEC0DE|D0D0E5|FACADE|0B5E55|5EABED|5EAF00D|5EED", + /* Former keys. */ + "24EEB426|FCDE4B8C|804177F8|AAE76E8E|D7CBA633|D5B8044C|1726CB60", + /* Famous words. */ + "DEADBEEF|C0FFEE", + /* ASCII. */ + "^(2[0DEF]|3[0-9F]|[46].|[57][0-A])*$", + "^(..)*[46]2[46]5[57]2[46]e[46]4|[46]2[57]0[46]a|[46]a[46]5[46]e[46]4[57]2[46]9[57]3", + NULL + }; + int i; + int check_regexp(const char *expr,const char *string, int sanitize); + + if (pub_root) { + dp = gcry_md_read(md, 0); + + sprintf(keyid, "%02X%02X%02X%02X%02X%02X%02X%02X", + dp[12], dp[13], dp[14], dp[15], dp[16], dp[17], dp[18], dp[19]); + } else { + keyid[0] = 0; + } + + for (i = 0; vanity_patterns[i]; i++) { + if (check_regexp(vanity_patterns[i], keyid, 0)) { + return vanity_patterns[i]; + } + } + if (!vanity_patterns[i]) { + return NULL; + } + } + + return NULL; +} static void do_generate_keypair (struct para_data_s *para, @@ -3363,6 +3450,7 @@ do_generate_keypair (struct para_data_s *para, int rc; int did_sub = 0; u32 timestamp; + char keyid[17]; if( outctrl->dryrun ) { @@ -3453,13 +3541,15 @@ do_generate_keypair (struct para_data_s *para, linked list. The first packet is a dummy packet which we flag as deleted. The very first packet must always be a KEY packet. */ - start_tree (&pub_root); - start_tree (&sec_root); - timestamp = get_parameter_u32 (para, pKEYCREATIONDATE); if (!timestamp) timestamp = make_timestamp (); + check_vanity(NULL, keyid); + + start_tree (&pub_root); + start_tree (&sec_root); + /* Note that, depending on the backend (i.e. the used scdaemon version), the card key generation may update TIMESTAMP for each key. Thus we need to pass TIMESTAMP to all signing function to @@ -3492,6 +3582,17 @@ do_generate_keypair (struct para_data_s *para, } } + /* Check key for vanity value. */ + if (!rc && getenv("GNUPG_MINE")) { + char const *matching_pattern = check_vanity(pub_root, keyid); + if (matching_pattern) { + log_debug("Found key %s with %s, delta = %d\n", + keyid, matching_pattern, delta); + } else { + exit(0); + } + } + if(!rc && (revkey=get_parameter_revkey(para,pREVOKER))) { rc = write_direct_sig (pub_root, pub_root, pri_sk, revkey, timestamp); @@ -3637,7 +3751,7 @@ do_generate_keypair (struct para_data_s *para, keydb_release (pub_hd); keydb_release (sec_hd); - if (!rc) + if (!rc && !getenv("GNUPG_MINE")) { int no_enc_rsa; PKT_public_key *pk; diff --git a/g10/trustdb.c b/g10/trustdb.c index dedb18c..f8c2dda 100644 --- a/g10/trustdb.c +++ b/g10/trustdb.c @@ -1852,8 +1852,8 @@ sanitize_regexp(const char *old) /* Used by validate_one_keyblock to confirm a regexp within a trust signature. Returns 1 for match, and 0 for no match or regex error. */ -static int -check_regexp(const char *expr,const char *string) +int +check_regexp(const char *expr,const char *string, int sanitize) { #ifdef DISABLE_REGEX /* When DISABLE_REGEX is defined, assume all regexps do not @@ -1863,19 +1863,59 @@ check_regexp(const char *expr,const char *string) int ret; char *regexp; - regexp=sanitize_regexp(expr); + regexp=sanitize?sanitize_regexp(expr):xstrdup(expr); #ifdef __riscos__ ret=riscos_check_regexp(expr, string, DBG_TRUST); #else { + static struct { + char *pattern; + regex_t pat; + } pat_cache[100]; + static int initialized = 0; regex_t pat; + int i, i_empty; + + if (!initialized) { + for (i = 0; i < 100; i++) { + pat_cache[i].pattern = 0; + } + initialized = 1; + } + + for (i = 0, i_empty = -1; i < 100; i++) { + if (i_empty < 0 && !pat_cache[i].pattern) { + i_empty = i; + } + if (strcmp(pat_cache[i].pattern, regexp) == 0) { + break; + } + } - ret=regcomp(&pat,regexp,REG_ICASE|REG_NOSUB|REG_EXTENDED); + if (i < 100) { + memcpy(&pat, &pat_cache[i].pat, sizeof (pat)); + ret = 0; + } else { + ret=regcomp(&pat,regexp,REG_ICASE|REG_NOSUB|REG_EXTENDED); + } if(ret==0) { + if (i_empty < 0) { + for (; i < 100; i++) { + if (!pat_cache[i].pattern) { + i_empty = i; + } + } + } + if (i_empty >= 0) { + pat_cache[i_empty].pattern = xstrdup(regexp); + memcpy(&pat_cache[i_empty].pat, &pat, sizeof (pat)); + } ret=regexec(&pat,string,0,NULL,0); - regfree(&pat); + if (i_empty < 0) { + regfree(&pat); + } ret=(ret==0); } } @@ -1970,7 +2010,7 @@ validate_one_keyblock (KBNODE kb, struct key_item *klist, || opt.trust_model != TM_PGP || (uidnode && check_regexp(kr->trust_regexp, - uidnode->pkt->pkt.user_id->name)))) + uidnode->pkt->pkt.user_id->name, 1)))) { /* Are we part of a trust sig chain? We always favor the latest trust sig, rather than the greater or
This bit uses the fudge value set earlier in each variant process. (I'm lucky here in that GnuPG's keyring seems to be concurrency-safe - although the loop does wait for each child process to finish, appending its key to the keyring, before continuing to fork more child processes.) The fudge value is a delta that modifies the apparent timestamp of the key, adjusting it by up to $LIBGCRYPT_MINE seconds into the past. The timestamp affects the key fingerprint, so this is an easy way to diversify the single pair of primes that gets reused usefully up to thousands of times. (Beyond that, super-linear scaling in GnuPG's keyring access imposes the law of diminishing returns, and it's better to just generate a new pair of primes and write the resulting keys into a fresh keyring.)

The result is a stream of fast, identical keys (with just a differing timestamp), which I then filter with a few regular expressions (compiling them just once, in the parent process, and caching the compiled form for re-use in the child processes). to get a slightly slower stream of candidate keys whose fingerprint is, in some way, memorable and/or meaningful.

I chose one (FAC02779DEC0DED1) fairly arbitrarily - there were several others that took me a long while to reject. I don't need them anymore, so if anyone wants them, I'll sell them for 5mBTC per unique key. You really shouldn't take up this offer though, since these keys would come to you pre-compromised: I have access to the private keys. You'd have to just take my word for it that I'll delete each keyring that I can sell. It's really just for fun, a novelty that you could use for unimportant things. A few cool ones from a total of more than a million:
8F0230444444444C A8111111111C2A72 AB4F37D43CA5CADE 5F2833DEAD52BEEF BB3418DC0FFEEBAD AAA3B57C1CADA555 E30E7A290B314159 9CBC000011112CE1 1251D7C715EAF00D

Saturday 18 October 2014

Indeterminate hash table traversal

On and off for a few weeks I've been trying to figure out why the gnetlist testsuite has been failing. I could've sworn that I regenerated the golden output two years ago and painstakingly checked that the differences were only in the order of output with the help of a special-purpose script.

I hadn't run the testsuite in a while, so it's likely that this was the first time I've run it since I got a new hard disk and started using a newer OS. (I'm using debian jessie now.) When I looked at the differences between the golden outputs and the currently produced outputs, it seemed clear that the differences were only in the order of output, and not due to the output being incorrect. At least, not more incorrect than the golden output.

Think things through, don't draw hasty conclusions



I pretty quickly focused on the possibility that GLib's hash table implementation had changed between Ubuntu 10.10's version and jessie's. But at first I was convinced that there was no material change in the hash table implementation: all the hash table stuff was still doing the same thing, albeit in a more cache-friendly way. I even copied GLib's ghash.c from version 2.26.1 (the one that Ubuntu 10.10 used), renamed a few functions and used those instead of the jessie-native GLib's functions, but still the output stayed in the different order.

Still, I was convinced that there must have been a change in hash table traversal order. What else could be reordering the output? I've barely worked on gEDA since getting my new hard disk, and in any case I also ran the testsuite of the very commit that last touched the golden outputs: still reordered output.

The simplicity (and reliability) of blunt tools



My next step was to use a blunt tool: duplicating more exactly the Ubuntu 10.10 environment. I downloaded the relevant debs and unpacked them in a staging directory, and built gEDA against those versions of GLib and a few others. (That required editing the pkg-config package metadata files to point at locations within the staging directory, then setting $PKG_CONFIG_PATH and also using some LDFLAGS=... and $LD_LIBRARY_PATH trickery.)

Finally! Linking to the Ubuntu 10.10 version of GLib reproduced the golden output. That meant that the change in output was a result of the change in computing environment, not some subtle bug I've introduced with my patches. Now I can confidently make some more order-changing patches (to decouple program output order from hash table implementation). Going from a known-good to a presumed-good state is a lot less scary than going from unknown-if-good to presumed-no-worse. I'll probably still check the regenerated golden output with my order-aware diff-checking script, just to be sure that I don't add bugs.

Mystery solved



With the gEDA source code validated (it now passes the testsuite, given the right libraries), I still want to satisfy my curiosity: why did the output order change, when it seemed like the GLib hash table implementation hadn't changed? Because, well, it had, and I had only overlooked what had changed: the hashing function itself, g_str_hash, that transforms a string to a hash value.

What I failed to realize during my first diff-hunt was that GLib 2.26.1's g_str_hash lived in a different file than the rest of the hash table implementation, so I simply didn't see the old version (and never noticed it was missing) while looking at changes to glib/ghash.c.

Don't depend on the order of hash table traversal!

Tuesday 24 June 2014

Words from football matches

So I'm football-illiterate; I thought that FIFA's 3-letter abbreviation for Nigeria was NIG, but it's actually NGA. (NIG is Niger.) So we won't be seeing the delightfully offensive NIG - GER scoreboard during this World Cup.

We can still search for other words. Grepping for the Cartesian product of all participating countries in /usr/share/dict/words yields a few matches, although none are complete words, let alone offensive ones:
  • apalaCHICOLa
  • hITACHI
  • siNGAPORe
  • BELCHIng
  • ganGRENED
  • GREENGrocer
  • maCHINED
  • miCROCHIp
Chile features a lot!

Wednesday 30 April 2014

Getting my old Thunderbird profile into Icedove

Do we have to play these branding games with directory names - especially hidden-by-convention ones?

I just wasted half an hour or so trying to coax Icedove into seeing the account that I used with Thunderbird just a month or two ago. So it could just magically see it, or even just permit me to import the old folder into a new one. Whatever, I just wanted my mail!

Making software easy to use for non-geeks is a laudable goal, but I don't think you reach that goal by disappearing so many control elements that the configuration dialogs appear empty. It reminds me of something one of our computer science lecturers said once, long ago: "Emacs is an operating system masquerading as an editor; Vi is an editor masquerading as nothing." "Nothing" is not a good user interface. It has no affordances, and so while it might not be an intimidating interface, it is undiscoverable (yay, start clicking random elements whose appearance suggest little about their function).

After some helpless flailing around and a quick look through some unhelpful forum threads (I swear, they're becoming less and less useful with every passing year as each generation of cargo cult users passes down its placebo magic spells to the next) I got some help on Freenode, in #debian:
<mtn> berndj-blackout: rename the folder to .icedove ;)
TL;DR at this point: doing exactly that solved my problem.

Unfortunately I had already started icedove before I got to IRC. So when I blindly did
$ mv ~/.thunderbird ~/.icedove
hoping it would simply rename the directory, it actually moved ~/.thunderbird into ~/.icedove, since by then the latter directory already existed, so mv(1) interprets the command as a request to put the source into the destination as a subdirectory:
stat("/tmp/a", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 lstat("/tmp/b", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 lstat("/tmp/a/b", 0x7fff31da12e0) = -1 ENOENT (No such file or directory) rename("/tmp/b", "/tmp/a/b") = 0
(Can you spot the race condition? You can't use mv(1) as a synchronization primitive.)

It took me a while to realize what had happened. This was the clue:
$ du -sh ~/.icedove/2qeibe1d.default/ 16M /home/berndj/.icedove/2qeibe1d.default/ $ du -sh ~/.icedove/ 5.7G /home/berndj/.icedove/
Where are those 5.7GB?Hiding in ~/.icedove/.thunderbird of course!

Wednesday 5 March 2014

Simulating filesystem errors with gdb

A prospective client needs to get a bunch of files from in-field gadgets onto the Internet. s3fs / s3fuse seem to be a convenient way to get the files onto Amazon's S3. The application demands that the in-field gadget keep retrying until it knows that a file has finally reached the mothership; to do this, I am to write a program that does the copying, that is adequately paranoid about possible failure modes.

One easily-overlooked failure mode is that close(2) can fail. Earlier write(2) operations can appear to succeed, because they may simply be writing the data to local buffers, not yet checking if the data has reached the other side of the network.

My initial assumption was that moving the files across the network using a shell script would fail to take care of all the weird corner cases, such as error-on-close. Why speculate though, when we have gdb?
$ gdb /bin/cp GNU gdb (GDB) 7.2-ubuntu Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /bin/cp...(no debugging symbols found)...done. (gdb) break close Breakpoint 1 at 0x402930 (gdb) run /tmp/a /tmp/c Starting program: /bin/cp /tmp/a /tmp/c Breakpoint 1, close () at ../sysdeps/unix/syscall-template.S:82 82 ../sysdeps/unix/syscall-template.S: No such file or directory. in ../sysdeps/unix/syscall-template.S (gdb) cont Continuing. ... Breakpoint 1, close () at ../sysdeps/unix/syscall-template.S:82 82 in ../sysdeps/unix/syscall-template.S (gdb) p errno $1 = 0 (gdb) set errno = 5 (gdb) p errno $2 = 5 (gdb) return -1 Return value type not available for selected stack frame. Please use an explicit cast of the value to return. (gdb) return (int) -1 Make selected stack frame return now? (y or n) y #0 0x0000000000406697 in ?? () (gdb) cont Continuing. /bin/cp: closing `/tmp/a': Input/output error Program exited with code 01.

Bingo! It does work (err, it does fail?) I still have some homework to do: is it necessary to use fsync(2) before closing the file, in order to really make sure that all pending errors get reported? That's one thing that cp(1) doesn't do, according to strace(1).