Building upon our previous adventure with SSH file transfers, where we discovered a clever method to bypass SCP for single files, today we're transferring multiple files and even directories.

Last time, I shared a neat trick for moving individual files using gzip and base64 within SSH sessions. Now, if you have multiple files, it's quite a bit of work to do this one by one. Today, we are improving on the technique to efficiently handle multiple files:

1. Tar, Compress, Convert, Copy

In comparison to the single-file method, not much changes:

tar -cf - *.conf| gzip -c | base64 -w 0; echo
# or if you are on a modern system, use xz
tar -cf - *.conf| xz | base64 -w 0; echo

The above one-liner would compress all files ending with .conf . You could also list the files explicitly and even add directories.

This works mostly thanks to the tar command, as it takes care of packing everything into an uncompressed archive. Compression is then done by gzip (or xz).

A note on macOS: macOS ships with a base64 utility that does not support the -w 0 flag. Simply leave it off, it's not needed.

Copy the resulting string into your clipboard and head on to step 2.

2. Paste, Decode, Decompress, Untar

Over at your destination, the reversal is also quite short.

Important: The command below overwrites existing files without asking. Make sure you are in the correct directory, or adjust the tar command at the end.

echo "PASTED_BASE64_STRING" | base64 --decode | gunzip | tar -xf -
# or if you've used xz
echo "PASTED_BASE64_STRING" | base64 --decode | unxz | tar -xf -

Voila! The files are decoded, decompressed, and extracted right where you need them.

Pro Tip: Verifying Integrity of Multiple Files

Don’t forget to check if your files made the journey intact! A quick way is to compare checksums between source and destination. You can find how to do just that in the other article.

Wrapping Up

Gone are the days of cumbersome file transfers. At least for small files and directories. While this might not be necessary when using tools like Ansible, Chef, Puppet or even Kubernetes, it is still a good tool to keep in the devops/sysadmin toolbelt. Especially when working directly on servers.

Other Posts

  • When Homebrew's Postgres Upgrade Fails

    Internally we upgraded our development machines from Postgresql 12 to version 13. Things went sideways because we also use Postgis in a lot of our projects. This is what to do when this happens.