Writing to Amazon S3 storage from MOVEit Automation
UPDATE: Please note this version has been superseded by MOVEit Automation 2018, which delivers S3 connectors out of the box. Please read this technical top tip, which includes everything you need to know:
MOVEit Automation unfortunately does not contain a connector for the increasingly popular Amazon S3 storage; it does however contain the always useful ‘Command Line App’ script so you can upload files into Amazon S3 storage from MOVEit Automation by following the steps below.
In essence, any command that you can issue from a command prompt, you can also issue from the script.
The first thing to do is to download and install the Amazon CLI (command line interface). The 64bit version of this can be found at https://s3.amazonaws.com/aws-cli/AWSCLI64.msi
Download and run the executable, afterwards you will need to configure your connection. Although the installation is machine specific, the configuration needs to be performed for each user who will use the CLI – in our case, this will be the service account that MOVEit Automation is running under.
To configure the CLI, run a command prompt as the service account and issue this command:
You will be prompted to enter both your access key and secret access key, everything else can be left as default. If you do not have access keys, you will need to create a new account in IAM.
You are now ready to write to AWS S3 storage. As you’ve configured your account already, you don’t need to worry about credentials. All that you need to know is the name of the bucket that you intend to write to. In this example, the bucket has been named www.pro2col-aws-s3.com.
We will test writing a file to the S3 bucket first by using the command prompt that you opened using the MOVEit Automation service account.
This is the command to use to write the file ‘test.txt’ into the folder named ‘home’ in the bucket:
aws s3 cp test.txt s3://www.pro2col-aws-s3.com/home/
This gives this result:
upload: .\test.txt to s3://www.pro2col-aws-s3.com/home/test.txt
Once we have seen that it works, we just need to wrap it up in a MOVEit Automation task. This is an example task we have created:
The download step is completely standard and could be taking files from anywhere. The Command Line script needs to ‘behave as a destination’ and has two parameters.
The first is the full path to the aws.exe program:
The second is the parameter string:
The parameter is this:
s3 cp [InputFile] “s3://www.pro2col-aws-s3.com/home/[OrigName]”
It’s important to use the copy (‘cp’) command rather than move as Automation will want to clean the file up itself. The “[InputFile]” macro contains the name of the source file as it appears in the cache. Be sure to enclose the target path in quotes if there are any spaces in either the path or file name.
You’ll notice I have used the “[OrigName]” macro at the end of the filename; if you do not specify a name (either in full or by macro inference), the temporary cache name will be used. Executing the task with debug on give output similar to the following:
2017-06-16 09:47:02.33 34: Running script Command Line App on 'c:/temp/test.txt' 2017-06-16 09:47:02.33 34: Task Upload to S3 !: Original AppParms='s3 cp [InputFile] "s3://www.pro2col-aws-s3.com/home/test.txt"' 2017-06-16 09:47:02.33 34: Task Upload to S3 !: AltReturnCodes=',0,' 2017-06-16 09:47:02.33 34: Task Upload to S3 !: Parsed AppParms='s3 cp C:\TEMP\MIC\34-0014\atc10006.tmp "s3://www.pro2col-aws-s3.com/home/test.txt"' 2017-06-16 09:47:02.33 34: Task Upload to S3 !: SafeAppPath='C:\PROGRA~1\Amazon\AWSCLI\AWS.exe' 2017-06-16 09:47:02.33 34: Task Upload to S3 !: FullCommandLine='C:\PROGRA~1\Amazon\AWSCLI\AWS.exe s3 cp C:\TEMP\MIC\34-0014\atc10006.tmp "s3://www.pro2col-aws-s3.com/home/test.txt"' 2017-06-16 09:47:02.33 34: Created process ID 2464 2017-06-16 09:47:04.79 34: Task Upload to S3 !: Command completed with a successful return code. (#0) 2017-06-16 09:47:04.79 34: $$ Logging c:/temp/test.txt to size 0 err 0 2017-06-16 09:47:04.79 34: Ran script Command Line App OK on C:\TEMP\MIC\34-0014\atc10006.tmp
Need Additional UK-Based MOVEit Training and Support?
Not only are Pro2col a certified Gold Partner, Ipswitch named us as their UK File Transfer Partner of the Year! Our MOVEit technical team are here to help you get the most out of your solution.