Ask Your Question
0

How to integrate Puppet with AWS S3 ?

asked 2015-09-24 04:15:14 -0600

Ryuzaki gravatar image

Hello, I am new to Puppet and would like to understand the process of integrating Puppet with AWS S3.

Basically, I want to be able to perform upload/download operation. I found out a module on Forge (malnick/s3 ) but it only supports downloading from S3, not uploading. I need to be able to do both.

Any ideas ?

-Ryuzaki

edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted
1

answered 2015-09-24 12:19:44 -0600

GregLarkin gravatar image

updated 2015-09-25 10:40:45 -0600

Hi Ryuzaki,

Puppet is designed to configure the state of server in your infrastructure, so the malnick/s3 module can be used to fetch a file from S3 that needs to be installed on your server for it to function correctly. That might be a .tar.gz file, .war file or something along those lines.

Because uploading a file to S3 does not typically contribute to the process of configuring a server, there's no core function for doing so and likely no custom module either.

If this really is a requirement for configuring your server, I would suggest doing something like:

  • Install an S3 command line toolset with uploading support
  • Declare an exec resource that invokes the S3 upload script with the proper command line arguments

Hope that helps!

edit flag offensive delete link more

Comments

Thanks, Greg. Yeah, that indeed helped. I guess I will use S3CMD tool and an exec resource to get this done.

Ryuzaki gravatar imageRyuzaki ( 2015-09-25 06:17:10 -0600 )edit
0

answered 2015-09-29 22:53:39 -0600

DarylW gravatar image

Hey Ryuzaki!

I have a little bit of additional information that I can add to this discussion: I have been playing with keeping files on amazon s3 with puppet, and ended up writing my own modules.

Here are a few things I've discovered along the way...

  • The AWS module does not have any built in type to assist with reading/writing files to s3
  • Make sure you are using per instance EC2 IAM roles instead of manually managing keys
  • S3 Objects have ETag metadata available.
    • For small files, the md5 is the same as the etag (and there is a clever 'unless' example in branan/puppet-module-s3file line 38, but using curl only works publicly available s3 buckets. You can do something similar with the 'aws s3api head-object' call )
    • For large files, the ETag is the md5sum of (the md5sum of each part of the multipart upload) followed by -N, where N is the number of multi-parts. For files under 5g, you could 'edit' the metadata in place and trick s3 into recalculating the etag into an md5. You also could store the ETag value locally, and use that in some unless/onlyif logic to keep you from redownloading the file if it is the same.
    • If you are using s3cmd, it adds some custom metadata which includes the MD5. you may be able to extract that and use it with the built in md5sum abilities of puppet.

I have been playing around with a fork of malnick/s3 where I'm trying to cache the ETag to not have to download the file to a temp file every time I want to check if it exists. If there is a better way to create a custom resource that uses the ETag or mtime/ctime instead of md5sum to version the resource, I'd love to hear it so we can add functionality that the community can use!

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2015-09-24 04:15:14 -0600

Seen: 724 times

Last updated: Sep 29 '15