I have not had a chance to play with G3, so I don’t know how well this method would work with it. Of course it would have to be modified, but I don’t know in what way. It is also possible that G3 changed things in such a way that a S3 plugin would be easier to write.

I’m sorry I can’t be more helpful!

I was wondering if you’ve had a chance to try this solution with the Gallery3 beta.

I’ve been looking for a way to host my gallery2 files on S3 for a long time but I am also keen to up upgrade to Gallery3 especially as it appears to be reaching a Beta4 or RC1 stage.

Is your S3 solution compatible with Gallery3?

Thanks.

it seems to me that your solution works only when the original webserver has enough disk space for all your media files. When that this the case, and one is willing to keep the S3/Cloudfront copies up to date manually (or perhaps with a rsync script), your method will work.

However, if the situation is like mine, where the webserver doesn’t have enough disk space for all your files, I don’t think it will work.

That being said, I very much appreciate the link and the comment. Thanks!

The modifications to make on the web server are minimal:
- only modify 2 lines in Gallery2’s .htaccess

But there are some drawbacks and it’s only good for gallery which are not updated by several people.

Have a look here:

http://www.flughafen-kurier.ch/blog/2009/07/24/how-to-use-s3-and-cloudfront-for-gallery2/

Comments and improvements are welcome.

In Gallery2, the thumbnails are functionally treated the same as all the other re-sized copies of the image. So yes, they are on S3.

If you’re moving a whole directory to S3, and the directory structure is more straightforward than the image referencing scheme for Gallery2, you may not even need a cgi script to redirect requests. You may be able to use only .htaccess URL rewrites.

I am not familiar with MediaWIki, so I don’t know how much guidance I can give there. I wish you luck!

I think your work is great. We do use access control for images; this is the right way for us to use S3.

I’m rewriting your solution it to work with MediaWiki (MW). I’m also looking to see what other people already may have done. Your code and instructions are great.

Like Gallery2, MW creates thumbnails for every image. We have this configured to be done when the image is originally stored. People can specify the size of the thumbnails in their image tags. A thumbnail in that sense is just a page-specific scaling of the image and can get pretty large. Thumbnails are rebuilt if none is available for a page when it’s rendered.

By using your model, are Gallery’s thumbnails also being stored on s3? If not, did you mess around with them to see if it made a difference?

I’m not so cheap that I’m trying to move the diskspace for the thumbnails to s3 because of cost. I’m more interested in getting as much storage out of the local box so that I can also migrate my php apps to ec2.

I’m experimenting with this on a local set of boxes today to see how it works. In MW, the storage of thumbnails is done in the same base directory as the fat image files. I want to see if I can just remote the whole directory. We have a few dozen separate MW instances to relocate. It will be a lot cleaner to have the same model with s3 in the mix.

Thanks again for your work and thanks, in advance, for any comments.

The fact that images are sent through amazon’s server (since your server just redirects the user to the temporary URL) makes things much faster. If your server is getting hammered, you won’t be serving lengthy requests (images might be big, after all).

Currently my implementation is very hacking (ashamed to show it, strange variables everywhere and strange files lying around), but the concept seems to work really well.

I really wish I could make this into a gallery2 module, but it seems that one of the functions I’d need to modify is not modifiable through the module system, which is a problem. (Doesn’t seem Gallery.class can be changed in custom modules).

If anyone has ideas, please tell me.

Mike, if you are interested in what I’m doing, feel free to contact me through email (genia4@gmail.com). I’m also available ton google talk (same address). I can explain in more detail what I did (and provide code snippets).

That sounds really interesting. Right now, I’m using s3, but mounted via S3FS (my derivatives are local, but the full sizes are on s3), since I want the image firewall. If there was some way to access amazon directly while preserving some security, that would be great (my security needs are more along the lines of not leaking URLs through referer headers and less for actual “security”, so a temporary URL that expired after a few minutes would be plenty).

Looking forward…

As soon as I have something completely working, I’ll publish the code and give a URL to a demo site.

This method should lighten the load on the server considerably, since the rewrite.py file and all the associated hackery needs not be called anymore – requests go straight to amazon’s servers.

If anyone else is interested in this solution, please tell me

I’m open to any ideas to make this thing better, of course. Good luck!

Thanks for the great post (and the nicely drawn diagram on the other detailed page)!

I could put a copy of the script on my website, but I was trying to keep credit where it’s due.

I think this link is to the same script, at a different location.

http://pypi.python.org/pypi/path.py/

If I’m wrong, of course, please delete this post.

Cool! I’m happy that it’s working for you. If you have any suggestions on how I can make any of the steps clearer, let me know.

I think, however, that I’m going to stick with the mounted FS idea. Even though the permissions shouldn’t offer a core.DownloadItem to an unauthorized user, anyone who has one will be able to download it or email it to someone without authorization (in my context, I’m more worried about an accidental release of a private image, and less of a deliberate attempt by a user to try and view images when not logged in). Perhaps running a minimal subset of gallery on Amazon’s EC2 can help that…

In any event, nice!

Nice work!

The idea is that Amazon S3 has a massive ‘cloud’ setup with huge bandwidth, greater than any single server could ever have. As much of the data intensive parts of a page should come directly from S3, and bypass the bandwidth limited server.

Am I missing something obvious here? Aside from the double bandwidth hit, why not just mount your g2data/albums on s3 (using the S3FS linked above), and be done with it in one step? What’s the problem with that method?