S3-Stat Conflict with Drupal S3 Downloads Module
Hey guys, I was absolutely loving this service until I was no longer to access the edit screen for my drupal users. Pinged my dev guy about the problem and this is what he said:----
These are huge amounts of files..What Drupal S3 Downloads module does.. it retrieves the S3 buckets, as well as all the files in them to store the filenames in Drupals DB. So everytime it rechecks, it has to requery S3 and ends up being sent back this HUGE array of filenames..In PHP, each script has a max execution time, generally it's 30 seconds (set by your service provider). I set my local copy to 300 seconds, and it's still timing out.So given that info, the user edit page shows a list of files that can be added to the user, but it first retrieves a list by first updating the list of files (hence querying S3 everytime). This is why it's timing on specifically on this page.It may have been okay at first because the amount of log files were small, but as they grew over time.. the list of files became larger and larger to a point to where it's to large to send past 30 seconds.So basically.. you're going to have to figure out another solution to store all those log files.. because right now it's not compatible with the Drupal S3 Download Module.----Do you know of any workarounds to this?Thanks...
Clint
coreintegral
Sunday, March 27, 2011
Your dev guy is right. If you have a naive S3 client that attempts to list all files in a bucket without filtering, it will certainly choke on the sheer volume of logfiles that Amazon delivers.Two options spring to mind:- you could run a script to delete your old logfiles each night, thus keeping only a single day's files around for us to process and avoiding the "thousands of files" issue.- you could register a separate bucket that you use just for logging, and have your logfiles sent there instead.The second option seems like the easiest to set up, and has the least number of things that can go wrong. Here's a quick step-by-step rundown of how to do it:- create a new bucket (using Amazon's tools), called "mysitelogs"- In S3stat, "add" your original bucket/distribution again, expanding the "+ more" section and changing the "Log Bucket" options to point at that new bucket (and probably changing the delivery folder too). Keep the Report Bucket the same.Hopefully that will get you there. If anything comes off the rails during the process, send me an email and I'll help you through it.
Jason Kester
Monday, March 28, 2011
[ reply to this topic ]
[ return to topic list ]
|