There are 4 separate tasks that need to be accomplished:
1) Web App Script (using Django/boto or similar) - The script needs to be able to execute the following tasks:
- When file is loaded from website, automatically run a pre-processing script (this would require 1 small-instance)
- Launch multiple spot cluster compute instances automatically if pre-processing script exits cleanly, and then run our HPC application. We are planning on using Rocks ([url removed, login to view]) or Starcluster ([url removed, login to view]) for resource management (I have Starcluster configured as of now)
- Terminate all instances if there are no more simulations to run
I was recommended by Amazon to set up a web tier of instances on EC2 to handle the web requests, a processing tier for the initial pre-processing and orchestration, and SQS for the message queues. Amazon can scale out our tiers depending on usage, either by the number of web requests, or the size of the message queues using the CloudWatch metrics service.
2) User Registration and Authentication - Django or similar
3) Large file upload to servers using jQuery and simple web page design for landing page and file upload page (Similar to [url removed, login to view] . Once the user has logged into our website we would like to allow them to upload multiple [url removed, login to view] files.)
4) Database setup for storage of user data on Amazon persistent storage.
Let me know if you have any questions.