Distributed system recomendations...
Hi, we are currently looking at developing a very large web application. We have to be able to store a vast number of photos and serve them quickly. We need the solution to be high availability, fault tolerant, and easily scalable.
Our current thoughts are to have Networked Attached Storage sitting behind some database servers that in turn sit behind the webservers.
Do any of you guys have any thought or suggestions?
Ta muchly,
Ben.
mairving posted this at 17:55 — 19th April 2006.
They have: 2,256 posts
Joined: Feb 2001
Ben, welcome to TWF.
Is this going to be running on 'nix/Apache/MySQL or Windows/SQL Server?
benbroad posted this at 20:11 — 19th April 2006.
They have: 2 posts
Joined: Apr 2006
Yeah, sorry that info would be handy... Our current servers use Fedora Core 3 / MySQL 5 / PHP 5 - standard LAMP setup. I dont mind changing the Linux distro and the database system but definately continuing with the LAMP style setup.
We did some time tests to see how long it takes to serve a page that has 500 thumbnail images on and found that there is very little difference between storing the data in the database and storing it as files - even taking caching into account.
We are looking into using a distributed database rather than a distributed file system as it seems the easier way to go - possibly MySQL cluster or PostGRE cluster.
To give you an idea of scale, we are looking at about 30GB of storage for the photos when we get to beta stage, expanging to about 3TB over 24 months. We are currently trying to figure out the processing requirements for serving this data, but we are working with a lot of "best guess" estimates for a system we havn't strated writing yet. Frustrating.
Want to join the discussion? Create an account or log in if you already have one. Joining is fast, free and painless! We’ll even whisk you back here when you’ve finished.