SuperFord.org Forums
| SFO Forums > Site-Related Forums > SFO News and Feedback |
| Corrupt Database table restored from 9/1/03 backup - user photos/videos/audio files affected |
| Thread Statistics: Users to Post: 4 | Total Posts: 9 | Total Views: 1841 |
| You must be logged in to post in or subscribe to this thread. | Pages: 1 |
|
E. Long
 
 |
• Subscriber since 1/1/2001 |
|
• Atlanta, GA, USA |
|
• Registered on 1/23/2001 |
|
• 2,229 posts |
• 10 Vehicles
|
|
 Posted:9/19/2003 16:46 |
|
|
Sometime this morning before 9AM CST the database on the server started having issues with the table that houses all member photos, videos, and sounds. After several hours of attempting to rebuild the table to no avail, it was determined that we would have to go back to a 2 week old backup.
Everything is up and running again. Folks who posted photos, audio files, or video files after 9/1/03 will need to re-post them. All other data was unaffected.
We will be running daily preventative maintenance checks on the database for the next several weeks to ensure this doesn't happen again.
If you have any questions regarding this incident, please contact us via the support page.
-Eric
'67 Galaxie 500 - 390 FE, .030" over, FE to AOD adapter, disc brake conversion. The Daily Driver.
'00 Excursion - 7.3L PSD, LANDYOT Gen-II Radius Rods, Factory Tech Valve Body, 200K+ miles and going |
|
Bob Gervais
 |
|
• North Providence, RI, USA |
|
• Registered on 7/23/2002 |
|
• 4 posts |
|
|
|
|
Posted:9/19/2003 23:40 |
|
|
|
Isn't two weeks a long time between backups? Or is this a normal amount of time? |
|
E. Long
|
• Subscriber since 1/1/2001 |
|
• Atlanta, GA, USA |
|
• Registered on 1/23/2001 |
|
• 2,229 posts |
• 10 Vehicles
|
|
|
Posted:9/20/2003 10:20 |
|
|
| Quote: | | Isn't two weeks a long time between backups? Or is this a normal amount of time? |
Hi Bob,
Here's a little background to the framework of the site:
SuperFord.org consists of nearly two dozen database tables that are used to keep track of all of the data for the site (user accounts, photo albums, files posted by members, forum posts, club registries, traffic monitoring, album sections, etc). In most cases, each of these individual tables are very small and are backed up every other day as well as once per month.
The strategy with a backup system is to also keep a history of backups. For example, you don't want to just keep backing up a table each day and writing over previous backups. The reason for keeping a history of backups is in case something happens on Thursday during the week that causes problems on Friday. Going back to the backup from the night before wouldn't be a good solution because the data was already compromised then. For these situations, you would want to go back to a backup earlier in the week.
Since a site like this is visited by 10's of thousands of people per month, we also keep a backup for the entire month. This covers all of our bases in the event that something creeps up that was caused much earlier in the week or earlier in the month.
Now, with regards to the 2-week old backup of the media files. This is a special case where it is difficult to make daily/frequent backups. When database tables are backed up, the safest way to do this is locking out the table which means nobody can access the data. The media table is currently about 12 Gigabytes in size and grows at an exponential rate. To keep daily backups of this would require a lot more hard drive space than is currently available on the server. It would also require locking members out of viewing/posting media files each day for at least 1 hour while the backup runs.
So, the reason why we don't keep daily backups of the table that houses all posted media from members is because it is currently impossible with this server. In the nearly 2 years that we've been running this database structure, this is the first corruption problem we've had which would suggest that situations like these are highly unlikely to occur. With the new database maintenance procedures we've implemented as of yesterday, we also feel that something like this will never happen again unless a hardware error creeps up that is impossible to avoid (i.e. crashed hard drive, etc). The SuperFord.org site runs on a RAID hard drive setup which means in the event that a hard drive does crash, it can rebuild itself automatically w/o risk of data loss when a new drive is installed.
In situations like this, preventative maintenance is what keeps the site alive and running w/o downtime.
I hope this helps explain the procedure.
-Eric
'67 Galaxie 500 - 390 FE, .030" over, FE to AOD adapter, disc brake conversion. The Daily Driver.
'00 Excursion - 7.3L PSD, LANDYOT Gen-II Radius Rods, Factory Tech Valve Body, 200K+ miles and going |
|
Bob Gervais
|
|
• North Providence, RI, USA |
|
• Registered on 7/23/2002 |
|
• 4 posts |
|
|
|
|
Posted:9/20/2003 12:58 |
|
|
|
Thanks for explaining it Eric. Was there another problem? I uploaded images this morning, and they are gone now. Did you need to go back to a different backup again? |
|
E. Long
|
• Subscriber since 1/1/2001 |
|
• Atlanta, GA, USA |
|
• Registered on 1/23/2001 |
|
• 2,229 posts |
• 10 Vehicles
|
|
|
Posted:9/20/2003 18:45 |
|
|
| Quote: | | Thanks for explaining it Eric. Was there another problem? I uploaded images this morning, and they are gone now. Did you need to go back to a different backup again? |
You're quick -- we reverted back to last night's backup this morning due to an error we found while optimizing the database. It was one of those instances where we should have complied with the "if it ain't broke, don't fix it" motto. We will save further database optimization procedures until the server is upgraded when extended downtime can be planned.
-Eric
'67 Galaxie 500 - 390 FE, .030" over, FE to AOD adapter, disc brake conversion. The Daily Driver.
'00 Excursion - 7.3L PSD, LANDYOT Gen-II Radius Rods, Factory Tech Valve Body, 200K+ miles and going |
|
Shawn Clemons
 |
|
• Raleigh, NC, USA |
|
• Registered on 9/24/2001 |
|
• 14 posts |
• 1 Vehicle
|
|
|
Posted:9/24/2003 11:11 |
|
|
|
The image counts for affected folders are still incorrect. Any way to fix this? |
|
E. Long
|
• Subscriber since 1/1/2001 |
|
• Atlanta, GA, USA |
|
• Registered on 1/23/2001 |
|
• 2,229 posts |
• 10 Vehicles
|
|
|
99ExplorerXLT
 |
|
• Round Lake Beach, IL, USA |
|
• Registered on 2/21/2003 |
|
• 7 posts |
• 3 Vehicles
|
|
|
Posted:11/9/2003 18:31 |
|
|
Did this have adverse affects on the movies we have uploaded? For somereason I can't view any of mine anymore. They are there, but it always says the format is wrong now or something... 
'99 Explorer XLT - lifted 5" on 32s
'87 Bronco II
lifted/locked/welded/caged on 33s
'97 F250 HD - stock |
|
E. Long
|
• Subscriber since 1/1/2001 |
|
• Atlanta, GA, USA |
|
• Registered on 1/23/2001 |
|
• 2,229 posts |
• 10 Vehicles
|
|
|
| SFO Forums > Site-Related Forums > SFO News and Feedback |
| Corrupt Database table restored from 9/1/03 backup - user photos/videos/audio files affected |
| Thread Statistics: Users to Post: 4 | Total Posts: 9 | Total Views: 1841 |
| You must be logged in to post in or subscribe to this thread. | Pages: 1 |
|
|
|
