Site Migration - Complete
Submitted: Monday, Jun 30, 2014 at 11:06
ThreadID:
108551
Views:
3725
Replies:
7
FollowUps:
2
This Thread has been Archived
ExplorOz - David & Michelle
This is the week it all has to happen as our current servers will be shutdown on Saturday. As some of you know a lot of the operation has been moved - image server, mobile site server and part of the mapping server. So the rest goes this week.
Here is a rough schedule:
Monday (Business Hours) - Move smaller databases - should have no effect on ExplorOz
Monday (Evening) - Migrate
Forum,
Blogs and Events database. Will cause shutdown of these systems. Expect to start 8PM WST (10PM EST).
Tuesday (Business Hours) - Move all connections to the new web server so all connections to ExplorOz will then be via AWS.
Tuesday (Evening) - Migrate
Places database. Will cause shutdowns to
Places,
Treks and most mapping type functions. Start 8PM WST (10PM EST).
Wednesday (Business Hours) - Move other sites to AWS (itbeyond and other client sites).
Wednesday (Evening) - Migrate final ExplorOz database. This will cause site shutdown for several hours start 8PM WST (10PM EST).
By now I will be tired and hoping it is all working ok. If any of these planned operations fail it will put me back to the next day so the schedule will slide towards Friday. I hope this schedule is workable and that my 6 weeks of planning gets us through. If I am really stuck Thursday Evening and Friday would be the targeted downtime.
I will update this post as we go. I am working on the systems all the time so there could be some minor hiccups. Anything major please email me direct david at exploroz.com but I will most likely know about it anyway.
David
Reply By: ExplorOz - David & Michelle - Tuesday, Jul 01, 2014 at 01:20
Tuesday, Jul 01, 2014 at 01:20
OK Monday progress is not too bad. We now have the
Forum,
Blogs, Events,
Wildflowers and
Recipes database now running on the new servers. As this new machine is not on the same local network as the web server there may be very slight extra delays in doing stuff but really you should not see these.
For the techos types: the transfer of 2.2GB took 2:20 and there was some issues with users, security and enabling of clr on the SQL box. But it is now running and I am happy that it is not yet midnight here. It will be before I am finished testing but a good start.
The problem I have is the transfer of the main database which is 15GB will take too long to do in one hit so tomorrows tasks apart from switching the web site over will be arranging to break up and transfer the production database.
Time for more testing, then a sleep, then a bike ride in the morning if I can get out of bed.
David
AnswerID:
535282
Reply By: ExplorOz - David & Michelle - Tuesday, Jul 01, 2014 at 21:45
Tuesday, Jul 01, 2014 at 21:45
Tuesday Evening Update:
Tonight I will not need to bring the site down - YEAH. So you can all keep on enjoying yourselves. There is a very large database copy running at the same time and this may slow the system down a little but the bulk of the ExplorOz data is being copied to the new database server now and will be for the next 8 hours or so. Once this is copied I will transfer the really dynamic stuff and the updates to the data I am currently moving and we should be ready to turn the new database on tomorrow night.
There have been a number of hiccups today and if these effected you I am really sorry. Some of the periodic jobs ran several time and sent many people multiple emails. I believe most of these are now behind us. Apart from that not too bad at this point considering what is being done.
David
AnswerID:
535313
Reply By: ExplorOz - David & Michelle - Wednesday, Jul 02, 2014 at 17:26
Wednesday, Jul 02, 2014 at 17:26
Wednesday Evening will be a total shutdown night starting at 8PM WST or 10PM EST. I must bring the system down and make the final database migrations for all the ExplorOz sites.
I have no idea how long it will take but lets start by saying 1AM WST or 3AM EST as an estimate. I will update the message on the system as I move along.
David
AnswerID:
535344
Follow Up By: ExplorOz - David & Michelle - Thursday, Jul 03, 2014 at 02:28
Thursday, Jul 03, 2014 at 02:28
Gee that was not too bad - 12:20 and the system is up and running. I am happy that most of it works however the GMT date thing is still concerning me a little.
However we are now almost fully transferred - I have the mapping and tracking servers still to move to make EO complete, then the itbeyond site a couple of customer sites and database and then job done (hopefully).
So things are mostly on track. Thanks for your patience at this time we are almost there.
David
FollowupID:
819230
Reply By: ExplorOz - David & Michelle - Thursday, Jul 03, 2014 at 18:54
Thursday, Jul 03, 2014 at 18:54
OMG, "David it is newsletter day are you not ready to send this using the new system yet" was the question this morning. "
Well maybe" was the answer. In the middle of the hundreds of tasks I am on this one presented itself and
well it is sending.
I was hoping it would be so much faster to run the job but sending 55,000 customised & personalised emails does take a wee bit of time. The migration is really going
well actually a few hiccups (like losing some tracking data today) but all in all very happy with the progress.
I have only the map and tracking systems to migrate for EO still as there is something stopping the programs from running on the new system. I have been working on the other customer sites and database and I am almost finished these so will be able to dedicate more resources to the map server once these are finished.
I do not expect to have to stop the ExplorOz web and mobile servers again so if all goes to plan there will be no more scheduled outages and hopefully no unscheduled outages.
David
AnswerID:
535395
Reply By: ExplorOz - David & Michelle - Friday, Jul 04, 2014 at 20:38
Friday, Jul 04, 2014 at 20:38
Ok it had to happen - one, ok ok two errors today. They are recoverable but bugga bugga pooh bottom. I hate that.
1. So what was effected - darn mapping & tracking server.
So what happened,
well on Wednesday night when I was moving the ExplorOz database I made a small change to one of the largest table in the system the Tracking points table. I had three columns in there that I no longer needed and these could only be removed when the system was down. So I removed them and understanding this would break stuff edited the procedures that update this table. So all seemed good - UNTIL I found out it was not recording the data correctly today. So the ah ha moment which happened about 1 hour ago was that I had about 5 weeks ago adjusted the way the data was written to this table from the tracking server due to performance issue and this was not right and hence was not working BUT to make it just that bit worse my usual line that says email me if there is a failure was commented out so I was not alerted.
What does all this really mean,
well easy if you used tracking since Wednesday night the data is not currently showing. Thus it appears you have not moved. The program is now fixed so new updates will be fine.
What about our missing data,
well that is recoverable as I log everything received twice just for this problem, but as it has never happened before I do not have a process/program on hand to run just yet. So either over the weekend or early next week I will write a program to recover the points from the log.
2. When dealing with 20+ domain names during this transfer I forgot to work on two being eotrackme.com and eotrackme.com.au. So these are still with the old manager for the moment.
So what,
well I cannot point the tracking clients at the new server myself and have to wait for the current/old ISP to change the data for me. Hopefully this will be done tonight so the GMT date thing will go away (hopefully). I will transfer the domain controls also but this can take up to 1 week to complete.
Enough of the negatives - the BIG positive is that apart from a small amount of tracking traffic the whole job is done and dusted. There are plenty of problems for me to fix next week but most of you will never see or know about these.
I am really very happy with the migration and the process apart from the tracking stuff ups was almost perfect. Now to let the system settle down so I can start the job of tweaking the setup to minimise costs and improve performance.
David
AnswerID:
535464
Follow Up By: Allan B (Member, SunCoast) - Friday, Jul 04, 2014 at 23:23
Friday, Jul 04, 2014 at 23:23
Mighty job David.
I'm sure that the majority of EO site users have little idea of the scale of complexities behind the screen!
FollowupID:
819338
Reply By: Member - Stuart P (WA) - Friday, Jul 04, 2014 at 23:30
Friday, Jul 04, 2014 at 23:30
thank you both
AnswerID:
535475
Reply By: Member - Frank P (NSW) - Saturday, Jul 05, 2014 at 18:03
Saturday, Jul 05, 2014 at 18:03
Fabulous job, David and Michelle. Thank you.
One thing though, and I guess you're still tweaking but I'll mention it anyway ...
When I click Back to Thread or Back to
Forum, it is slower than it used to be. In fact on my machine it is all a bit slower.
Win 7
Home Premium fully updated. Slimbrowser fully updated (a front end for IE), IE 11 fully updated. It is marginally faster in Chrome and about the same as Slimbrowser/IE in Firefox.
Cheers
Frank
AnswerID:
535492