a week before raya i have lots of things need to be done.Apply patches for all development servers, housekeeping, clear out logs,make sure all the mount point in production server is not 100% utilised and so many many things before having a long holiday.
My concerned is of course Production Server.Im so worried cz Im still new in RAC(Oracle Cluster) environment.For production database...i have to make sure all the backup is working fine including net backup,file system backup, export and RMAN.
I already know that PNB had requested a new environment for Income Distribution.I tot it was done by his new DBA.The manager said he need this environment before hari raya.Everyday keep on asking me whether is completed or not.That's mean it has to be done by ME and I have to complete it before raya.
Huh....so stress...org lain dah bersukaria nak bercuti....im still struggling completing my task.
Actually we had some problem on front end setup for the new environment.That was the last day and im so worried.Alhamdulillah...Shan from iflex helps me alot.We found out a really minor silly mistakes and on 12.00pm...the new environment is ready.
Wah..so so happy cz that day our PM gave us half day.yeay....
and...
While packing to go back home, I ask Mat, our DCO Head...what else to check..I had done on my database site. Then mat asked me to check the replication status. Then i checked the status with confidence.From the result its shown that outstanding writing and the number is keep on increasing..not even one writes to DR site.Huh.....sounds like we are having a disaster.rasa nak menangis je takut tak dpt nak beraya....
if database crashed during hari raya...we have no backups on DR site.huh...so scary....and for sure our boss will kill us.
we still monitor the performance about 1 hour..checking network,database,connection and so on...still increasing...OMG..what should I do....Im not expert in Veritas Replication...
Then we decided to call our vendor,Koon Hoong from GCIS.Thank God,he's available.He asked me to run some command to know the status and details of the server.I have sent the output to him and still no solution.I request him to remote my notebook but he dont have any software to remote.Normally we used Webex.
Hmm...end up..he guided me thru Gtalk.Huhuhu...so scared cz Im still new in VVR.I only know the basic commands.But..i have to...if not....tak boleh raya...
after 4 hours troubleshooting...we found out that the connection is not establish.But the network is OK.Maybe lost of connection in DR server.So we decided to reboot the DR server remotely.Its too risky.If Having problem in DR..we have to go to DR site in Subang area...huh...too far from here..means confirm mmg tak boleh raya la...
Alhamdulillah...God Bless us...the server successfully rebooted and we just resume the replication and all things works as usual.That time is 6.45pm.huhuhuhu...Mat have to go back hometown in kedah and I have to take care my both kids since my hubby also got fibre breakdown in Melaka...huhuhu....That is we have to pay when involving in operation team.My hubby works in operation team too.Both are dealing with critical system and should be NO DOWNTIME by hook or by crook.Pity my sons.
...many2 thanks to Koon Hoong for his kindness.he's a cool and nice person.Siap buat lawak lg masa critical tu..Huhuhu....reduced my stress.really impressed the GCIS team while doing their work.Always smiles and cool even in stress and critical period.I still remember when we are doing downtime when the system crashed. Basu from Iflex as DBA, Kiat Fei and Hoong from Veritas.And that is the precious time cz we learn how to tackle the problem.And since all the experts are here...I asked so many2 things even silly question to them...huhuhu...nasib la....on job training.
anyway...thats all before raya incident.......as per Koon Hoong said..now i can peacefully having my hari raya,InsyaAllah.Thank you very very much ..u really saved my day.
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment