Robert Casey
2016-08-02 18:44:33
Hi MUSTANG Community-
IRIS Data Services will be attempting to upgrade MUSTANG's database from Postgres 9.1 to 9.5. This should bring some notable performance benefits to the database as well as bring it up to date for support by our CentOS 7 platform. We underwent a mock upgrade on a separate server in April and the process went well. We also know that the time to complete the process takes approximately 15 hours for a data cluster of this size.
What we will attempt to do is transfer user connections to a backup database during this upgrade time period, in hopes that user disruption will be only a few minutes at most. We will halt metrics processing for about 48 hours in order to complete snapshot backups and carry out the full upgrade process. Once completed with the upgrade, we will transfer services to the new database and resume metrics processing shortly thereafter.
The timing will look like the following, subject to change based on circumstances:
August 16: Shut down metrics processing at 3 PM PDT. User access to MUSTANG will be unaffected.
August 17: Transfer MUSTANG services to backup database at 2PM PDT. Brief downtime during transfer. Begin upgrade process in the background on primary DBMS.
August 18: Monitor for completion of upgrade and re-indexing process. Resume user access to upgraded database, then resume metrics processing.
Please feel free to contact me with questions or concerns. Our intent is to have a minimum of impact on user activity, even as this is a major upgrade process on our server.
-Rob
-
Robert Casey
2016-08-17 17:04:53
A reminder message that we are in the process of upgrading the MUSTANG database. Yesterday, we halted metrics processing to get final, stable backup snapshots. Today, we will be bringing up a database instance at our alternate data center to continue to serve users while we upgrade the Seattle primary database. There may be a few minutes of disruption of service between 2PM and 3PM PDT as we redirect network traffic to that backup db. We will make every effort to minimize downtime. The main effect users will notice is that recently arriving data will not yet have metrics available. We will hopefully see this processing resume tomorrow on our new db platform.
Thank you for your patience as we complete this important update.
-Rob
On Aug 2, 2016, at 11:45 AM, Rob Casey <rob<at>iris.washington.edu> wrote:
Hi MUSTANG Community-
IRIS Data Services will be attempting to upgrade MUSTANG's database from Postgres 9.1 to 9.5. This should bring some notable performance benefits to the database as well as bring it up to date for support by our CentOS 7 platform. We underwent a mock upgrade on a separate server in April and the process went well. We also know that the time to complete the process takes approximately 15 hours for a data cluster of this size.
What we will attempt to do is transfer user connections to a backup database during this upgrade time period, in hopes that user disruption will be only a few minutes at most. We will halt metrics processing for about 48 hours in order to complete snapshot backups and carry out the full upgrade process. Once completed with the upgrade, we will transfer services to the new database and resume metrics processing shortly thereafter.
The timing will look like the following, subject to change based on circumstances:
August 16: Shut down metrics processing at 3 PM PDT. User access to MUSTANG will be unaffected.
August 17: Transfer MUSTANG services to backup database at 2PM PDT. Brief downtime during transfer. Begin upgrade process in the background on primary DBMS.
August 18: Monitor for completion of upgrade and re-indexing process. Resume user access to upgraded database, then resume metrics processing.
Please feel free to contact me with questions or concerns. Our intent is to have a minimum of impact on user activity, even as this is a major upgrade process on our server.
-Rob
----------------------
MUSTANG QA (http://ds.iris.edu/message-center/topic/mustang-qa/)
Sent via IRIS Message Center (http://ds.iris.edu/message-center/)
Update subscription preferences at http://ds.iris.edu/account/profile/
-
It took an extra day of cleanup operations, but it looks like we are done with the DB upgrade and we will shortly switch MUSTANG web services back to the primary database here in Seattle. There may be momentary blips in service, but nothing too disruptive is expected.
Thank you for your patience as we resume normal operations before we get into the weekend.
-Rob
On Aug 17, 2016, at 10:04 AM, Robert Casey <rob<at>iris.washington.edu> wrote:
A reminder message that we are in the process of upgrading the MUSTANG database. Yesterday, we halted metrics processing to get final, stable backup snapshots. Today, we will be bringing up a database instance at our alternate data center to continue to serve users while we upgrade the Seattle primary database. There may be a few minutes of disruption of service between 2PM and 3PM PDT as we redirect network traffic to that backup db. We will make every effort to minimize downtime. The main effect users will notice is that recently arriving data will not yet have metrics available. We will hopefully see this processing resume tomorrow on our new db platform.
Thank you for your patience as we complete this important update.
-Rob
On Aug 2, 2016, at 11:45 AM, Rob Casey <rob<at>iris.washington.edu> wrote:
Hi MUSTANG Community-
IRIS Data Services will be attempting to upgrade MUSTANG's database from Postgres 9.1 to 9.5. This should bring some notable performance benefits to the database as well as bring it up to date for support by our CentOS 7 platform. We underwent a mock upgrade on a separate server in April and the process went well. We also know that the time to complete the process takes approximately 15 hours for a data cluster of this size.
What we will attempt to do is transfer user connections to a backup database during this upgrade time period, in hopes that user disruption will be only a few minutes at most. We will halt metrics processing for about 48 hours in order to complete snapshot backups and carry out the full upgrade process. Once completed with the upgrade, we will transfer services to the new database and resume metrics processing shortly thereafter.
The timing will look like the following, subject to change based on circumstances:
August 16: Shut down metrics processing at 3 PM PDT. User access to MUSTANG will be unaffected.
August 17: Transfer MUSTANG services to backup database at 2PM PDT. Brief downtime during transfer. Begin upgrade process in the background on primary DBMS.
August 18: Monitor for completion of upgrade and re-indexing process. Resume user access to upgraded database, then resume metrics processing.
Please feel free to contact me with questions or concerns. Our intent is to have a minimum of impact on user activity, even as this is a major upgrade process on our server.
-Rob
----------------------
MUSTANG QA (http://ds.iris.edu/message-center/topic/mustang-qa/)
Sent via IRIS Message Center (http://ds.iris.edu/message-center/)
Update subscription preferences at http://ds.iris.edu/account/profile/
-
