I needed a mechanism to assure that the task will automatically reconnect to the database if and when that connection was broken

2012-06-03

One of my projects has a long-running task that constantly needs information from the database. I needed a mechanism to assure that the task will automatically reconnect to the database if and when that connection was broken.

I came up with this scheme using a trick with rescue blocks (code abbreviated for clarity) in this gist.

def my_task

    while(true) do
      begin
        database_access_here
      rescue Exception => ex
        begin
          ActiveRecord::Base.connection.reconnect!
        rescue
          sleep 10
          retry # will retry the reconnect
        else
          retry # will retry the database_access_here call
        end
      end
    end
  end

Here's a line-by-line explanation:

Line 4: This is where your application's database access logic would be.

Line 5: Catch a database access exception here

Here is where it gets interesting:

Line 7: Open a new block and retry the connection.

Line 10: This retry will retry the reconnect method and will loop as long as the database connection is still down.

Line 11: The else clause will execute if _no_ exception happened in line 10, and will retry the original database call in line 4.

In my case and example, I am not counting retries because I don't care that I've failed - I must continue to retry. You may want to use "retry if retries < 3" as a break mechanism.

I have also removed some mailer code that notifies me when the reconnect fails so I can (manually) see what happened to the connection. The moment the connection is re-established, life goes on as normal within the infinite while loop.

Filed under

ActiveRecord
Ruby

Other Tags

API GW
AWS
ActiveRecord
Agile
Alexa
Analysis
Ansible
BDD
BLE
C
CAB
CloudFormation
CloudFront
CloudWatch
Cross-compile
Cucumber
DevOps
Devops
DotNet
Embedded
Fitbit
GNU
GitHub Actions
Governance
How-to
Inception
IoT
Javascript
Jest
Lambda
Mac OS X
MacRuby
Metrics
MySQL
NetBeans
Objective-C
PMO
Product Management
Programme management
Project Management
Quality Assurance
Rails
Raspberry Pi
Remote compilation
Remote debugging
Remote execution
Risk Assessment
Route 53
Ruby
S3
SPA
Self Organising Teams
SpecFlow
TDD
Unit testing
VSM
Value
arm
contract testing
inception
nrf51
pact
planning
rSpec
ruby
ssh