Ooohh umm...no transaction?
WhenEVER I make an insert, update, or delete -- I always use a transaction.
That's what we should always do, right?
Right?
WhenEVER I make an insert, update, or delete -- I always use a transaction.
That's what we should always do, right?
Right?
> For our setup/task (just update few fields on a table) the process of probing different batch sizes & Sidekiq thread numbers with couple of thousands/millions records took about 5 hours [...]
You want to commit a set of statements that is a business transaction.
---
In this case, it's okay for individual records to fail. They can either be manually processed, re-run, or root caused for failure and handled individually.
A lot of the work I do involves analyzing and improving Ruby applications' performance. Often, simply wrapping transactions around the database operations will help - without making other (often much needed) structural changes.
Typically, these get run during off-peak hours. This running in 30 minutes vs 12 hours simply isn't important.