勉強会で大量件数をデータパッチするバッチを作った

[rvillage]

課題

addresses#faxには、ユーザが入力したFAX番号が登録されています。（ハイフンあり、半角数字）
バッチスクリプトで、addresses#compress_faxに圧縮番号（半角数値のみ）を登録してください。

mysql> desc addresses;
+--------------+-------------+------+-----+---------+----------------+
| Field        | Type        | Null | Key | Default | Extra          |
+--------------+-------------+------+-----+---------+----------------+
| id           | int(11)     | NO   | PRI | NULL    | auto_increment |
| fax          | varchar(50) | YES  |     | NULL    |                |
| compress_fax | varchar(20) | YES  |     | NULL    |                |
+--------------+-------------+------+-----+---------+----------------+
3 rows in set (0.00 sec)

注意

• なんらかの登録ができないレコードがあった場合、中断して失敗したレコードの内容と失敗理由を分かるようにしてください
• 中断したあとも途中から再スタートできるようにしてください
• マシンリソースは使いすぎず、かつできるだけ早く実行完了してください

データ

準備

# データベース作成
CREATE DATABASE mokumoku;
USE mokumoku;

# テーブル作成
CREATE TABLE addresses(id INT NOT NULL PRIMARY KEY AUTO_INCREMENT, fax VARCHAR(50), compress_fax VARCHAR(20));

HOST=localhost PORT=3306 DB=mokumoku USER=root PASS=__PASS__ bundle exec ruby ./lib/db_seeds.rb

リセット

# addresses#compress_faxをNULLで一括更新
UPDATE addresses SET compress_fax = NULL;

後始末

# データベース削除
DROP DATABASE mokumoku;

[rvillage]

1行ずつupdateするのは効率悪い

[rvillage]

DBデータの読み込み方

each

[1] pry(main)> Address.all.each {|e| puts e}
  Address Load (1.0ms)  SELECT `addresses`.* FROM `addresses`
#<Address:0x007f833fadbc80>
#<Address:0x007f833fadbb40>
...

find_each

[5] pry(main)> Address.find_each(batch_size: 700) {|e| puts e}
  Address Load (0.9ms)  SELECT  `addresses`.* FROM `addresses` ORDER BY `addresses`.`id` ASC LIMIT 700
#<Address:0x007f833fadbc80>
#<Address:0x007f833fadbb40>
...
  Address Load (0.5ms)  SELECT  `addresses`.* FROM `addresses` WHERE (`addresses`.`id` > 700) ORDER BY `addresses`.`id` ASC LIMIT 700

find_in_batches

[9] pry(main)> Address.find_in_batches(batch_size: 700) {|e| puts e.count}
  Address Load (0.9ms)  SELECT  `addresses`.* FROM `addresses` ORDER BY `addresses`.`id` ASC LIMIT 700
700
  Address Load (0.5ms)  SELECT  `addresses`.* FROM `addresses` WHERE (`addresses`.`id` > 700) ORDER BY `addresses`.`id` ASC LIMIT 700
300

limit

[10] pry(main)> Address.all.limit(10)
  Address Load (0.4ms)  SELECT  `addresses`.* FROM `addresses` LIMIT 10
=> [#<Address:0x007f833fc78b38 id: 1, fax: "0120-000-001", compress_fax: "0120000001">,
    #<Address:0x007f833fc78980 id: 2, fax: "0120-000-002", compress_fax: "0120000002">,
    #<Address:0x007f833fc78840 id: 3, fax: "0120-000-003", compress_fax: "0120000003">,
    #<Address:0x007f833fc78700 id: 4, fax: "0120-000-004", compress_fax: "0120000004">,
    #<Address:0x007f833fc785c0 id: 5, fax: "0120-000-005", compress_fax: "0120000005">,
    #<Address:0x007f833fc78480 id: 6, fax: "0120-000-006", compress_fax: "0120000006">,
    #<Address:0x007f833fc78340 id: 7, fax: "0120-000-007", compress_fax: "0120000007">,
    #<Address:0x007f833fc78200 id: 8, fax: "0120-000-008", compress_fax: "0120000008">,
    #<Address:0x007f833fc780c0 id: 9, fax: "0120-000-009", compress_fax: "0120000009">,
    #<Address:0x007f833fc73ef8 id: 10, fax: "0120-000-010", compress_fax: "0120000010">]

[rvillage]

updateのやり方

save!

[13] pry(main)> address = Address.first
  Address Load (0.4ms)  SELECT  `addresses`.* FROM `addresses` ORDER BY `addresses`.`id` ASC LIMIT 1
=> #<Address:0x007f8339bfb0f8 id: 1, fax: "0120-000-001", compress_fax: "0120000001">
[14] pry(main)> address.fax = '0120-111-111'
=> "0120-111-111"
[15] pry(main)> address.save!
  (0.2ms)  BEGIN
  SQL (0.4ms)  UPDATE `addresses` SET `fax` = '0120-111-111' WHERE `addresses`.`id` = 1
  (4.9ms)  COMMIT
=> true

update_attributes!

[16] pry(main)> address.update_attributes!(fax: '0120-222-222')
  (0.2ms)  BEGIN
  SQL (0.4ms)  UPDATE `addresses` SET `fax` = '0120-222-222' WHERE `addresses`.`id` = 1
  (2.6ms)  COMMIT
=> true

オブジェクト渡しでのbulk update

[17] pry(main)> address1 = Address.find(1)
  Address Load (0.4ms)  SELECT  `addresses`.* FROM `addresses` WHERE `addresses`.`id` = 1 LIMIT 1
=> #<Address:0x007f833fbb8d10 id: 1, fax: "0120-222-222", compress_fax: "0120000001">
[18] pry(main)> address2 = Address.find(2)
  Address Load (0.4ms)  SELECT  `addresses`.* FROM `addresses` WHERE `addresses`.`id` = 2 LIMIT 1
=> #<Address:0x007f833fb68928 id: 2, fax: "0120-000-002", compress_fax: "0120000002">
[19] pry(main)> address1.fax = '0120-111-111'
=> "0120-111-111"
[20] pry(main)> address2.fax = '0120-222-222'
=> "0120-222-222"
[22] pry(main)> Address.import([address1, address2], on_duplicate_key_update: %i(fax))
  Class Create Many Without Validations Or Callbacks (23.6ms)  INSERT INTO `addresses` (`id`,`fax`,`compress_fax`) VALUES (1,'0120-111-111','0120000001'),(2,'0120-222-222','0120000002') ON DUPLICATE KEY UPDATE `addresses`
=> #<struct ActiveRecord::Import::Result failed_instances=[], num_inserts=1, ids=[]>

配列渡しでのbulk update

[23] pry(main)> address1_value = [1, '0120-111-000']
=> [1, "0120-111-000"]
[24] pry(main)> address2_value = [2, '0120-222-000']
=> [2, "0120-222-000"]
[25] pry(main)> Address.import(%i(id fax), [address1_value, address2_value], on_duplicate_key_update: %i(fax))
  Class Create Many Without Validations Or Callbacks (16.9ms)  INSERT INTO `addresses` (`id`,`fax`) VALUES (1,'0120-111-000'),(2,'0120-222-000') ON DUPLICATE KEY UPDATE `addresses`.`fax`=VALUES(`fax`)
=> #<struct ActiveRecord::Import::Result failed_instances=[], num_inserts=1, ids=[]>

[rvillage]

思ったより確率高かった。。

- Time.new.usec % 1_000
+ Time.new.usec % 10_000