ZFS

eBay HDD Adventures

Reformatting from 520 to 512-byte sector size


Aaron Runkle
@aaronrunkle
eBay HDD Adventures

So recently one of the hard drives failed on my 2TB zfs storage pool/array on Ubuntu Server 18 LTS.  In fact, it had started to degrade my entire pool’s performance.  I decided to buy a replacement drive off of eBay for cheap, and it came formatted with 520-byte sectors versus the typical standard of 512.  Apparently some EMC or NetApp RAID arrays are configured for a the slightly larger sector size.  Here are some example errors I ran into:

ZFS ZPOOL Add

sudo zpool labelclear -f /dev/disk/by-id/scsi-35000cca01b8bebda
failed to check state for /dev/disk/by-id/scsi-35000cca01b8bebda

ZoL ZPOOL LabelClear

sudo zpool labelclear -f /dev/disk/by-id/scsi-35000cca01b8bebda
failed to check state for /dev/disk/by-id/scsi-35000cca01b8bebda

GDISK

sudo gdisk /dev/disk/by-id/scsi-35000cca01b8bebda
GPT fdisk (gdisk) version 1.0.3

Problem reading disk in BasicMBRData::ReadMBRData()!
Warning! Read error 22; strange behavior now likely!
Warning! Read error 22; strange behavior now likely!
Partition table scan:
  MBR: MBR only
  BSD: not present
  APM: not present
  GPT: not present


***************************************************************
Found invalid GPT and valid MBR; converting MBR to GPT format
in memory. THIS OPERATION IS POTENTIALLY DESTRUCTIVE! Exit by
typing 'q' if you don't want to convert your MBR partitions
to GPT format!
***************************************************************

Command (? for help): w
Caution! Secondary header was placed beyond the disk's limits! Moving the
header, but other problems may occur!
Warning! The claimed last usable sector is incorrect! Do you want to correct
this problem? (Y/N): y
Have adjusted the second header and last usable sector value.

Partition(s) in the protective MBR are too big for the disk! Creating a
fresh protective or hybrid MBR is recommended.

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/disk/by-id/scsi-35000cca01b8bebda.
Unable to save backup partition table! Perhaps the 'e' option on the experts'
menu will resolve this problem.
Warning! An error was reported when writing the partition table! This error
MIGHT be harmless, or the disk might be damaged! Checking it is advisable.

SMARTCTL

sudo smartctl -a /dev/disk/by-id/scsi-35000cca01b8bebda
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.15.0-24-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               HITACHI
Product:              HUS72302CLAR2000
Revision:             C1D6
Compliance:           SPC-4
User Capacity:        2,000,398,933,520 bytes [2.00 TB]
Logical block size:   520 bytes
Rotation Rate:        7200 rpm
Form Factor:          3.5 inches
Logical Unit id:      0x5000cca01b8bebda
Serial number:        xxxxxxxx
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Tue Sep  4 00:42:32 2018 CDT
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Disabled or Not Supported

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:     37 C
Drive Trip Temperature:        85 C

Manufactured in week 47 of year 2012
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  4
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  933
Elements in grown defect list: 0

Vendor (Seagate) cache information
  Blocks sent to initiator = 0

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0     2555         0      2555   15462919       4844.619           0
write:         0   132436         0    132436       9406       3633.211           0
verify:        0       56         0        56       5178      31032.620           0

Non-medium error count:        0

SMART Self-test log
Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]
     Description                              number   (hours)
# 1  Foreground long   Completed                   -   22589                 - [-   -    -]
# 2  Background short  Completed                   -   22585                 - [-   -    -]

(I also ran a short and long SMART test via the following to ensure the drive itself was OK.  Hint: It works fine, no errors)

# Short test
sudo smartctl -t short /dev/disk/by-id/scsi-35000cca01b8bebda

# Long test (-C forces to run in the foregound.  WARNING:  You can't cancel this with -X if run with -C!)
sudo smartctl -t long -C /dev/disk/by-id/scsi-35000cca01b8bebda

So, to recap, the drive is OK as proven by the short and long SMART tests, it’s just that the other tools can’t seem to read it.  After some googling, this FreeNAS thread reported a similar issue and gave me the golden one-liner to fix: (Note, it takes a while so you probably want to run it in screen or tmux)

sudo sg_format --verbose --format --size=512 --six /dev/disk/by-id/scsi-35000cca01b8bebda

I won’t pretend to know exactly what the –six mode change does (even after reading the man page), but it worked!

sudo sg_format --verbose --format --size=512 --six /dev/disk/by-id/scsi-35000cca01b8bebda
    HITACHI   HUS72302CLAR2000  C1D6   peripheral_type: disk [0x0]
      PROTECT=1
      << supports protection information>>
      Unit serial number: xxxxxxx       
      LU name: 5000cca01b8bebda
    mode sense (6) cdb: 1a 00 01 00 fc 00 
Mode Sense (block descriptor) data, prior to changes:
  Number of blocks=3846921026 [0xe54b5b42]
  Block size=520 [0x208]
    mode select (6) cdb: 15 11 00 00 18 00 

A FORMAT UNIT will commence in 15 seconds
    ALL data on /dev/disk/by-id/scsi-35000cca01b8bebda will be DESTROYED
        Press control-C to abort

A FORMAT UNIT will commence in 10 seconds
    ALL data on /dev/disk/by-id/scsi-35000cca01b8bebda will be DESTROYED
        Press control-C to abort

A FORMAT UNIT will commence in 5 seconds
    ALL data on /dev/disk/by-id/scsi-35000cca01b8bebda will be DESTROYED
        Press control-C to abort
    format unit cdb: 04 18 00 00 00 00 

Format unit has started

And profit!

sudo zpool add tank spare /dev/disk/by-id/scsi-35000cca01b8bebda
sudo zpool status
  pool: tank
 state: ONLINE
  scan: scrub repaired 0B in 2h48m with 0 errors on Sat Sep  1 04:48:14 2018
config:

        NAME                        STATE     READ WRITE CKSUM
        tank                        ONLINE       0     0     0
          raidz2-0                  ONLINE       0     0     0
            [drives]
        spares
          scsi-35000cca01b8bebda    AVAIL   

errors: No known data errors