5.2.x kernel: WD 8TB USB Drives: Unaligned partial completion (resid=78, sector_sz=512)

5.2.x kernel: WD 8TB USB Drives: Unaligned partial completion (resid=78, sector_sz=512)

[Justin Piszcz]

Attached 2 x brand new Western Digital 8TB USB 3.0 drives awhile back and
ran some file copy tests and was getting these warnings-- is there any way
to avoid these warnings?  I did confirm with parted that the partition was
aligned but this appears to be something related to the firmware on the
device according to [1] and [2]?

[1] https://patchwork.kernel.org/patch/9573203/
[2] https://patchwork.kernel.org/patch/9597797/

Part of the patch in [2] - if the firmware is buggy is there a patch or
workaround for these drives (when used in the USB enclosures) to avoid this
issue?

+	default:
+		/*
+		 * In case of bogus fw or device, we could end up having
+		 * an unaligned partial completion. Check this here and
force
+		 * alignment.
+		 */
+		resid = scsi_get_resid(SCpnt);
+		if (resid & (sector_size - 1)) {
+			sd_printk(KERN_INFO, sdkp,
+				"Unaligned partial completion (resid=%u,
sector_sz=%u)\n",
+				resid, sector_size);
+			resid = min(scsi_bufflen(SCpnt),
+				    round_up(resid, sector_size));
+			scsi_set_resid(SCpnt, resid);
+		}

Errors:

Jul 18 16:25:02 name kernel: [87305.605993] sd 8:0:0:0: [sdg] Unaligned
partial completion (resid=78, sector_sz=512)
Jul 18 16:25:02 name kernel: [87305.605993] sd 8:0:0:0: [sdg] Unaligned
partial completion (resid=78, sector_sz=512)

Jul 18 16:25:15 name kernel: [87318.292262] sd 7:0:0:0: [sdf] Unaligned
partial completion (resid=78, sector_sz=512)
Jul 18 16:25:15 name kernel: [87318.292262] sd 7:0:0:0: [sdf] Unaligned
partial completion (resid=78, sector_sz=512)

Re: 5.2.x kernel: WD 8TB USB Drives: Unaligned partial completion (resid=78, sector_sz=512)

[Markus Reichelt]

I can reproduce this with kernel 5.2.2 and a WD drive (WD40EZRZ-22GXCB0)
in a USB 3.0 enclosure featuring a JMicron SATA Bridge (ID 152d:2329)

I'm also interested in getting rid of the warnings.

* Justin Piszcz <jpiszcz@lucidpixels.com> wrote:

> Attached 2 x brand new Western Digital 8TB USB 3.0 drives awhile
> back and ran some file copy tests and was getting these warnings--
> is there any way to avoid these warnings?  I did confirm with
> parted that the partition was aligned but this appears to be
> something related to the firmware on the device according to [1]
> and [2]?
> 
> [1] https://patchwork.kernel.org/patch/9573203/
> [2] https://patchwork.kernel.org/patch/9597797/
> 
> Part of the patch in [2] - if the firmware is buggy is there a
> patch or workaround for these drives (when used in the USB
> enclosures) to avoid this issue?
> 
> +	default:
> +		/*
> +		 * In case of bogus fw or device, we could end up having
> +		 * an unaligned partial completion. Check this here and
> force
> +		 * alignment.
> +		 */
> +		resid = scsi_get_resid(SCpnt);
> +		if (resid & (sector_size - 1)) {
> +			sd_printk(KERN_INFO, sdkp,
> +				"Unaligned partial completion (resid=%u,
> sector_sz=%u)\n",
> +				resid, sector_size);
> +			resid = min(scsi_bufflen(SCpnt),
> +				    round_up(resid, sector_size));
> +			scsi_set_resid(SCpnt, resid);
> +		}
> 
> Errors:
> 
> Jul 18 16:25:02 name kernel: [87305.605993] sd 8:0:0:0: [sdg] Unaligned
> partial completion (resid=78, sector_sz=512)
> Jul 18 16:25:02 name kernel: [87305.605993] sd 8:0:0:0: [sdg] Unaligned
> partial completion (resid=78, sector_sz=512)
> 
> Jul 18 16:25:15 name kernel: [87318.292262] sd 7:0:0:0: [sdf] Unaligned
> partial completion (resid=78, sector_sz=512)
> Jul 18 16:25:15 name kernel: [87318.292262] sd 7:0:0:0: [sdf] Unaligned
> partial completion (resid=78, sector_sz=512)

-- 
left blank, right bald

Re: 5.2.x kernel: WD 8TB USB Drives: Unaligned partial completion (resid=78, sector_sz=512)

[Alan Stern]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: TEXT/PLAIN; charset=UTF-8, Size: 679 bytes --]

On Sat, 3 Aug 2019, Justin Piszcz wrote:

> Attached 2 x brand new Western Digital 8TB USB 3.0 drives awhile back and
> ran some file copy tests and was getting these warnings-- is there any way
> to avoid these warnings?  I did confirm with parted that the partition was
> aligned but this appears to be something related to the firmware on the
> device according to [1] and [2]?
> 
> [1] https://patchwork.kernel.org/patch/9573203/
> [2] https://patchwork.kernel.org/patch/9597797/

Just out of curiosity, why did you not address your email to the author 
or committer of this patch?  Surely they would be the people in the 
best position to answer your questions.

Alan Stern

RE: 5.2.x kernel: WD 8TB USB Drives: Unaligned partial completion (resid=78, sector_sz=512)

[Justin Piszcz]

-----Original Message-----
From: Alan Stern [mailto:stern@rowland.harvard.edu] 
Sent: Saturday, August 3, 2019 3:00 PM
To: Justin Piszcz
Cc: 'LKML'; linux-usb@vger.kernel.org; linux-scsi@vger.kernel.org
Subject: Re: 5.2.x kernel: WD 8TB USB Drives: Unaligned partial completion (resid=78, sector_sz=512)

On Sat, 3 Aug 2019, Justin Piszcz wrote:

> Attached 2 x brand new Western Digital 8TB USB 3.0 drives awhile back and
> ran some file copy tests and was getting these warnings-- is there any way
> to avoid these warnings?� I did confirm with parted that the partition was
> aligned but this appears to be something related to the firmware on the
> device according to [1] and [2]?
> 
> [1]�https://patchwork.kernel.org/patch/9573203/
> [2] https://patchwork.kernel.org/patch/9597797/

Just out of curiosity, why did you not address your email to the author 
or committer of this patch?  Surely they would be the people in the 
best position to answer your questions.

[ .. ]

Fixed.

I had been researching this issue and did not find any useful notes on the mailing lists--hopefully this will help others if there's a copy of the response on the mailing list.

These drives [1] [2] are the ones noted as affected when used on Linux [1] plugged in directly to a Linux PC (w/USB 3.0/XHCI) and [2] when in a USB enclosure with a JMicron SATA Bridge.

[1] WDBBGB0080HBK-NESN - https://www.wd.com/products/external-storage/my-book-new.html#WDBBGB0080HBK-NESN
[2] WD40EZRZ-22GXCB0 - https://www.wd.com/products/internal-storage/wd-blue-pc-desktop-hard-drive.html  (In a USB enclosure with a JMicron SATA Bridge - per Markus)

Regards,

Justin.

Re: 5.2.x kernel: WD 8TB USB Drives: Unaligned partial completion (resid=78, sector_sz=512)

[Martin K. Petersen]

Justin,

> Attached 2 x brand new Western Digital 8TB USB 3.0 drives awhile back and
> ran some file copy tests and was getting these warnings-- is there any way
> to avoid these warnings?  I did confirm with parted that the partition was
> aligned but this appears to be something related to the firmware on the
> device according to [1] and [2]?

Please send us the output of:

# sg_vpd -p bl /dev/sdN
# sg_vpd -p bdc /dev/sdN
# sg_readcap -l /dev/sdN

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: 5.2.x kernel: WD 8TB USB Drives: Unaligned partial completion (resid=78, sector_sz=512)

[Martin K. Petersen]

Justin,

> Attached 2 x brand new Western Digital 8TB USB 3.0 drives awhile back and
> ran some file copy tests and was getting these warnings-- is there any way
> to avoid these warnings?  I did confirm with parted that the partition was
> aligned but this appears to be something related to the firmware on the
> device according to [1] and [2]?

Please send us the output of:

# sg_vpd -p bl /dev/sdN
# sg_vpd -p bdc /dev/sdN
# sg_readcap -l /dev/sdN

-- 
Martin K. Petersen	Oracle Linux Engineering

RE: 5.2.x kernel: WD 8TB USB Drives: Unaligned partial completion (resid=78, sector_sz=512)

[Justin Piszcz]

-----Original Message-----
From: Martin K. Petersen [mailto:martin.petersen@oracle.com] 
Sent: Wednesday, August 7, 2019 10:04 PM
To: Justin Piszcz
Cc: 'LKML'; linux-usb@vger.kernel.org; linux-scsi@vger.kernel.org
Subject: Re: 5.2.x kernel: WD 8TB USB Drives: Unaligned partial completion (resid=78, sector_sz=512)


Justin,

> Attached 2 x brand new Western Digital 8TB USB 3.0 drives awhile back and
> ran some file copy tests and was getting these warnings-- is there any way
> to avoid these warnings?  I did confirm with parted that the partition was
> aligned but this appears to be something related to the firmware on the
> device according to [1] and [2]?

Please send us the output of:

# sg_vpd -p bl /dev/sdN
# sg_vpd -p bdc /dev/sdN
# sg_readcap -l /dev/sdN

[ .. ]

Disk type:
---
Disk /dev/sdf: 7.3 TiB, 8001562869760 bytes, 15628052480 sectors
Disk model: My Book 25EE

# sg_vpd -p bl /dev/sdf > /tmp/sg_vpd_bl.txt
# sg_vpd -p bdc /dev/sdf > /tmp/sg_vpd_bdc.txt
# sg_readcap -l /dev/sdf > /tmp/sg_readcap.txt
#  ls -l /tmp/sg_vpd_bl.txt /tmp/sg_vpd_bdc.txt /tmp/sg_readcap.txt
-rw-r--r-- 1 root root 421 Aug  8 04:26 /tmp/sg_readcap.txt
-rw-r--r-- 1 root root 244 Aug  8 04:25 /tmp/sg_vpd_bdc.txt
-rw-r--r-- 1 root root 972 Aug  8 04:25 /tmp/sg_vpd_bl.txt

Output:

sg_readcap.txt
Read Capacity results:
   Protection: prot_en=0, p_type=0, p_i_exponent=0
   Logical block provisioning: lbpme=0, lbprz=0
   Last LBA=15628052479 (0x3a38127ff), Number of logical blocks=15628052480
   Logical block length=512 bytes
   Logical blocks per physical block exponent=3 [so physical block length=4096 bytes]
   Lowest aligned LBA=0
Hence:
   Device size: 8001562869760 bytes, 7630885.0 MiB, 8001.56 GB, 8.00 TB

sg_vpd_bdc.txt
Block device characteristics VPD page (SBC):
  Nominal rotation rate: 5400 rpm
  Product type: Not specified
  WABEREQ=0
  WACEREQ=0
  Nominal form factor: 3.5 inch
  ZONED=0
  RBWZ=0
  BOCS=0
  FUAB=0
  VBULS=0
  DEPOPULATION_TIME=0 (seconds)

sg_vpd_bl.txt
Block limits VPD page (SBC):
  Write same non-zero (WSNZ): 0
  Maximum compare and write length: 0 blocks [Command not implemented]
  Optimal transfer length granularity: 8 blocks
  Maximum transfer length: 65535 blocks
  Optimal transfer length: 65535 blocks
  Maximum prefetch transfer length: 65535 blocks
  Maximum unmap LBA count: 0 [Unmap command not implemented]
  Maximum unmap block descriptor count: 0 [Unmap command not implemented]
  Optimal unmap granularity: 0 blocks [not reported]
  Unmap granularity alignment valid: false
  Unmap granularity alignment: 0 [invalid]
  Maximum write same length: 0 blocks [not reported]
  Maximum atomic transfer length: 0 blocks [not reported]
  Atomic alignment: 0 [unaligned atomic writes permitted]
  Atomic transfer length granularity: 0 [no granularity requirement
  Maximum atomic transfer length with atomic boundary: 0 blocks [not reported]
  Maximum atomic boundary size: 0 blocks [can only write atomic 1 block]

RE: 5.2.x kernel: WD 8TB USB Drives: Unaligned partial completion (resid=78, sector_sz=512)

[Justin Piszcz]

-----Original Message-----
From: Martin K. Petersen [mailto:martin.petersen@oracle.com] 
Sent: Wednesday, August 7, 2019 10:04 PM
To: Justin Piszcz
Cc: 'LKML'; linux-usb@vger.kernel.org; linux-scsi@vger.kernel.org
Subject: Re: 5.2.x kernel: WD 8TB USB Drives: Unaligned partial completion (resid=78, sector_sz=512)


Justin,

> Attached 2 x brand new Western Digital 8TB USB 3.0 drives awhile back and
> ran some file copy tests and was getting these warnings-- is there any way
> to avoid these warnings?  I did confirm with parted that the partition was
> aligned but this appears to be something related to the firmware on the
> device according to [1] and [2]?

Please send us the output of:

# sg_vpd -p bl /dev/sdN
# sg_vpd -p bdc /dev/sdN
# sg_readcap -l /dev/sdN

[ .. ]

Disk type:
---
Disk /dev/sdf: 7.3 TiB, 8001562869760 bytes, 15628052480 sectors
Disk model: My Book 25EE

# sg_vpd -p bl /dev/sdf > /tmp/sg_vpd_bl.txt
# sg_vpd -p bdc /dev/sdf > /tmp/sg_vpd_bdc.txt
# sg_readcap -l /dev/sdf > /tmp/sg_readcap.txt
#  ls -l /tmp/sg_vpd_bl.txt /tmp/sg_vpd_bdc.txt /tmp/sg_readcap.txt
-rw-r--r-- 1 root root 421 Aug  8 04:26 /tmp/sg_readcap.txt
-rw-r--r-- 1 root root 244 Aug  8 04:25 /tmp/sg_vpd_bdc.txt
-rw-r--r-- 1 root root 972 Aug  8 04:25 /tmp/sg_vpd_bl.txt

Output:

sg_readcap.txt
Read Capacity results:
   Protection: prot_en=0, p_type=0, p_i_exponent=0
   Logical block provisioning: lbpme=0, lbprz=0
   Last LBA=15628052479 (0x3a38127ff), Number of logical blocks=15628052480
   Logical block length=512 bytes
   Logical blocks per physical block exponent=3 [so physical block length=4096 bytes]
   Lowest aligned LBA=0
Hence:
   Device size: 8001562869760 bytes, 7630885.0 MiB, 8001.56 GB, 8.00 TB

sg_vpd_bdc.txt
Block device characteristics VPD page (SBC):
  Nominal rotation rate: 5400 rpm
  Product type: Not specified
  WABEREQ=0
  WACEREQ=0
  Nominal form factor: 3.5 inch
  ZONED=0
  RBWZ=0
  BOCS=0
  FUAB=0
  VBULS=0
  DEPOPULATION_TIME=0 (seconds)

sg_vpd_bl.txt
Block limits VPD page (SBC):
  Write same non-zero (WSNZ): 0
  Maximum compare and write length: 0 blocks [Command not implemented]
  Optimal transfer length granularity: 8 blocks
  Maximum transfer length: 65535 blocks
  Optimal transfer length: 65535 blocks
  Maximum prefetch transfer length: 65535 blocks
  Maximum unmap LBA count: 0 [Unmap command not implemented]
  Maximum unmap block descriptor count: 0 [Unmap command not implemented]
  Optimal unmap granularity: 0 blocks [not reported]
  Unmap granularity alignment valid: false
  Unmap granularity alignment: 0 [invalid]
  Maximum write same length: 0 blocks [not reported]
  Maximum atomic transfer length: 0 blocks [not reported]
  Atomic alignment: 0 [unaligned atomic writes permitted]
  Atomic transfer length granularity: 0 [no granularity requirement
  Maximum atomic transfer length with atomic boundary: 0 blocks [not reported]
  Maximum atomic boundary size: 0 blocks [can only write atomic 1 block]

Re: 5.2.x kernel: WD 8TB USB Drives: Unaligned partial completion (resid=78, sector_sz=512)

[Martin K. Petersen]

Justin,

>> Attached 2 x brand new Western Digital 8TB USB 3.0 drives awhile back
>> and ran some file copy tests and was getting these warnings-- is
>> there any way to avoid these warnings?  I did confirm with parted
>> that the partition was aligned but this appears to be something
>> related to the firmware on the device according to [1] and [2]?

> sg_vpd_bdc.txt
> Block device characteristics VPD page (SBC):
>   Nominal rotation rate: 5400 rpm
>   Product type: Not specified
>   WABEREQ=0
>   WACEREQ=0
>   Nominal form factor: 3.5 inch
>   ZONED=0

Damien: What can we do to limit the messages in cases like this? Would
it make sense to make the residual warning conditional on sd_is_zoned()?

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: 5.2.x kernel: WD 8TB USB Drives: Unaligned partial completion (resid=78, sector_sz=512)

[Damien Le Moal]

On 2019/08/12 19:12, Martin K. Petersen wrote:
> 
> Justin,
> 
>>> Attached 2 x brand new Western Digital 8TB USB 3.0 drives awhile back
>>> and ran some file copy tests and was getting these warnings-- is
>>> there any way to avoid these warnings?  I did confirm with parted
>>> that the partition was aligned but this appears to be something
>>> related to the firmware on the device according to [1] and [2]?
> 
>> sg_vpd_bdc.txt
>> Block device characteristics VPD page (SBC):
>>   Nominal rotation rate: 5400 rpm
>>   Product type: Not specified
>>   WABEREQ=0
>>   WACEREQ=0
>>   Nominal form factor: 3.5 inch
>>   ZONED=0
> 
> Damien: What can we do to limit the messages in cases like this? Would
> it make sense to make the residual warning conditional on sd_is_zoned()?
> 

These WD drives are regular disks, not SMR. Making the warning conditional on
sd_is_zoned() will not reduce the amount of messages. REPORT ZONES is the only
command that could result in a resid not being aligned to the block size since
by definition the command reply is composed of 64B zone descriptors. But that
command is now processed through a device method and is not a REQ_OP_XXX
anymore, so as an internal req, it does not go through sd_done() for completion
right ? All other zone commands either have no buffer, or the exact same
requirement as regular disks, expecting a block aligned resid. So I think that
using sd_is_zoned() is not relevant to this problem.

Bottom line: this USB adapter is weird and likely triggers all the unaligned
resid. I do get regularly reports of similar problem with SAS HBAs, all always
fixed with HBA FW updates. Not sure if there is a FW available for these drives.
I will ask internally.

In the mean time, rate limiting or removing the sd_printk() call may be the only
option. Failing all commands with an invalid resid would be safer I guess, but
will at best likely cause a lot of retry on these buggy devices, and break the
drive operation/perfomance completely in the worst case.

Or the usb mass storage device driver could add some silent forced resid
alignment too.

Best regards.

-- 
Damien Le Moal
Western Digital Research

Re: 5.2.x kernel: WD 8TB USB Drives: Unaligned partial completion (resid=78, sector_sz=512)

[Damien Le Moal]

On 2019/08/12 19:12, Martin K. Petersen wrote:
> 
> Justin,
> 
>>> Attached 2 x brand new Western Digital 8TB USB 3.0 drives awhile back
>>> and ran some file copy tests and was getting these warnings-- is
>>> there any way to avoid these warnings?  I did confirm with parted
>>> that the partition was aligned but this appears to be something
>>> related to the firmware on the device according to [1] and [2]?
> 
>> sg_vpd_bdc.txt
>> Block device characteristics VPD page (SBC):
>>   Nominal rotation rate: 5400 rpm
>>   Product type: Not specified
>>   WABEREQ=0
>>   WACEREQ=0
>>   Nominal form factor: 3.5 inch
>>   ZONED=0
> 
> Damien: What can we do to limit the messages in cases like this? Would
> it make sense to make the residual warning conditional on sd_is_zoned()?
> 

Justin,

Can you send the output of "lsscsi" for these drives ? I need the exact disk
model ref name and FW version to see if there is an update for this problem, if
it is a known one. If it is not, I will signal it and get a fix started.

Best regards.

-- 
Damien Le Moal
Western Digital Research

Re: 5.2.x kernel: WD 8TB USB Drives: Unaligned partial completion (resid=78, sector_sz=512)

[Justin Piszcz]

On Tue, Aug 13, 2019 at 6:48 AM Damien Le Moal <Damien.LeMoal@wdc.com> wrote:
>
> On 2019/08/12 19:12, Martin K. Petersen wrote:
> >
> > Justin,
> >
> >>> Attached 2 x brand new Western Digital 8TB USB 3.0 drives awhile back
> >>> and ran some file copy tests and was getting these warnings-- is
> >>> there any way to avoid these warnings?  I did confirm with parted
> >>> that the partition was aligned but this appears to be something
> >>> related to the firmware on the device according to [1] and [2]?
> >
> >> sg_vpd_bdc.txt
> >> Block device characteristics VPD page (SBC):
> >>   Nominal rotation rate: 5400 rpm
> >>   Product type: Not specified
> >>   WABEREQ=0
> >>   WACEREQ=0
> >>   Nominal form factor: 3.5 inch
> >>   ZONED=0
> >
> > Damien: What can we do to limit the messages in cases like this? Would
> > it make sense to make the residual warning conditional on sd_is_zoned()?
> >
>
> Justin,
>
> Can you send the output of "lsscsi" for these drives ? I need the exact disk
> model ref name and FW version to see if there is an update for this problem, if
> it is a known one. If it is not, I will signal it and get a fix started.
>

Requested lsscsi output:


[7:0:0:0]    disk    WD       My Book 25EE     4004  /dev/sdf
[7:0:0:1]    enclosu WD       SES Device       4004  -
[8:0:0:0]    disk    WD       My Book 25EE     4004  /dev/sdg
[8:0:0:1]    enclosu WD       SES Device       4004  -

Host: scsi7 Channel: 00 Target: 00 Lun: 00
  Vendor: WD       Model: My Book 25EE     Rev: 4004
  Type:   Direct-Access                    ANSI SCSI revision: 06
Host: scsi7 Channel: 00 Target: 00 Lun: 01
  Vendor: WD       Model: SES Device       Rev: 4004
  Type:   Enclosure                        ANSI SCSI revision: 06
Host: scsi8 Channel: 00 Target: 00 Lun: 00
  Vendor: WD       Model: My Book 25EE     Rev: 4004
  Type:   Direct-Access                    ANSI SCSI revision: 06
Host: scsi8 Channel: 00 Target: 00 Lun: 01
  Vendor: WD       Model: SES Device       Rev: 4004
  Type:   Enclosure                        ANSI SCSI revision: 06

[7:0:0:0]    disk    WD       My Book 25EE     4004  /dev/sdf
  state=running queue_depth=1 scsi_level=7 type=0 device_blocked=0 timeout=30
[7:0:0:1]    enclosu WD       SES Device       4004  -
  state=running queue_depth=1 scsi_level=7 type=13 device_blocked=0 timeout=30
[8:0:0:0]    disk    WD       My Book 25EE     4004  /dev/sdg
  state=running queue_depth=1 scsi_level=7 type=0 device_blocked=0 timeout=30
[8:0:0:1]    enclosu WD       SES Device       4004  -
  state=running queue_depth=1 scsi_level=7 type=13 device_blocked=0 timeout=30

[7:0:0:0]    disk    none                              /dev/sdf
[7:0:0:1]    enclosu 5000cca252e63312  -
[8:0:0:0]    disk    none                              /dev/sdg
[8:0:0:1]    enclosu 5000cca252e6a2f9  -

[7:0:0:0]    disk    WD       My Book 25EE     4004  /dev/sdf   8.00TB
[7:0:0:1]    enclosu WD       SES Device       4004  -               -
[8:0:0:0]    disk    WD       My Book 25EE     4004  /dev/sdg   8.00TB
[8:0:0:1]    enclosu WD       SES Device       4004  -               -


[7:0:0:0]    disk    usb:1-1.2:1.0                   /dev/sdf
[7:0:0:1]    enclosu usb:1-1.2:1.0                   -
[8:0:0:0]    disk    usb:1-1.1:1.0                   /dev/sdg
[8:0:0:1]    enclosu usb:1-1.1:1.0                   -

# smartctl -a /dev/sdg
/dev/sdg: Unknown USB bridge [0x1058:0x25ee (0x4004)]

# smartctl -a /dev/sdf
/dev/sdf: Unknown USB bridge [0x1058:0x25ee (0x4004)]

Regards,

Justin.