Overflow error

[Mathnerd314]

I get this error when running python3 ./extract-partitions.py --max-size 0 --debug -d dump:

2016-04-05 19:15:06,314 LGLAF.py: DEBUG: Using endpoints 83 (IN), 02 (OUT)
2016-04-05 19:15:06,376 extract-partitions: DEBUG: Opened fd 28 for disk
2016-04-05 19:15:06,409 extract-partitions: INFO: Skipping partition modem (mmcblk0p1), already found at dump/mmcblk0p1.bin
2016-04-05 19:15:06,417 extract-partitions: INFO: Skipping partition sbl1 (mmcblk0p2), already found at dump/mmcblk0p2.bin
<...>
2016-04-05 19:15:06,683 extract-partitions: INFO: Dumping partition system (mmcblk0p34) to dump/mmcblk0p34.bin (3489660928 bytes)
2016-04-05 19:15:06,684 partitions: DEBUG: Will read 3489660928 bytes at disk offset 469762048
Traceback (most recent call last):
  File "~/lglaf/partitions.py", line 75, in laf_open_disk
    yield fd_num
  File "./extract-partitions.py", line 61, in main
    dump_partitions(comm, disk_fd, args.outdir, args.max_size * 1024)
  File "./extract-partitions.py", line 46, in dump_partitions
    partitions.dump_partition(comm, disk_fd, out_path, part_offset, part_size, current_size)
  File "~/lglaf/partitions.py", line 167, in dump_partition
    data = laf_read(comm, disk_fd, read_offset // BLOCK_SIZE, chunksize)
  File "~/lglaf/partitions.py", line 83, in laf_read
    header, response = comm.call(read_cmd)
  File "~/lglaf/lglaf.py", line 168, in call
    header = self.read(0x20)
  File "~/lglaf/lglaf.py", line 148, in read
    buff = self._read(need, timeout=timeout)
  File "~/lglaf/lglaf.py", line 256, in _read
    array = self.usbdev.read(self.ep_in, 4*1024*1024, timeout=timeout)
  File "/nix/store/7s6iyjnm8r8c21ij9wiscrrcmda33dym-python3-3.4.4-env/lib/python3.4/site-packages/usb/core.py", line 988, in read
    self.__get_timeout(timeout))
  File "/nix/store/7s6iyjnm8r8c21ij9wiscrrcmda33dym-python3-3.4.4-env/lib/python3.4/site-packages/usb/backend/libusb1.py", line 833, in bulk_read
    timeout)
  File "/nix/store/7s6iyjnm8r8c21ij9wiscrrcmda33dym-python3-3.4.4-env/lib/python3.4/site-packages/usb/backend/libusb1.py", line 936, in __read
    _check(retval)
  File "/nix/store/7s6iyjnm8r8c21ij9wiscrrcmda33dym-python3-3.4.4-env/lib/python3.4/site-packages/usb/backend/libusb1.py", line 595, in _check
    raise USBError(_strerror(ret), ret, _libusb_errno[ret])
usb.core.USBError: [Errno 75] Overflow

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./extract-partitions.py", line 64, in <module>
    main()
  File "./extract-partitions.py", line 61, in main
    dump_partitions(comm, disk_fd, args.outdir, args.max_size * 1024)
  File "/nix/store/7s6iyjnm8r8c21ij9wiscrrcmda33dym-python3-3.4.4-env/lib/python3.4/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "~/lglaf/partitions.py", line 78, in laf_open_disk
    comm.call(close_cmd)
  File "~/lglaf/lglaf.py", line 169, in call
    validate_message(header, ignore_crc=True)
  File "~/lglaf/lglaf.py", line 126, in validate_message
    raise RuntimeError("Expected trailer %r, found %r" % (tail_exp, tail))
RuntimeError: Expected trailer b'\xff\xff\xff\xff', found b'\x00\x00\x00\x00'

I have an LG G2 LS980, here is the various lsusb output (-t, -v):

/:  Bus 07.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/5p, 480M
    |__ Port 2: Dev 12, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 2: Dev 18, If 0, Class=Communications, Driver=, 480M
        |__ Port 2: Dev 18, If 1, Class=CDC Data, Driver=, 480M
        |__ Port 2: Dev 18, If 2, Class=Vendor Specific Class, Driver=, 480M

Bus 007 Device 018: ID 1004:633a LG Electronics, Inc. 
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.10
  bDeviceClass          239 Miscellaneous Device
  bDeviceSubClass         2 
  bDeviceProtocol         1 Interface Association
  bMaxPacketSize0        64
  idVendor           0x1004 LG Electronics, Inc.
  idProduct          0x633a 
  bcdDevice            2.32
  iManufacturer           1 LG Electronics Inc.
  iProduct                2 LGE Android Phone
  iSerial                 3 LGLS9804ad507f4
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           98
    bNumInterfaces          3
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              500mA
    Interface Association:
      bLength                 8
      bDescriptorType        11
      bFirstInterface         0
      bInterfaceCount         2
      bFunctionClass          2 Communications
      bFunctionSubClass       2 Abstract (modem)
      bFunctionProtocol       1 AT-commands (v.25ter)
      iFunction               7 CDC Serial
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         2 Communications
      bInterfaceSubClass      2 Abstract (modem)
      bInterfaceProtocol      1 AT-commands (v.25ter)
      iInterface              5 CDC Abstract Control Model (ACM)
      CDC Header:
        bcdCDC               1.10
      CDC Call Management:
        bmCapabilities       0x00
        bDataInterface          1
      CDC ACM:
        bmCapabilities       0x02
          line coding and serial state
      CDC Union:
        bMasterInterface        0
        bSlaveInterface         1 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               9
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass        10 CDC Data
      bInterfaceSubClass      0 
      bInterfaceProtocol      0 
      iInterface              6 CDC ACM Data
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x01  EP 1 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        2
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass       255 Vendor Specific Class
      bInterfaceSubClass    255 Vendor Specific Subclass
      bInterfaceProtocol    255 Vendor Specific Protocol
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
Binary Object Store Descriptor:
  bLength                 5
  bDescriptorType        15
  wTotalLength           22
  bNumDeviceCaps          2
  USB 2.0 Extension Device Capability:
    bLength                 7
    bDescriptorType        16
    bDevCapabilityType      2
    bmAttributes   0x00000002
      HIRD Link Power Management (LPM) Supported
  SuperSpeed USB Device Capability:
    bLength                10
    bDescriptorType        16
    bDevCapabilityType      3
    bmAttributes         0x00
    wSpeedsSupported   0x000f
      Device can operate at Low Speed (1Mbps)
      Device can operate at Full Speed (12Mbps)
      Device can operate at High Speed (480Mbps)
      Device can operate at SuperSpeed (5Gbps)
    bFunctionalitySupport   1
      Lowest fully-functional device speed is Full Speed (12Mbps)
    bU1DevExitLat           1 micro seconds
    bU2DevExitLat         500 micro seconds
can't get debug descriptor: Resource temporarily unavailable
Device Status:     0x0000
  (Bus Powered)

[Mathnerd314]

The other partitions I dumped just by running the extract-partition script a bunch of times, but the system partition is too big for that.

[Lekensteyn]

This looks like a manual modification:

  File "~/lglaf/lglaf.py", line 256, in _read
    array = self.usbdev.read(self.ep_in, 4*1024*1024, timeout=timeout)

The default is:

        # device seems to use 16 KiB buffers.
        array = self.usbdev.read(self.ep_in, 2**14, timeout=timeout)

What if you change it back to the original value?

[Mathnerd314]

It gives the same error regardless.

I also tried sprinkling in half-second delays, but that doesn't seem to affect behavior either.

[Lekensteyn]

Are you able to obtain a USB capture using Wireshark while executing this? (dumpcap -i usbmon7 -w usb7.pcapng)

Based on http://libusb.sourceforge.net/api-1.0/packetoverflow.html I would guess that the device wants to send more than acceptable by the library. Can you check this against the capture file?

[Mathnerd314]

As mentioned on IRC, I did get a capture; summary:

6227    259.041777  host    7.19.2  LGLAF   96  READ(26,964469,15872,0)
6230    259.042650  7.19.3  host    LGLAF   9792    READ(26,964469,15872,0) <- has EOVERFLOW set
6231    259.043377  host    7.19.2  LGLAF   96  CLSE(26,0,0,0)
6234    259.044267  7.19.3  host    LGLAF   6240    Continuation

The EOVERFLOW signifies that the packet is too short; normal requests give responses of size 15968 and this only gave a response of size 9792.
After that, the exception handler calls CLSE, which doesn't get a proper CLSE response, but instead the 6176 overflow bytes from the request.

I'm not sure I trust the overflow to be correct, and there seems to be no way (from PyUSB) to access the data from the overflow, so it's probably best to fail and retry the request. But the error handler should flush the buffer instead of shutting everything down.

[Lekensteyn]

From http://www.makelinux.net/ldd3/chp-13-sect-3:

Generally, the error values -EPROTO, -EILSEQ, and -EOVERFLOW indicate hardware problems with the device, the device firmware, or the cable connecting the device to the computer.

Have you tried a different cable?

The data seems to be lost, the message after the CLSE is missing a header.

[Mathnerd314]

Have you tried a different cable?

I tried a different cable (fresh out-of-box Samsung phone cable), but I still gets overflow errors; I think the problem is the phone (it's pre-owned, maybe a loose connection).

The data seems to be lost, the message after the CLSE is missing a header.

Well, the message is the rest of the data (you can see the ext4 filesystem tags). So in theory calling read() yet again will return a CLSE header.

Meanwhile, I have opened PR #5, which recovers gracefully from overflow/timeout. This has solved the problem and let me create a full backup of the phone.