Debugging a bricked Ditto X4

Debugging a bricked Ditto X4

Disclaimer
I am in no way affiliated with TC Electronics or anyone/anything to do with them. I’m just an annoyed customer with a bricked Ditto X4 and some technical know-how.

Symptoms
When power is applied, all the lights come on in red and green (people call it orange?) A few seconds later all the LEDs go off and the single solid red LED at LOOP1 comes on and stays on. It doesn’t respond to button presses.

If I put a blank SD-card in the X4 instead of the one it came with, it never boots past all of the lights coming on – it doesn’t get as far as turning on the LOOP1 red LED.

TC’s Technical Support
Turns out they’re neither technical nor supportive. I filed a ticket with a detailed bug report of the problem, but they just referred me to a repair shop in Texas and closed the support ticket. I tried repeatedly to get more technical information about debugging the issue, but didn’t get anywhere. Like most of the TC support encounters I’ve read about, it wasn’t helpful.

DIY
So here we are. A bricked X4 and a small workshop full of electronic toys. Here’s tonight’s efforts:


(http://www.promusic.news/wp-content/uploads/2018/12/8c61Cgf.jpg)

Architecture
It turns out to be powered by a small NXP i.MX283 ARM9 system-on-chip. There’s an ATMEL ATtiny48 microcontroller, too. The ISP header is exposed but unpopulated.

My guess (to be confirmed) is that IC1 (64k serial EEPROM) contains a boot loader that’s executed by the MX283. The boot loader (U-Boot) presumably boots a Linux kernel off the SD-card, which loads the pedal’s control software. This is speculation at present.

I don’t know what the ATtiny is doing.

Debugging Interfaces
There are at least 3 UART debug interfaces onboard.
J15 (not marked on the image above) is a read-only serial debug log at 115200 8N1
J11 is labeled “debug comm port”. I haven’t connected to it yet.
J17 is a read-write serial console attached to the 1st-stage bootloader, U-Boot. 115200 8N1.

J17 Serial Console
I used a Sparkfun “beefy” serial/USB adaptor (https://www.sparkfun.com/products/13746) to hookup the RX, TX and GND pins at J17. This is what I get at power on:

U-Boot 2013.01 (Nov 16 2015 - 17:20:34)

CPU:   Freescale i.MX28 rev1.2 at 454 MHz
BOOT:  I2C #0, master, 3V3
I2C:   ready
DRAM:  128 MiB
MMC:   MXS MMC: 0
*** Warning - bad CRC, using default environment

In:    serial
Out:   serial
Err:   serial
Net:   FEC0 [PRIME]
Warning: FEC0 using MAC address from net device
, FEC1
Warning: FEC1 using MAC address from net device

Hit any key to stop autoboot:  0 
<DIR>       1024 .
<DIR>       1024 ..
<DIR>      12288 lost+found
** File not found /update.tar **
Failed to mount ext2 filesystem...
** Unrecognized filesystem type **
ext4fs_devread read outside partition 1073750502
ext4fs_devread read outside partition 8520198
Failed to mount ext2 filesystem...
** Unrecognized filesystem type **
ext4fs_devread read outside partition 8520198
Failed to mount ext2 filesystem...
** Unrecognized filesystem type **
internal error
TCE-X4 U-Boot >

The environment shows more information about what’s happening at boot time:

boot_recovery=if ext2ls mmc 0:${partition_recovery}; then if $update_present; then if ext2load mmc 0:${partition_recovery} ${loadaddr} /boot/splash.update; then vl3lcd ${loadaddr} ${filesize}; fi; else if ext2load mmc 0:${partition_recovery} ${loadaddr} /boot/splash.recovery; then vl3lcd ${loadaddr} ${filesize}; fi; fi; mw.l $loadaddr 0; if ext2load mmc 0:${partition_recovery} ${loadaddr} /boot/uImage; then if iminfo $loadaddr; then setenv env_state ${env_state},STATE_BOOT_FROM_RECOVERY; setenv partition ${partition_recovery}; run bootargs_ext; bootm ${loadaddr}; exit; fi; else setenv env_error ${env_error},ERROR_RECOVERY_UIMAGE; fi; else setenv env_error ${env_error},ERROR_FS_MOUNT_RECOVERY; fi; echo 'internal error'; exit;
boot_working=if ext2ls mmc 0:${partition_working}; then if ext2load mmc 0:${partition_recovery} ${loadaddr} /boot/splash.working; then vl3lcd ${loadaddr} ${filesize}; fi; mw.l $loadaddr 0; if ext2load mmc 0:${partition_working} ${loadaddr} /boot/uImage; then if iminfo $loadaddr; then setenv env_state ${env_state},STATE_BOOT_FROM_WORKING; setenv partition ${partition_working}; run bootargs_ext; bootm ${loadaddr}; exit; else setenv env_error ${env_error},ERROR_WORKING_UIMAGE; fi; else setenv env_error ${env_error},ERROR_WORKING_UIMAGE; fi; else setenv env_error ${env_error},ERROR_FS_MOUNT_WORKING; fi; x4led 0x80;run boot_recovery;
bootargs_ext=setenv bootargs console=ttyAM0,${baudrate}n8 rootfstype=ext4 root=/dev/mmcblk0p${partition} rw rootwait ssp1 env_error=${env_error} env_state=${env_state}
bootcmd=run bootcmd_vl3
bootcmd_vl3=run init_scripts; run set_contrast; run mount_scratch; run test_recovery_button; run test_pot_calib_button; run test_update_present
bootdelay=1
bootfile=uImage
env_error=,ERROR_FS_MOUNT_WORKING,ERROR_RECOVERY_UIMAGE
eth1addr=00:04:00:00:00:01
ethact=FEC0
ethaddr=00:04:00:00:00:00
ethprime=FEC0
init_scripts=setenv env_error; setenv env_state; setenv linuxmode; setenv update_present false;
loadaddr=0x42000000
mount_scratch=if ext2ls mmc 0:${partition_scratch}; then true; else setenv env_error ${env_error},ERROR_FS_MOUNT_SCRATCH; false; fi
partition_boot=1
partition_extended=4
partition_recovery=2
partition_scratch=5
partition_user=7
partition_working=3
set_contrast=mw.l 80040088 20000000; mw.l 80064008 c0000000; mw.l 80064010 00ff00e3; mw.l 80064020 001e00ff; mw.l 80064004 00000001;
stderr=serial
stdin=serial
stdout=serial

LED control
From the environment we can see that the boot loader is using the command “x4led” to change the color of the LEDs in response to different boot events. For example, at the end of the “boot_working” script in the environment we can see the command “x4led 0x80”, which turns the LOOP1 LED red. That’s exactly the failure I’m seeing!

I played with it a bit. Here’s “x4led 0xfc” (the LEDs look green but they’re actually orange):

http://www.promusic.news/wp-content/uploads/2018/12/ljDuej6.jpg

And here’s “x4led 0x66”:

http://www.promusic.news/wp-content/uploads/2018/12/UJ1VEAv.jpg

Partitions
This is interesting:

partition_boot=1
partition_extended=4
partition_recovery=2
partition_scratch=5
partition_user=7
partition_working=3

The boot loader expects to find the partition layout of the SD-card as shown. In Linux parlance, the “working” partition would be /dev/mmcblk0p3 and the “recovery” partition would be /dev/mmcblk0p2.

Boot Failures
Here’s what happens when we try to run the “boot_working” script:

TCE-X4 U-Boot > run boot_working
Failed to mount ext2 filesystem...
** Unrecognized filesystem type **
ext4fs_devread read outside partition 8389126
Failed to mount ext2 filesystem...
** Unrecognized filesystem type **
internal error
TCE-X4 U-Boot >

Here’s the definition of “boot_working” made pretty:

if ext2ls mmc 0:${partition_working}; then 
    if ext2load mmc 0:${partition_recovery} ${loadaddr} /boot/splash.working; then 
        vl3lcd ${loadaddr} ${filesize}
    fi

    mw.l $loadaddr 0

    if ext2load mmc 0:${partition_working} ${loadaddr} /boot/uImage; then 
        if iminfo $loadaddr; then 
            setenv env_state ${env_state},STATE_BOOT_FROM_WORKING
            setenv partition ${partition_working}
            run bootargs_ext
            bootm ${loadaddr}
            exit
        else
            setenv env_error ${env_error},ERROR_WORKING_UIMAGE
        fi
    else
        setenv env_error ${env_error},ERROR_WORKING_UIMAGE
    fi
else 
    setenv env_error ${env_error},ERROR_FS_MOUNT_WORKING
fi

x4led 0x80
run boot_recovery

So we can see that U-Boot starts, runs “boot_working”, fails, sets the LOOP1 red LED, then calls “run boot_recovery”, which also fails, causing U-Boot to drop to a command-line. We can also see the individual errors happening for first boot_working, then boot_recovery. Here’s the error from boot_working:

Failed to mount ext2 filesystem...
** Unrecognized filesystem type **
ext4fs_devread read outside partition 8389126

And here’s the error from boot_recovery:

Failed to mount ext2 filesystem...
** Unrecognized filesystem type **
internal error

Diagnosis
It looks like there’s nothing wrong with the pedal, which is great news. Instead, the problem seems to be that it can’t boot from the SD-card due to errors mounting the working partition (mmcblk0p3) and the recovery partition (mmcblk0p2).

Time to take a peek at that SD-card. To be continued.

read more

Source: http://forum.tcelectronic.com