5 Oct 2024

Previous journal:	Next journal:
0217-2024-09-29.md	0219-2024-10-07.md

GFMPW-1 basic bring-up

Basic connection and proof of life

To an extent, you can use the chipIgnite Caravel eval board README for information on the board layout (even though it's slightly different for GFMPW).

0206 has more detail about what I did with a non-gf180 board, and it's basically the same, but here's a high-level summary:

My host PC is Windows 11 Home.
I've got an Ubuntu 22.04 VM in VirtualBox.
Linux user must be part of the dialout group to access the FTDI USB-serial interface, so: sudo adduser $USER dialout -- then reboot the VM.
Remove the M.2 card screw from the board, insert an M.2 card (push in firmly to snap in), replace the screw gently.
Plugging the board into my host PC recognises it in Windows. Using VirtualBox, I do a "USB pass through" to my Linux VM of the device "FTDI Single RS232-HS [0900]" (and this can be made permanent).
When connecting the board to Linux, sudo dmesg -T --follow should reveal the device connecting.

Clone the caravel_board repo:

git clone https://github.com/efabless/caravel_board # 286MB
cd caravel_board

Create a Python VENV:

python3 -m venv --prompt caravel_board .venv
echo '*' >> .venv/.gitignore
source .venv/bin/activate
python3 --version # Python 3.10.12

Do: pip3 install pyftdi

Create a "mapping" for lower-level usermode control of the FTDI USB-serial device, i.e. run lsusb to get the VID:PID (probably 0403:6014) and (as root) put into a new file called /etc/udev/rules.d/99-ftdi-everyone.rules:

# Per https://groups.google.com/g/weewx-user/c/kol0udZNuyc/m/1WhtZF0kBAAJ
# ...and https://github.com/algofoogle/journal/blob/master/0206-2024-06-26.md
# ...this ensures all users get full access to the FTDI USB interface on
# the caravel_board hardware, without having to be root:

SUBSYSTEM=="usb", ATTR{idVendor}=="0403", ATTR{idProduct}=="6014", MODE="666"

...then: sudo service udev restart and unplug and replug the board, and pass back thru to Linux VM (if necessary).

Try HKDebug:

cd firmware/gf180/util
python3 caravel_hkdebug.py

...and expect to see:

Success: Found one matching FTDI device at ftdi://ftdi:232h:1:4/1
Caravel data:
   mfg        = 0456
   product    = 20
   project ID = 1801dc4f
   project ID = f23b8018

Reading GFMPW-1 Caravel registers

Caravel's HKSPI allows the FTDI chip to control it to read and write certain 'registers', which have a default power-on state.

In HKDebug (i.e. python3 caravel_hkdebug.py), option 1 will read the first 19 registers, and here is their power-on state decoded:

reg 0x00 = b'00'  reg_hkspi_status          UNDEFINED/RESERVED
reg 0x01 = b'04'  reg_hkspi_chip_id[19:16]  ...
reg 0x02 = b'56'  reg_hkspi_chip_id[15:8]   ...Manufacturer ID: 0x456 (Efabless)
reg 0x03 = b'20'  reg_hkspi_chip_id[7:0]    Product ID (0x20): Caravel on GF180?
reg 0x04 = b'f2'  reg_hkspi_user_id[31:24]  ...
reg 0x05 = b'3b'  reg_hkspi_user_id[23:16]  ...
reg 0x06 = b'80'  reg_hkspi_user_id[15:8]   ...
reg 0x07 = b'18'  reg_hkspi_user_id[7:0]    ...32b project ID (0x1801dc4f, reversed bits)
reg 0x08 = b'02'  reg_hkspi_pll_ena         xxxxxx_1_0 = Use manual DCO; Disable DCO
reg 0x09 = b'01'  reg_hkspi_pll_bypass      xxxxxxx_1 = Bypass enabled; use direct ext. clk
reg 0x0a = b'00'  reg_hkspi_irq             xxxxxxx_0 = No action
reg 0x0b = b'00'  reg_hkspi_reset           xxxxxxx_0 = No action
reg 0x0c = b'00'  reg_hkspi_trap            ??
reg 0x0d = b'ff'  reg_hkspi_pll_trim[7:0]   ...
reg 0x0e = b'ef'  reg_hkspi_pll_trim[15:8]  ...
reg 0x0f = b'ff'  reg_hkspi_pll_trim[23:16] ...
reg 0x10 = b'03'  reg_hkspi_pll_trim[25:24] ...trim is 0x03ffefff
reg 0x11 = b'12'  reg_hkspi_pll_source      xx_010_010 = div-2 for user_clock2 and wb_clk_i
reg 0x12 = b'04'  reg_hkspi_pll_divider     xxx_00100 = mul-4

Edit caravel_hkdebug.py to add this extra "option 15" to read ALL of the HKSPI address space:

    elif k == '15':
        data = slave.exchange([CARAVEL_STREAM_READ, 0x00], 256)
        # Print header:
        print("   " + " ".join(f"-{i:x}" for i in range(16)))
        # Print data in hex format with 16 bytes per line:
        for i in range(0, len(data), 16):
            print(f"{i>>4:x}- ", end="")
            print(" ".join(f"{b:02X}" for b in data[i:i+16]))

Run HKDebug again and use option 15:

   -0 -1 -2 -3 -4 -5 -6 -7 -8 -9 -a -b -c -d -e -f
0- 00 04 56 20 F2 3B 80 18 02 01 00 00 00 FF EF FF
1- 03 12 04 00 00 00 00 00 00 00 00 00 00 00 09 00
2- 09 00 07 00 87 00 07 00 07 00 07 00 07 00 07 00
3- 07 00 07 00 07 00 07 00 07 00 07 00 07 00 07 00
4- 07 00 07 00 07 00 07 00 07 00 07 00 07 00 07 00
5- 07 00 07 00 07 00 07 00 07 00 07 00 07 00 07 00
6- 07 00 07 00 07 00 09 00 09 00 80 00 30 90 00 00
7- 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8- 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9- 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a- 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b- 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c- 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d- 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e- 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f- 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

user_defines.v shows the meaning of GPIO config values that appear as 16 bit values (big-endian) starting at HKSPI register address 0x1d,0x1e (for GPIO 0) and continuing up to address 0x67,0x68 (for GPIO 37). Note that these power-on values are NOT the actual currently-configured state of the GPIOs as defined in user_defines.v; they are the values-in-waiting that WOULD be loaded as configuration on the next activation of the GPIO configuration shift register cycle.

Checking the clock sources

There's a 10MHz oscillator on the Caravel eval board, feeding the clock input of the Caravel SoC. This can be measured via the xclk pin on the right-side header of the board.
The oscillator can be disabled, if needed; there is a header pin pair that can be shorted to do this -- it is near X1 (just below LED D1). When disabled, an external clock source can be provided via the xclk header pin on the right-side of the board. However, it's probably not necessary to do this as the on-board DLL can be used to get different clock speeds inside the chip.
Caravel has two internal clock sources: wb_clk_i (core clock, shared with the user project area and with the Caravel CPU) and user_clock2 which can be split off from the core clock source using the DLL.
You can make Caravel output both clocks on GPIOs 14 and 15 respectively by using a series of register writes (with option 14) as follows:
- Enable clock outputs:
  - Reg 0x1b, write data 6 (bit 1 is GPIO 14 clock output enable, bit 2 is GPIO 15 clock output enable).
- Configure GPIO 14 as a "management output" (GPIO_MODE_MGMT_STD_OUTPUT):
  - Reg 0x39, write data 0
  - Reg 0x3A, write data 0xb
- Repeat for GPIO 15:
  - Reg 0x3B, write data 0
  - Reg 0x3C, write data 0xb
- Activate the shift register that loads all GPIO config:
  - Reg 0x13, write 1
It should now be possible to see 10MHz clock outputs on GPIOs 14 and 15 (with 14 probably very slightly lagging 15, i.e. by about 8ns).
If you want, you could now try writing to the Caravel DLL registers to test different clocks.

Testing the Caravel CPU with basic blink firmware

The RISC-V toolchain needs to be installed, and I did that as described here (i.e. I did NOT follow the instructions linked from here). In particular, I did this:

sudo -s # Runs these commands below as root...
cd /opt
RISCVTOOLS=riscv64-unknown-elf-toolchain-10.2.0-2020.12.8-x86_64-linux-ubuntu14
wget https://static.dev.sifive.com/dev-tools/freedom-tools/v2020.12/$RISCVTOOLS.tar.gz
tar xf $RISCVTOOLS.tar.gz
rm $RISCVTOOLS.tar.gz
ln -s $RISCVTOOLS riscv-toolchain
exit # Exit 'sudo' session.
# Test:
/opt/riscv-toolchain/bin/riscv64-unknown-elf-gcc --version

Make a copy of the existing gf180 blink example in caravel_board:
```
cd firmware/gf180
cp -R blink basic-blink
cd basic-blink
```

Replace blink.c with this minimal example, which intentionally does NOT attempt to reconfigure any of the GPIOs:

#include <defs.h>

void delay(const int d)
{
    // Configure timer for a single-shot countdown:
    reg_timer0_config = 0;
    reg_timer0_data = d;
    reg_timer0_config = 1;
    // Loop, waiting for value to reach zero:
    reg_timer0_update = 1;  // latch current value
    while (reg_timer0_value > 0) {
        reg_timer0_update = 1;
    }
}

void main()
{
    // Configure Caravel's own "gpio" pin as an output:
    reg_gpio_mode1 = 1;
    reg_gpio_mode0 = 0;
    reg_gpio_ien = 1;
    reg_gpio_oeb = 0;

    while (1) {
        reg_gpio_out = 1;   // LED D3 OFF
        delay(2000000);
        reg_gpio_out = 0;   // LED D3 ON
        delay(2000000);
    }
}

Replace Makefile by doing: cp ../../chipignite/demos/Makefile . -- Note that, importantly, this Makefile specifies that the target RISC-V architecture is RV32I (via -march=rv32i)
Edit this new Makefile to ensure:
- TOOLCHAIN_PATH is set to /opt/riscv-toolchain/bin/
- The line setting PATTERN = demos is replaced with PATTERN = blink
- Any references to demos are replaced with $(PATTERN)
Compile and flash this blink firmware to the Caravel board's SPI Flash ROM by doing: make flash
- NOTE: While the flash is being erased and then programmed, LED D2 should be lit and D1 should blink.
LED D3 should blink after that.

Testing the mux

Inside caravel_board, create firmware/gf180/mux_test/:
- Copy in Makefile from above (and modify it to set PATTERN = mux_test)
- Copy in mux_tests.c from algofoogle-multi-caravel repo as mux_test.c (NOTE: There's a newer version of this file but it relies on firmware_apis.h which I don't yet know how to properly integrate from here into the caravel_board flow).
- Replace the stub.c include to stub.h (because Makefile already includes stub.c).
- Replace:
  - "reg_la0..." => "reg_la2..."
  - "reg_la1..." => "reg_la3..."
  - Why? See reg_la1_data_in weirdness below, if you care.
- Add in a blink loop at the end, based on the example above.
- Note that this mux_test.c firmware will:
  - Enable output for Caravel's gpio pin.
  - Select design 13 (one of the basic output test patterns).
  - Configure GPIO[37:8] to be bidirectional, thus allowing output control from the design.
  - Then select design 14 (inverted output test patterns).
  - Enter endless loop blinking the gpio pin.
Build/flash the firmware: make flash
Once this firmware runs and enters the blinking state, design 14 should be selected, which is a simple test pattern:
- Should be able to probe IO[31:24] and see 10101010 outputs, then on IO[23:16] see 01010101.

Other notes

The mux registers power up in a random state, which means the active project (the one controlling the IOs) is selected at random.
Additionally, its control lines (e.g. reset) might also be random.
Power-cycling is the only way to (hopefully) get different states on the registers, and they have a tendency to hold their state as they discharge, and otherwise tend to lean more to a given state at power-on (so it's not completely random; there are biases).
All designs have the same clock source (wb_clk_i). They can be reset through a variety of means.
Across various power-cycles, I was sometimes getting:
- Various GPIOs being stuck high or low
- IO28 oscillating at 2.5MHz -- I later worked out this was Diego's addr[0] pin. Sometimes it wouldn't work: GPIO[6] is Diego's reset, and this was floating.
- It looked like some pins were behaving as inputs (i.e. they could pick up noise, so seemingly weren't being driven)
- These were all clues that the mux might be working, as well as the designs, but that I was controlling it incorrectly.
- I got lucky when it seemed to start up one time with design 11 because this allows for some basic debug.

Next steps

See my next journal entry: 0219: GFMPW-1 bring-up (continued)

TODO

Make better firmware that uses helper macros to control mux
Provide example firmware and expected GPIO behaviour, inc. loopbacks
Provide links to all the various Verilog sources
Explain IO mapping to designs
List IOs for designs

`reg_la1_data_in` weirdness

Note

This section is a lot more information than you want, so here's a TL;DR:

In sky130 projects, if you want your firmware to assert something on the LA bank 0 pins (la_data_in[31:0]), write to the reg_la0_data register. If you want it to sense something on the corresponding LA bank 0 pins (la_data_out[31:0]), read from the reg_la0_data_in register. reg_la1_* is the next bank up.
In GFMPW projects, if you want your firmware to assert something on the LA bank 0 pins (la_data_in[31:0]), write to the reg_la2_data register. If you want it to sense something on the corresponding LA bank 0 pins (la_data_out[31:0]), read from the reg_la2_data_in register. reg_la3_* is the next bank up. Don't use reg_la1/0_* -- they exist but don't do what you want.

The above is all you really need to know to make working firmware that controls the mux. If you care about all the details, though, you can read on...

For the Caravel CPU to assert something on an LA pin (i.e. la_data_in[*]), it must write to one of the reg_la*_data registers. To sense an LA pin (i.e. la_data_out[*]) it must read from one of the reg_la*_data_in registers. This was unclear because there is still evidence of this in the 'caravel' repo but it doesn't work this way anymore.

Additionally for GFMPW, though, while the registers (namely their addresses in firmware headers) are correctly defined for sky130, they are not for gf180. In particular, if you have firmware for a GFMPW chip that accesses reg_la3_*/reg_la2_* (which GFMPW shouldn't have, since it only has 64 LA channels instead of 128, thus it lacks LA banks 3 and 2), it will actually be accessing the registers of LA banks 1 and 0 respectively (as though you were accessing reg_la1_*/reg_la0_*). If you try actually using reg_la0_data, it will be wrong: it will be accessing a register that is off by +8. This is a quirk of the generic defs.h pattern and the way the LA bank registers are reverse-numbered (I guess big-endian).

Evidently, this is an oddity that's known and accepted by the GFMPW version of the "firmware APIs", where reg_la3/2_* are used intentionally.

So, to compensate for this on GFMPW, either use the LogicAnalyzer_* functions (e.g. LogicAnalyzer_write), or use reg_la3/2_* directly, or create some abstraction of your own.

Additional evidence:

I've got a sky130 caravel_user_project counter demo chip (CI2304-E3), where the _data_in version means "into the CPU from mprj", while _data means "out from CPU, into mprj".
- I can confirm this successfully uses _data_in for the CPU to read from mprj.
- Matt's notes and demos.c seem to back this up also.
- _data_in regs are defined in caravel_mgmt_soc_litex => CSR_LA_IN_ADDR
I first picked up on this GFMPW discrepancy when I was compiling via caravel_board's firmware/gf180: it appeared that _data_in was the reverse, i.e. "out from CPU, into mprj". Certainly my mux control worked when I treated it this way (i.e. writing to reg_la0_data_in instead of reg_la0_data), and didn't work otherwise, and I speculated that this apparent reversal was due to a misalignment of register addresses, as a result of GFMPW's Caravel having only two LA banks instead of four.

The problem is that the register address ranges are different, but defs.h assumes always 4 banks of LA registers, and so its arithmetic is off and by a quirk of how they're reverse-numbered, reg_la0_* and reg_la1_* of each "set" ends up referring to the correct banks but of the NEXT set down, e.g.:

reg_la0_iena should refer to LA_IEN[31:0], but on GF180 instead refers to LA_OE[31:0].
reg_la0_oenb should refer to LA_OE[31:0], but on GF180 instead refers to LA_IN[31:0].
reg_la0_data_in should be LA_IN[31:0], but on GF180 instead refers to LA_OUT[31:0].
reg_la0_data should be LA_OUT[31:0], but on GF180 is invalid.
reg_la3_iena should be absent from GF180, but it refers to LA_IEN[63:32].

Here's a table where I try to illustrate this:

CSR base	Name	sky130	gf180
CSR_LA_BASE	-	0xf0003000	0xf0003000

CSR_LA_IEN_ADDR	-	0xf0003000	0xf0003000

CSR_LA_IEN_ADDR + 0	reg_la3_iena	0xf0003000: LA_IEN[127:96]	0xf0003000: LA_IEN[63:32]
CSR_LA_IEN_ADDR + 4	reg_la2_iena	0xf0003004: LA_IEN[95:64]	0xf0003004: LA_IEN[31:0]
CSR_LA_IEN_ADDR + 8	reg_la1_iena	0xf0003008: LA_IEN[63:32]	0xf0003008: ⚠️
CSR_LA_IEN_ADDR + 12	reg_la0_iena	0xf000300c: LA_IEN[31:0]	0xf000300c: ⚠️

CSR_LA_OE_ADDR	-	0xf0003010	0xf0003008

CSR_LA_OE_ADDR + 0	reg_la3_oenb	0xf0003010: LA_OE[127:96]	0xf0003008: LA_OE[63:32]
CSR_LA_OE_ADDR + 4	reg_la2_oenb	0xf0003014: LA_OE[95:64]	0xf000300c: LA_OE[31:0]
CSR_LA_OE_ADDR + 8	reg_la1_oenb	0xf0003018: LA_OE[63:32]	0xf0003010: ⚠️
CSR_LA_OE_ADDR + 12	reg_la0_oenb	0xf000301c: LA_OE[31:0]	0xf0003014: ⚠️

CSR_LA_IN_ADDR	-	0xf0003020	0xf0003010

CSR_LA_IN_ADDR + 0	reg_la3_data_in	0xf0003020: LA_IN[127:96]	0xf0003010: LA_IN[63:32]
CSR_LA_IN_ADDR + 4	reg_la2_data_in	0xf0003024: LA_IN[95:64]	0xf0003014: LA_IN[31:0]
CSR_LA_IN_ADDR + 8	reg_la1_data_in	0xf0003028: LA_IN[63:32]	0xf0003018: ⚠️
CSR_LA_IN_ADDR + 12	reg_la0_data_in	0xf000302c: LA_IN[31:0]	0xf000301c: ⚠️

CSR_LA_OUT_ADDR	-	0xf0003030	0xf0003018

CSR_LA_OUT_ADDR + 0	reg_la3_data	0xf0003030: LA_OUT[127:96]	0xf0003018: LA_OUT[63:32]
CSR_LA_OUT_ADDR + 4	reg_la2_data	0xf0003034: LA_OUT[95:64]	0xf000301c: LA_OUT[31:0]
CSR_LA_OUT_ADDR + 8	reg_la1_data	0xf0003038: LA_OUT[63:32]	0xf0003020: ⚠️
CSR_LA_OUT_ADDR + 12	reg_la0_data	0xf000303c: LA_OUT[31:0]	0xf0003024: ⚠️

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

0218-2024-10-05.md

0218-2024-10-05.md

5 Oct 2024

GFMPW-1 basic bring-up

Basic connection and proof of life

Reading GFMPW-1 Caravel registers

Checking the clock sources

Testing the Caravel CPU with basic blink firmware

Testing the mux

Other notes

Next steps

TODO

`reg_la1_data_in` weirdness

Files

0218-2024-10-05.md

Latest commit

History

0218-2024-10-05.md

File metadata and controls

5 Oct 2024

GFMPW-1 basic bring-up

Basic connection and proof of life

Reading GFMPW-1 Caravel registers

Checking the clock sources

Testing the Caravel CPU with basic blink firmware

Testing the mux

Other notes

Next steps

TODO

reg_la1_data_in weirdness

`reg_la1_data_in` weirdness