Feature Request: USB Pi Gadgets iSCSI Storage

[lilylarceny]

I believe every console with a USB enabled storage can be supported by retronas with a USB Pi gadget, and iSCSI Storage on the server. a $15 Pi Zero 2 can connect to the server and act as a USB drive connected to any usb port.

Though it may require a Pi4 for each gadget if the Zero is not powerful enough to process large data files fast enough, it will require more testing. I would love to see this automated, for easy implementation in all USB enabled consoles. Wii U, Xbox One, Anything that can use a USB Hard Drive could, in theory, use this. a $15 Pi Zero, or even a $45 Pi4 would still be better than having to connect an individual hard drive to every console. The console doesn't even need to be modded or running custom firmware if it already has USB storage for games.

Proof of concept:

I have this setup working on my home server, with a raspberry pi Zero 2W connected to my PlayStation in my TerraOnion MODE. it needs further improvement and some help from folks to make it optimized. Currently, the PS1 games play from the sever.

Steps taken to get this working:

I followed the steps the in this video to set up the server
https://youtu.be/PLbdvkDGTSI?t=305

This video has you create an image file on the server, and that image file is formatted as a drive and used to access by the gadget.

I followed these steps for the gadget:
https://matt.olan.me/making-a-piscsi-usb-drive-part-1/
part 2:
https://matt.olan.me/making-a-piscsi-usb-drive-part-2/

Problems:

Streaming Full Motion Video and CD Audio, Voice Acting, etc all skips and causes problems if the game relies on it. For instance, Dino Crisis is unplayable because the entire game is voice acted and the voices skip when streaming. Gran Turismo however plays just fine, and any skipping in the FMV can be bypassed and you can load right into the game. lots of games play fine. some have errors, I have not tested all of them.

On the server, for some strange reason, I cannot change the permission or ownership of the image file, not with sudo, not as root. Logging in as root is the only way I can add games to the image. It is read only for all users except root. I can't even add images with the "sudo" command.

Troubleshooting:

I tested the Pi Zero2 USB gadget with the games stored on the device's SD card instead of over the network and the FMVs did not skip. I do not believe that the USB port is any limitation of the Pi Zero 2.

The server is connected via Gigabit Ethernet to the local router, the network throughput should exceed the speed of the PlayStation CD rom drive read speed.

The RetroNas server seems to crawl when accessing of the image file the PlayStation games are stored in. the Image File is 2TB in size, and formatted in FAT32. I am running RetroNas on a 3.2Ghz Dual Core Intel PC 4GB of ram, with Debian Linux. Not a Raspberry Pi. It did not have any problem serving SMB to PS2 or other aspects of RetroNas, but might be too slow to handle the 2TB image file.

Additionally, the Pi Zero 2 may just not be able to process the data fast enough, and if I try this with a Pi4b instead it might work fine. I don't have a Pi4b to test with, though I keep watching the stock updates in case on comes in stock.

--

I think this is everything. If you have further questions, or you would like me to try something please feel free. I would love to see this implemented on a larger scale.

[sairuk]

thanks for the detailed post, the first steps in this direction for us is providing generalised ISCSI support after that i absolutely agree this is a great opportunity. Unfortunately on Debian ISCSI management is not part of Cockpit due to missing udisk modules something to consider.

i see three potential modes of operation the RetroNAS may want to handle

• As an ISCSI initiator to an existing NAS
• As an ISCSI target for smaller (e.g. Zero devices)
• Build and apply an initiator image for user with a RetroNAS target (e.g. Zero devices)

All this before we get to the gadget stage, the interfaces would be challenging and will require some thought but perhaps we just use a bash scripts/select.

On the gadget side of things again we have two potential modes of operation

• As a g_mass_storage device itself
• Assisting to setup up a compatible device with an image for g_mass_storage for use with a RetroNAS target or writing an image and injecting these settings

We may also want to look into support for other gadgets like g_ether this again requires some kind of ease of use interfacing in the menus/scripts

I have purchased a RockPI4 (USB3.0 OTG) specifically for this purpose and have both a Zero and RPI4B (USB2.0 OTG) to test this with.

The RockPI4 SE support is extremely flaky, their main Debian images do not ship with ISCSI support so i need to build a kernel myself and Armbian (which ships with ISCSI support) is missing the appropriate device tree info to operate in OTG mode. This board is a work in progress.

I don't have a MODE to replicate the skipping issues but can potentially simulate it with an emu over the network i suppose, particularly since the SD card was performant.

[sairuk]

i set this up and ran it on a zero (otg usb wireless on port 1) and a zerow, both worked ok on PC

Not enough power delivery from Wii nor PS3 ports, the zerow would just complain and boot loop

Need to have a read over https://www.cnx-software.com/2021/12/09/raspberry-pi-zero-2-w-power-consumption/

[lilylarceny]

Any luck using the second USB port for power simultaneously?

[sairuk]

Yes and no, from what i understand doing this may disable the usb otg client mode.

PS3 with both usb ports: device would boot completely and i could access it over the wifi, was not detected by PS3 as a gadget
PS3: with external power: booted fully, no devices including my normal external storage were detected until the power was removed from the pi
Wii with external power or both usb ports: Wii would shut itself off

I've ordered some powered usb hubs to test with

[lilylarceny]

correct me if I am wrong, but from what I have read, the power USB port just bridges pins to the OTG port, so in my head I imagine you could disconnect the power pins on your data USB cable and connect a power-only USB cable to the power port to a power brick with enough amps to power the ZeroW?

[sairuk]

i am assuming the same at this point in time but also need to test a powered hub.

[lilylarceny]

According to the TerraOnion Product page, the MODE only has USB 2.0
https://terraonion.com/en/producto/terraonion-mode/

Wii, Wii U, PS3, 360, etc all using USB 2.0, maybe I don't fully understand it, should the 2.0 speed be the hold up, or would a USB 3.0 connection to the console even make a difference?

Rather than a MODE, If you have a Nintendo Wii, why not try to run games from a Pi gadget on a Softmodded Wii. That would be something, eh?

[sairuk]

i don't imagine the USB 2.0 limitation is an issue for your use case as proven by your SD card test, nor would it be for something like Wii where it only supports 2.0 and the community already load off usb storage.

Using the g_mass_storage gadget on a USB2.0 only solution (Zero/RPI4) will provide challenges where the console will only accept 3.0 mass storage devices for input e.g. PS4

PS4 references

• https://matt.olan.me/making-a-piscsi-usb-drive-part-1/
• https://matt.olan.me/making-a-piscsi-usb-drive-part-2/

What we also arguably need to do is establish a list of games with streaming audio/stuttering video etc per system to test with e.g as you mentioned Dino Crisis on PS1 is a good example, Dan has some other titles listed in one of the gamecube issues iirc but we should probably collect them into a centralised listing.

[lilylarceny]

There was not a single FMV on PlayStation that played without stuttering streaming over iSCSI on my setup, but many games it did not matter to the game play at all, because the FMV was just the intro or a cut scene that could be skipped. Even the GTA2 FMV intro skipped. I suspect anything that is normally streamed from the disc is skipping, and anything that is loaded into memory runs without problem. I would make a guess that the Dino Crisis voice acting is streamed from the disc.

[sairuk]

We've added basic iSCSI (Targets) support and Gadget support as individual options the user may install now.

if an iSCSI block device is available the gadget block device selector will see it

[sairuk]

we're not supporting iSCSI target mode yet, the basic implementation will let you take a Retronas install and turn it into an OTG device sans vendor specific configuration. So the retronas becomes the OTG device and iSCSI initiator. It needs some more testing, but the basic idea is there.

We are not at a point of setting up an external device, nor configuring/exposing LUN targets for an external device to connect too.

[sairuk]

pi1541 was a first run at managing external devices (and then probably dexdrive writing memcards) we have a lot of stuff to do in that space but there is also alot of fundamentals missing from the current general approach to get us there in a sane way

[lilylarceny]

I see, this is why I couldn't figure out what was not working. Until RetroNas can function as the target for my pi iscsi g mass storage device, i will continue to struggle with my own implementation.

[sairuk]

i have also fixed a bug (i thought i already had) with auto start of the iscsi initiator connection. and when setting up my rpi zerow from scratch all works as expected for now.

for reference my rpi zero currently connects to a qnap nas target for testing general iscsi performance/otg.

I have added iscsi targets to our ideas page so it is not lost, we need an image management solution before that which also should cover things like floppy disk images, vhd etc. its a larger scope idea but once we work out what it looks like we can come back around to iscsi target options

[sairuk]

for anyone interested in testing this

i have setup an rpi zerow with

• wireless access
• installed retronas on the rpizero (it can be slow to install)
• installed iscsi and initiated the connection to a remote target through these scripts
• installed otg support and setup the block storage pointing it to the iscsi block device
• edited config.txt manually to enable dwc2 (considered vendor config)

rebooted in otg mode, and my block storage is available at startup