How to feed DVS events/binary events to object detection models when inference on Loihi 2 cloud

[gwgknudayanga]

Dear Experts,

I have 50ms temporal length files in .npy format , so that each file contains dvs events as numpy array(first column is time stamp, second column is x coordinate, third is y coordinate and 4th column is polarity). When training SNNs in slayer, i can use slayer.io.event class to feed the data to SNN as tensors. However when inference with the the trained model on Loihi 2 cloud , how can i feed those event files to the trained model?
For example when running pilotnet snn on Loihi 2 i can see the authors used "io.dataloader.SpikeDataloader(dataset=full_set,interval=steps_per_sample)" which is from lava.proc.io API. Also when running yolo_kp on Loihi, authors used io.encoder.DeltaEncoder. So in my case where i have to feed dvs events what is the correponding "lava.proc.io" APIs that i should use and how should they should be used ?

Thank you so much.
Thanks and Rgds,
Udayanga

[weidel-p]

Hi,

there are probably many ways to send DVS data to Loihi. PilotNet and Yolo both work on RGB data, so they are not great examples for your case.

One straight forward way is to bin the event data into frames (histograms counting the number of events per pixel per time-step) and then treat them in the same (or similar) way as in the Pilotnet example.

You can also take a look at the lava-peripherals repository, where we created processes to stream Prophesee EVS/DVS camera data to Loihi. https://github.com/lava-nc/lava-peripherals/tree/main

[gwgkarunasena57]

Hey weidel-p,

Thank you for the reply. Actually in my case, i need to feed events rather than histogram frames. In some previous Intel workshops intel presenters mentioned about DVSGesture.ipynb notebook where they process DVS Gesture dataset (where the data are binary events). But I cannot find DVSGesture.ipynb notebook as an example in lava tutorials or lava-dl tutorials. Do you know where that example notebook is?
(I am a INRC member with loihi2 cloud access with user name 'udayanga').
Thank you so much.

Thanks and Rgds,
Udayanga