[Feature] own models

[KeparYTbcc]

To get a free version , let user chose its own model from the huggingface library and run it on gpu to make it work then use this model instead of openai api

[GreyDGL]

Yes, this makes sense. I would put it on the task list.
This probably requires some modular design of the current sessions. Feel free to propose good ideas with PRs:)

[absane]

Hackishly, you can use LocalAI (another project) to simulate OpenAI's API. Then, just override the Open AI host when importing via Python.

openai.api_base = "http://host.docker.internal:8080"

You have to set the model as well. In my case I setup ggml-gpt4all-j.

Works very well and only took me 20 minutes to figure out. Granted, it wasn't a clean way to do it... .I replaced everything that mentioned "gpt-4" and other models with my model

[LarsBingBong]

Yeah LocalAI is a really cool project. And I would feel safer about using this tool if my penetration test session weren't going to OpenAI. But, hit a local model and API.

[GreyDGL]

@LarsBingBong Unfortunately that is impossible for now. There is no existing model that can complete pentest as well as GPT-4. If you have concerns, you may also OpenAI API. OpenAI claims that they won't collect information from it, and the data will be deleted after 30 days.

[LarsBingBong]

Fair enough. I'll consider my options. Thank you very much for the response. At the same time though isn't running through Local-AI exactly what @absane did?

[absane]

Fair enough. I'll consider my options. Thank you very much for the response. At the same time though isn't running through Local-AI exactly what @absane did?

So, LocalAI just mimics the OpenAI API. Thus, as far as PentestGPT or any other tool is concerned that uses OpenAI API, the requests and format of the response don't change. The cool thing about LocalAI is that you can hook it into your own local model OR forward requests to the OpenAI API. You can pick and choose and even implement rules to choose what goes to OpenAI and what stays local.

I prefer this way because it provides more flexibility while someone, or even myself, work on a local model that can do just as well as GPT4 for our particular use-case. One thing I have in mind is taking GPT4All or whatever flavor-of-the-day model there is and fine-tuning it on internal documentation like our past reports, notes, bash history logs, etc. OpenAI is fantastic, but I don't like the idea of feeding it endless streams of sensitive information, particularly those of our clients.

[SATUNIX]

Hmmm could this be closed as per v0.9? As an enhancement we could probably add an installer menu for most recent models as per HF. The GPT4ALL API has a function imported which automatically downloads a model for use if it is not present. Despite the dependency, all we would need to change on the GPT4API is a menu in place of the models string value to be a linked variable of available models on the system(also including ones for automatic download) as per GPT4ALL API.

[Whoami451]

So, LocalAI just mimics the OpenAI API. Thus, as far as PentestGPT or any other tool is concerned that uses OpenAI API, the requests and format of the response don't change. The cool thing about LocalAI is that you can hook it into your own local model OR forward requests to the OpenAI API. You can pick and choose and even implement rules to choose what goes to OpenAI and what stays local.

can you show more steps on how to make pentestgpt workk with localai? It would be great to simple how to that shows exactly what to change in order to make it work. Pretty please