FileStream: Event-Driven File Upload Engine

FileStream is an open-source project, which is aim to gain more experience in golang, the purpose of this project is to build robust file upload systems that are efficient, resumable, and ready for real-time integrations.

Purpose of this project

This project was born from a desire to learn Go deeply while building something practical and useful. The aim is to:

• Enhance skills in Go by working on a real-world project.
• Build a reusable, efficient engine for file uploads.
• Attract contributions from developers passionate about open-source collaboration.

Roadmap

Here are some exciting features in the pipeline:

• Chunked uploads - Upload large files in smaller, manageable chunks.
• WebSocket Notifications - Real time updates for uploading process.
• Event driven - Publish events via NATS or other event system.
• Storage - In starting level it's good to support Minio for example.
• Customizable options - Customizable options is like remove metadata or using virus scan.
• Resumable upload - Resume interrupted uploads without starting over.

How to contribute

Here’s how you can get involved:

1. Report issues: If you find any bug or issue, please open a issue
2. Fork and code: Check out the open issues and submit a pull request.

Project setup

1. Clone the repository:

   git clone https://github.com/Nicolas-ggd/filestream
   

Usage

Basic File Upload Example:

import(
    "github.com/gin-gonic/gin"
    "github.com/Nicolas-ggd/filestream"
)

func Upload(c *gin.Context) {
   // Retrieve the uploaded file
   uploadFile, err := c.FormFile("file")
   if err != nil {
     c.JSON(http.StatusBadRequest, gin.H{"error": "failed to get file"})
     return
   }

   // Open the uploaded file
   file, err := uploadFile.Open()
   if err != nil {
     fmt.Println("Error opening uploaded file:", err)
     c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to open uploaded file"})
     return
   }
   
   defer file.Close()


   // Construct the FileRequest
   fileReq := fstream.RFileRequest{
       File:            file,
       UploadFile:      uploadFile,
       MaxRange:        rangeMax,
       FileSize:        fileSize,
       UploadDirectory: "uploads/",
       FileUniqueName:  true, // remember that if you set true here, you will receive unique name file
   }


   // Call StoreChunk to handle the uploaded chunk
   prFile, err := fstream.StoreChunk(&fileReq)
   if err != nil {
      c.JSON(http.StatusBadRequest, gin.H{"error": "failed to store chunk"})
      return
   }
   
   // You can write your own if statement to check whatever you want
   if isLast {
    // You can perform your own logic here, before return 200 status, 
    // it's better to remove chunks which is uploaded and doesn't use anymore
     err = fstream.RemoveUploadedFile("/dir", "filename.jpeg")
     if err != nil {
        c.JSON(http.StatusBadRequest, gin.H{"error": err})	 
     }
   }

   c.JSON(http.StatusOK, gin.H{"message": "file chunk processed"})
}

Validate File Extensions:

   import(
     "github.com/Nicolas-ggd/filestream"
   )

    // Declare extension slice, slice contains all file extension which is ok for you to allow in your system
    var allowExtensions = []string{".jpg", ".jpeg", ".png", ".gif", ".bmp", ".webp"}

    // This function is very simple, but save your time to compare extension and current file to known if it's valid for you
    allowed := fstream.IsAllowExtension(allowExtensions, "filename.png")
	
    if allowed {
     // your logic goes here...	
    }

Remove EXIF Metadata:

   import(
     "github.com/Nicolas-ggd/filestream"
   )

   // filePath is the upload direction + filename
   err := fstream.RemoveExifMetadata(filePath)
   if err != nil {
     log.Fatalln(err)	
   }

License

FileStream is open-source software licensed under the MIT License.