Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: shrink 17-artwork-multimodal import size #42

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

refactor: shrink 17-artwork-multimodal import size #42

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions src/17-infinite-discovery/03-artworks-mutli2vec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ const client = weaviate.client({
async function main() {
await deleteIfExists(CLASS_NAME)
await prepareArtworkCollection()
const artworks = await getArtworks()
const artworks = await getArtworks("2024-11-12_artworks.json")
await insertArtworks(artworks)
}

Expand All @@ -30,13 +30,16 @@ async function prepareArtworkCollection() {
vectorizer: "multi2vec-clip",
moduleConfig: {
"multi2vec-clip": {
dimensions: 512,
textFields: [
"title",
"medium",
"materials",
"categories",
"tags",
"additionalInformation",
"date",
"artistName",
],
imageFields: ["image"],
},
Expand Down Expand Up @@ -114,6 +117,10 @@ async function prepareArtworkCollection() {
name: "artistGender",
dataType: ["text"],
},
{
name: "isCurated",
dataType: ["boolean"],
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@egdbear - decided to insert them into the same collection with a flag.

},
],
}

Expand All @@ -135,6 +142,8 @@ async function insertArtworks(
artworks: GravityArtwork[],
batchSize: number = BATCH_SIZE
) {
const start = Date.now()
console.log("Started inserting artworks at: ", start)
console.log(`Inserting artwork: ${artworks.length}`)

const batches = chunk(artworks, batchSize)
Expand Down Expand Up @@ -168,6 +177,7 @@ async function insertArtworks(
artistBirthday: artwork.artist_birthday,
artistGender: artwork.artist_gender,
image: imageData[artwork.id],
isCurated: false,
},
id: generateUuid5(artwork.id),
}
Expand All @@ -177,7 +187,7 @@ async function insertArtworks(
await batcher.do()
}
process.stdout.write("\n")
console.log("Done.")
console.log("Finished inserting artworks at: ", Date.now() - start)
}

main().catch(console.error)
8 changes: 4 additions & 4 deletions src/17-infinite-discovery/helpers.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ import { fromPairs, defaults } from "lodash"
*
* https://drive.google.com/drive/u/1/folders/1Lh7msUc0R_JlpNEzApZ4YbqB8x5tbpws
*/
export async function getArtworks() {
const filePath = path.join(__dirname, "./data/artworks.json")
export async function getArtworks(filename: string) {
const filePath = path.join(__dirname, `./data/${filename}`)
Comment on lines +14 to +15
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: Maybe a filename could be optional here? Also this method is being used here.

const data = await fs.promises.readFile(filePath, "utf-8")
const artworks: GravityArtwork[] = JSON.parse(data)
return artworks
Expand Down Expand Up @@ -60,8 +60,8 @@ export function resizeImage(
height: number
}
) {
const { width, height } = defaults(options, { width: 500, height: 500 })
return `https://d7hftxdivxxvm.cloudfront.net/?src=${src}&resize_to=fit&width=${width}&height=${height}&grow=false`
const { width, height } = defaults(options, { width: 224, height: 224 })
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the default resolution configured in the CLIP docs, so I have have copied that here. It is what has made the biggest difference.

return `https://d7hftxdivxxvm.cloudfront.net/?src=${src}&resize_to=fit&width=${width}&height=${height}&grow=false&quality=75`
}

/**
Expand Down