Skip to content

Commit

Permalink
refactor: log
Browse files Browse the repository at this point in the history
  • Loading branch information
@CatchZeng
CatchZeng committed Mar 4, 2021
1 parent fc10c6f commit 918de8a
Show file tree
Hide file tree
Showing 4 changed files with 65 additions and 58 deletions.
86 changes: 44 additions & 42 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,17 +27,36 @@ pip install bing-images

### Fetch Image Urls

fetch_image_urls.py

```py
from bing_images import bing

urls = bing.fetch_image_urls("cat", limit=100, file_type='png', filters='+filterui:aspect-square+filterui:color2-bw')
urls = bing.fetch_image_urls("cat", limit=10, file_type='png', filters='+filterui:aspect-square+filterui:color2-bw')
print("{} images.".format(len(urls)))
counter = 1
for url in urls:
print("{}: {}".format(counter, url))
counter += 1
```

Run

```
❯ python fetch_image_urls.py
10 images.
1: http://pngimg.com/uploads/cat/cat_PNG50521.png
2: http://pngimg.com/uploads/cat/cat_PNG1616.png
3: https://pngimg.com/uploads/cat/cat_PNG50532.png
4: https://pngimg.com/uploads/cat/cat_PNG1621.png
5: https://pngimg.com/uploads/cat/cat_PNG1618.png
6: http://pngimg.com/uploads/cat/cat_PNG1624.png
7: http://www.pngmart.com/files/5/Black-Cat-PNG-Transparent.png
8: http://www.myiconfinder.com/uploads/iconsets/256-256-a96249f4c8a9753fd904f8be023dc25c-cat.png
9: https://pngimg.com/uploads/cat/cat_PNG1619.png
10: http://pngimg.com/uploads/cat/cat_PNG50521.png
```

> **filters**: You can visit <https://cn.bing.com/images/search?q=cat> to operate filters through the UI, and then observe the link changes.
### Download using multithreading
Expand All @@ -63,48 +82,31 @@ Run
```shell
❯ python download.py
Save path: /Users/catchzeng/Desktop/cat
Downloading image #1 from http://pngimg.com/uploads/cat/cat_PNG100.png
Downloading image #2 from http://www.freepngimg.com/download/cat/22193-3-adorable-cat.png
Downloading image #3 from http://pngimg.com/uploads/cat/cat_PNG106.png
Downloading image #5 from https://pngimg.com/uploads/cat/cat_PNG50433.png
Downloading image #6 from https://pngimg.com/uploads/cat/cat_PNG50434.png
Downloading image #4 from http://pngimg.com/uploads/cat/cat_PNG50509.png
Downloading image #8 from http://pngimg.com/uploads/cat/cat_PNG50480.png
Downloading image #9 from https://pngimg.com/uploads/cat/cat_PNG50447.png
Downloading image #10 from http://pngimg.com/uploads/cat/cat_PNG50529.png
Downloading image #7 from http://pngimg.com/uploads/cat/cat_PNG119.png
Downloading image #11 from https://pngimg.com/uploads/cat/cat_PNG50438.png
Image #4 Downloaded
Downloading image #12 from http://pngimg.com/uploads/cat/cat_PNG50498.png
Image #3 Downloaded
Downloading image #13 from https://pngimg.com/uploads/cat/cat_PNG50465.png
Image #12 Downloaded
Downloading image #14 from https://pngimg.com/uploads/cat/cat_PNG50417.png
Image #2 Downloaded
Downloading image #15 from https://pngimg.com/uploads/cat/cat_PNG50440.png
Image #6 Downloaded
Downloading image #16 from http://pngimg.com/uploads/cat/cat_PNG113.png
Image #8 Downloaded
Downloading image #17 from http://pngimg.com/uploads/cat/cat_PNG50442.png
Image #1 Downloaded
Downloading image #18 from https://purepng.com/public/uploads/large/purepng.com-catanimalscat-981524673949tj5ns.png
Image #7 Downloaded
Downloading image #19 from https://pngimg.com/uploads/cat/cat_PNG50435.png
Image #9 Downloaded
Downloading image #20 from https://cdn.pixabay.com/photo/2017/02/22/16/55/cat-2089916_960_720.png
Image #15 Downloaded
Image #20 Downloaded
Image #14 Downloaded
Image #17 Downloaded
Image #16 Downloaded
Image #18 Downloaded
Image #11 Downloaded
Image #13 Downloaded
Image #19 Downloaded
Image #5 Downloaded
Image #10 Downloaded
Downloading images
#1 http://pngimg.com/uploads/cat/cat_PNG50509.png Downloaded
#2 https://pngimg.com/uploads/cat/cat_PNG50498.png Downloaded
#3 http://www.freepngimg.com/download/cat/22193-3-adorable-cat.png Downloaded
#4 http://pngimg.com/uploads/cat/cat_PNG106.png Downloaded
#5 https://pngimg.com/uploads/cat/cat_PNG50465.png Downloaded
#6 https://pngimg.com/uploads/cat/cat_PNG50417.png Downloaded
#7 https://pngimg.com/uploads/cat/cat_PNG50480.png Downloaded
#8 http://pngimg.com/uploads/cat/cat_PNG119.png Downloaded
#9 https://pngimg.com/uploads/cat/cat_PNG50438.png Downloaded
#10 http://pngimg.com/uploads/cat/cat_PNG100.png Downloaded
#11 https://pngimg.com/uploads/cat/cat_PNG50447.png Downloaded
#12 https://pngimg.com/uploads/cat/cat_PNG50440.png Downloaded
#13 https://pngimg.com/uploads/cat/cat_PNG50433.png Downloaded
#14 https://www.pngarts.com/files/1/Baby-Cat-PNG-Free-Download.png Downloaded
#15 https://cdn.pixabay.com/photo/2017/02/22/16/55/cat-2089916_960_720.png Downloaded
#16 https://pngimg.com/uploads/cat/cat_PNG50434.png Downloaded
#17 http://pngimg.com/uploads/cat/cat_PNG50529.png Downloaded
#18 http://pngimg.com/uploads/cat/cat_PNG113.png Downloaded
#19 https://purepng.com/public/uploads/large/purepng.com-catanimalscat-981524673949tj5ns.png Downloaded
#20 https://pngimg.com/uploads/cat/cat_PNG50435.png Downloaded
Renaming images
Finished renaming
Done
Elapsed Time: 29.91s
Elapsed time: 20.76s
```

![](./images/cat.jpg)
Expand Down
25 changes: 15 additions & 10 deletions bing_images/bing.py
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
try:
from util import file_name, rename, make_image_dir, download_image
from util import get_file_name, rename, make_image_dir, download_image
except ImportError: # Python 3
from .util import file_name, rename, make_image_dir, download_image
from .util import get_file_name, rename, make_image_dir, download_image
from typing import Counter, List
from multiprocessing.pool import ThreadPool
from time import time as timer
Expand Down Expand Up @@ -89,6 +89,7 @@ def download_images(
urls = fetch_image_urls(query, limit, adult, file_type, filters)
entries = get_image_entries(urls, image_dir)

print("Downloading images")
ps = pool_size
if limit < pool_size:
ps = limit
Expand All @@ -98,11 +99,13 @@ def download_images(

print("Done")
elapsed = timer() - start
print("Elapsed Time: %.2fs" % elapsed)
print("Elapsed time: %.2fs" % elapsed)


def rename_images(dir, prefix):
files = os.listdir(dir)
index = 1
print("Renaming images")
for f in files:
if f.startswith("."):
print("Escape {}".format(f))
Expand All @@ -111,33 +114,35 @@ def rename_images(dir, prefix):
name = rename(f, index, prefix)
dst = os.path.join(dir, name)
os.rename(src, dst)
print("Rename {} to {}".format(src, dst))
index = index + 1
print("Finished renaming")


def download_image_entries(entries, pool_size):
counter = 0
counter = 1
results = ThreadPool(pool_size).imap_unordered(
download_image_with_thread, entries)
for (url, result, path) in results:
for (url, result) in results:
if result:
print("#{} {} Downloaded {}".format(counter, url, path))
print("#{} {} Downloaded".format(counter, url))
counter = counter + 1


def get_image_entries(urls, dir):
entries = []
i = 0
for url in urls:
name = file_name(url, i, "#tmp#")
name = get_file_name(url, i, "#tmp#")
path = os.path.join(dir, name)
entries.append((url, path))
i = i + 1
return entries


def download_image_with_thread(entry):
url, path = entry
print("Downloading image from {}".format(url))
result = download_image(url, path)
return (url, result, path)
return (url, result)


if __name__ == '__main__':
Expand Down
10 changes: 5 additions & 5 deletions bing_images/util.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,21 +15,21 @@ def download_image(url, path) -> bool:
shutil.copyfileobj(r.raw, f)
return True
else:
print("[!] download image: {}\n[!] Err :: {}".format(url, r.content))
print("[!] Download image: {}\n[!] Err :: {}".format(url, r.status_code))
return False
except Exception as e:
print("[!] download image: {}\n[!] Err :: {}".format(url, e))
print("[!] Download image: {}\n[!] Err :: {}".format(url, e))
return False

def file_name(url, index, prefix='image') -> str:
def get_file_name(url, index, prefix='image') -> str:
try:
path = urllib.parse.urlsplit(url).path
filename = posixpath.basename(path).split('?')[0]
type, _ = file_data(filename)
result = "{}_{}.{}".format(prefix, index, type)
return result
except Exception as e:
print("[!] Issue getting: {}\n[!] Err :: {}".format(url, e))
print("[!] Get file name: {}\n[!] Err :: {}".format(url, e))
return prefix

def rename(name, index, prefix='image') -> str:
Expand All @@ -38,7 +38,7 @@ def rename(name, index, prefix='image') -> str:
result = "{}_{}.{}".format(prefix, index, type)
return result
except Exception as e:
print("[!] Issue getting: {}\n[!] Err :: {}".format(name, e))
print("[!] Rename: {}\n[!] Err :: {}".format(name, e))
return prefix

def file_data(name):
Expand Down
2 changes: 1 addition & 1 deletion setup.cfg
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[metadata]
name = bing_images
version = 0.0.4
version = 0.0.5
author = CatchZeng
author_email = [email protected]
description = Python library to fetch image urls and download using multithreading from Bing.com.
Expand Down

0 comments on commit 918de8a

Please sign in to comment.