Tools
A list of built-in tools available
| Tool Name | Description | Use API KEY ? |
|---|---|---|
| arxiv_search | Search 3 most relevant research papers from arxiv.org | No |
| calculator | Solve arithmetic calculations and return a number | No |
| code_interpreter | Execute Python codes | No |
| write_file | Write strings to a file in the local hardisk | No |
| read_file | Read strings from a existing file in the local hardisk | No |
| google_search | Search relevant results from google.com | No |
| text_to_speech | Convert text into speech | No |
| text_to_image | Generate images based on input prompts | No |
| text_to_video | Generate videos based on input prompts | No |
| image_caption | Generate a caption for a given image | No |
| image_to_text | Generate a prompt for StableDiffusion that matches the input image | No |
| search_doc | search for the most relevant text chunk in a document | No |
| shell | Execute bash commands and returns the output | No |
| get_today_weather | Get the current weather information for a given location | Yes |
| get_future_weather | Get the weather information in the upcoming days for a given location | Yes |
| web_page | Get web content from a given url | No |
| wikipedia | Search relevant results from Wikipedia | No |
| wolfram_alpha | A WolframAlpha engine for solving algebraic equations | Yes |
Description: Search engine from arxiv.org. It returns several relevant paper titles, authors, and short summaries. The input should be a search query.
Location: arixv_search.ArxivSearch
Example: ArxivSearch()._run("Attention for transformer")
Published: 2022-09-30
Title: Adaptive Sparse and Monotonic Attention for Transformer-based Automatic Speech Recognition
Authors: Chendong Zhao, Jianzong Wang, Wen qi Wei, Xiaoyang Qu, Haoqian Wang, Jing Xiao
Summary: The Transformer architecture model, based on self-attention and multi-head
attention, has achieved remarkable success in offline end-to-end Automatic
Speech Recognition (ASR). However, self-attention and multi-head attention
cannot be easily applied for streaming or online ASR. For self-attention in
Transformer ASR, the softmax normalization function-based attention mechanism
makes it impossible to highlight important speech information. For multi-head
attention in Transformer ASR, it is not easy to model monotonic alignments in
different heads. To overcome these two limits, we integrate sparse attention
and monotonic attention into Transformer-based ASR. The sparse mechanism
introduces a learned sparsity scheme to enable each self-attention structure to
fit the corresponding head better. The monotonic attention deploys
regularization to prune redundant heads for the multi-head attention structure.
The experiments show that our method can effectively improve the attention
mechanism on widely used benchmarks of speech recognition.
Published: 2021-12-28
Title: Pale Transformer: A General Vision Transformer Backbone with Pale-Shaped Attention
Authors: Sitong Wu, Tianyi Wu, Haoru Tan, Guodong Guo
Summary: Recently, Transformers have shown promising performance in various vision
tasks. To reduce the quadratic computation complexity caused by the global
self-attention, various methods constrain the range of attention within a local
region to improve its efficiency. Consequently, their receptive fields in a
single attention layer are not large enough, resulting in insufficient context
modeling. To address this issue, we propose a Pale-Shaped self-Attention
(PS-Attention), which performs self-attention within a pale-shaped region.
Compared to the global self-a
Attention for transformer
Description: A calculator that can compute arithmetic expressions. Useful when you need to perform math calculations. The input should be a mathematical expression.
Location: calculator.Calculator
Example: Calculator()._run("sqrt(1+1)*pi")
4.442882938158366
Description: Execute Python codes. The input is a string containing Python codes.
Location: code_interpreter.PythonCodeInterpreter
Example: PythonCodeInterpreter()._run("import os\nprint(\"helloworld\")")
helloworld
Code executed successfully.
Description: Write strings to a file in hardisk. The inputs are file_path and a text string.
Location: file_operation.WriteFile
Example: WriteFile()._run("hello_world.text", "hello_world")
File written successfully to hello_world.text.
Description: Read a file from hardisk. The input is file_path.
Location: file_operation.ReadFile
Example: ReadFile()._run("hello_world.text")
hello_world
Description: Search results from Google. The input is a search query.
Location: google_search.GoogleSearch
Example: GoogleSearch()._run("What is mMTC?")
SearchResult(url=https://blog.antenova.com/what-is-mmtc-in-5g-how-does-it-work, title=What is mMTC in 5G? How does it work?, description=Nov 24, 2021 — Massive Machine-Type Communications (mMTC) is one of three core 5G service areas. It has been created specifically to enable a huge volume ...)
SearchResult(url=https://www.gigabyte.com/Solutions/mmtc, title=[5G-mMTC]Smart City Solution, description=mMTC (Massive Machine-Type Communications) is a new service category of 5G that can support extremely high connection density of online devices. Network ...)
SearchResult(url=https://en.wikipedia.org/wiki/MMTC_Ltd, title=MMTC Ltd, description=MMTC Ltd., Metals and Minerals Trading Corporation of India, is one of the two highest earners of foreign exchange for India and India's largest public ...)
SearchResult(url=https://inseego.com/resources/5g-glossary/what-is-mmtc/, title=What is mMTC: Massive Machine Type Communications?, description=mMTC enables ultra-low latency connections, improved network performance, and low energy usage for a wide variety of IoT use cases. mMTC is already being rolled ...)
Description: Convert input texts into a speech audio
Location: gradio.TTS
Example: TTS()._run("Please surprise me and speak in whatever voice you enjoy. Vielen Dank und Gesundheit!")
tmpl2_vozq_bklfm9yd.mov
the audio file saved into: /var/folders/xv/d1zbl5b50p58ttlb7bh655340000gn/T/tmpl2_vozq_bklfm9yd.wav
Description: Generate an image based on the input prompt
Location: gradio.TextToImage
Example: TextToImage()._run("""a man sitting on top of a rock holding a camera, enterprise workflow engine, photo blurring, nature landscape, canon dslr, images on the sales website, expectation of adventure, over the shoulder point of view, photo model, connecting life, website banner, travellers, bird on his shoulder, by David G. Sorensen, smartphone photography, visa pour l'image""")
the image file saved into: ~/Downloads/prompt/Gentopia-AI/Gentopia/gentopia/tools/c45fc4a8-af17-4d04-8d56-3e6ea6eea3d9/tmp71ynw3ir.jpg
Description: Generate a video based on the input prompt
Location: gradio.TextToVideo
Example: TextToVideo()._run(""a man sitting on top of a rock holding a camera, enterprise workflow engine, photo blurring, nature landscape, canon dslr, images on the sales website, expectation of adventure, over the shoulder point of view, photo model, connecting life, website banner, travellers, bird on his shoulder, by David G. Sorensen, smartphone photography, visa pour l'image""")
tmphoavpb0s8cdgdhfa.mp4
the video file saved into: /var/folders/xv/d1zbl5b50p58ttlb7bh655340000gn/T/tmphsmt6somlfgw_496.mp4
Description: Generate a caption for a given image
Location: gradio.ImageCaption
Example: ImageCaption()._run("images.jpeg")
a man sitting on a rock taking a picture with his camera
Description: Generate a prompt for StableDiffusion that matches the input image
Location: gradio.ImageToPrompt
Example: ImageToPrompt()._run("images.jpeg")
("a man sitting on top of a rock holding a camera, enterprise workflow engine, photo blurring, nature landscape, canon dslr, images on the sales website, expectation of adventure, over the shoulder point of view, photo model, connecting life, website banner, travellers, bird on his shoulder, by David G. Sorensen, smartphone photography, visa pour l'image", {'visible': True, '__type__': 'update'}, {'visible': True, '__type__': 'update'}, {'visible': True, '__type__': 'update'})
Description: Search for the most relevant text chunk in a document
Location: search_doc.SearchDoc
Example: TODO
Description: Execute bash commands and returns the output.
Location: bash.RunShell
Example: RunShell()._run("mkdir -p test/hello")
Description: Get the current weather information for a given location
Location: weather.GetTodayWeather
Example: GetTodayWeather()._run("San francisco")
Today's weather report for San Francisco is:
overall: Overcast,
name: San Francisco,
region: California,
country: United States of America,
localtime: 2023-07-20 10:10,
temperature: 15.0(C), 59.0(F),
percipitation: 0.0(mm), 0.0(inch),
pressure: 1017.0(milibar),
humidity: 87,
cloud: 100,
body temperature: 15.3(C), 59.5(F),
wind speed: 7.2(kph), 4.5(mph),
visibility: 16.0(km), 9.0(miles),
UV index: 3.0,
Description: Get the weather information in the upcoming days for a given location
Location: weather.GetFutureWeather
Example: GetFutureWeather()._run("San francisco",2)
The weather forecast for San francisco at 2 days later is:
over all weather: Sunny,
max temperature: 21.8(C), 71.2(F),
min temperature: 11.3(C), 52.3(F),
average temperature: 15.5(C), 59.8(F),
max wind speed: 16.9(kph), 10.5(mph),
total precipitation: 0.0(mm), 0.0(inch),
will rain today: 0,
chance of rain: 0,
total snow: 0.0(cm),
will snow today: 0,
chance of snow: 0,
average visibility: 10.0(km), 6.0(miles),
average humidity: 81.0,
UV index: 5.0,
sunrise time: 06:04 AM,
sunset time: 08:27 PM,
moonrise time: 09:38 AM,
moonset time: 10:56 PM,
Description: Get web content from a given url
Location: web_page.WebPage
Example: WebPage()._run("https://www.bbc.com/")
BBC - Homepage HomepageAccessibility linksSkip to contentAccessibility HelpBBC AccountNotificationsHomeNewsSportWeatheriPlayerSoundsBitesizeCBeebiesCBBCFoodHomeNewsSportReelWorklifeTravelFutureCultureTVWeatherSoundsMore menu Search BBC Search BBC HomeNewsSportWeatheriPlayerSoundsBitesizeCBeebiesCBBCFoodHomeNewsSportReelWorklifeTravelFutureCultureTVWeatherSoundsClose menu BBC Homepage Outrage after women paraded naked by mob in India PM Narendra Modi said the assault in Manipur state had "shamed" the country. India Outrage after women paraded naked by mob in India 'We have no more room,' NYC mayor tells migrants US 'We have no more room,' NYC mayor tells migrants Jury awards girl $800,000 for chicken McNugget burn US Jury awards girl $800,000 for chicken McNugget burn Where Oppenheimer escaped his legacy Travel Where Oppenheimer escaped his legacy Can Barbie's Ryan Gosling win an Oscar? Culture Can Barbie's Ryan Gosling win an Oscar? News N Korea 'not returning US calls' on missing soldier The US has reached out to the North, but there is still no word on the whereabouts of Travis King. Asia N Korea 'not returning US calls' on missing soldier Wheat prices soar after Russia threatens ships Russia announces it will treat ships heading for Ukrainian ports as potential military targets. Europe Wheat prices soar after Russia threatens ships Iraq expels Swedish envoy as Quran row escalates The ambassador is told to leave amid protests that began over the burning of a Quran in Stockholm. Middle East Iraq expels Swedish envoy as Quran row escalates Sport Ashes: Root bowled for 84 to leave England 351-4 Follow live text, Test Match Special commentary and in-play clips from the second day of the fourth Ashes Test between England and Australia at Emirates Old Trafford. Cricket Ashes: Root bowled for 84 to leave England 351-4 The Open round one: Rahm level, McIlroy one over, Fleetwood leads Follow Radio 5 Live coverage and text updates from the first round of the Open Championship at Royal Liverpool. Golf The Open round one: Rahm level, McIlroy one over, Fleetwood leads 'Goosebumps' - New Zealand and Australia's night of emotion New Zealand's players had tears in their eyes after defeating Norway in the opening game of the 2023 Women's World Cup - while Australia's own winning feeling was one of relief. Football 'Goosebumps' - New Zealand and Australia's night of emotion Artificial Intelligence Artist's work copied by AI more times than Picasso Greg Rutkowski is among the artists calling for more protection from artificial intelligence tools UK Artist's work copied by AI more times than Picasso The three job categories safe from AI AI will be able to replace some workers – but computers won't overtake all Worklife The three job categories safe from AI The dead Hollywood stars still working Artificial intelligence is giving some celebrities a new kind of immortality Future The dead Hollywood stars still working Singapore Weather Edit Thu 26°C Fri 31°C 25°C Sat 32°C 24°C Sun 31°C 24°C Visit Reel The most amazing videos from the BBC The age of the Universe might be drastically wrong The dark history of Denmark's room of 10,000 brains The unexpected link between a sexy doll and an atomic bomb Asia News Xi meets Kissinger as US seeks to defrost China ties The 100-year-old diplomat's surprise trip to Beijing follows a flurry of visits by US officials. China Xi meets Kissinger as US seeks to defrost China ties New Zealand mourns shooting deaths before World Cup Police are investigating how the 24-year-old suspect evaded the nation's strict gun control laws. Asia New Zealand mourns shooting deaths before World Cup India bans rice shipments to curb price rises The country has struggling with a surge in food prices after bad weather damaged crops. Business India bans rice shipments to curb price rises Women's World Cup: Co-hosts New Zealand stun Norway in opener - reaction Reaction as New Zealand claim superb win over Norway in the...
Description: Search relevant results from Wikipedia
Location: wikipedia.Wikipedia
Example: ``
Description: A WolframAlpha engine for solving algebraic equations
Location: wolfram_alpha.WolframAlpha
Example: WolframAlpha()._run("x + y = 6; x*y = 9")
x = 3, y = 3