pre-release 0.9.4

What's Changed

• Introduced events table, improved crawling control, fixed an issue with auto-maintenance feature and some code clean up by @pzaino in #286
• Fixed some issue with the auto-maintenance feature for the new tables by @pzaino in #287
• Added the new bulk mode for addSource and the new addCategory commands as well as other improvements by @pzaino in #288
• Started working on supporting multiple DBMS and improved vulnerability scanning by @pzaino in #293
• Improved ruleset parsing robustness and bumped go to release 1.22.3 to resolve CVE-2024-24788 / GO-2024-2824 by @pzaino in #300
• Bump github.com/oschwald/maxminddb-golang from 1.12.0 to 1.13.0 by @dependabot in #298
• Bump golang.org/x/net from 0.25.0 to 0.26.0 by @dependabot in #303
• Updated links to the CROWler GPT by @pzaino in #305
• Completed centralization of plugins, added a simpel example and updated documentation by @pzaino in #309
• Improved multi-occurence data scraping by @pzaino in #311
• More plugin consolidation work completed by @pzaino in #312
• Changed git clone for selenium component from ssh to https, so no github login is required by @pzaino in #315
• Improved human-like behavior and added more SSL analysis by @pzaino in #317
• Improved documentation by @pzaino
• Added Plugin support for Detection rules by @pzaino
• Created list of contributors to the project by @pzaino in #344
• Bump github.com/aws/aws-sdk-go from 1.54.4 to 1.54.6 by @dependabot in #343
• Updated DB schema to latest release by @pzaino in #346

Full Changelog: v0.9.3...v0.9.4

pre-release 0.9.3

Before you upgrade to this pre-release

This pre-release introduced the DB schema version 1.3, so you'll need to rebuild your Database. If you are using the provided Docker Compose this happens automatically, otherwise please rebuild your DB using the new DB Schema setup script in pkg/database

What's Changed

• Improved docker image security build by @pzaino in #225
• Improved documentation by @pzaino in #226
• Fixed some minor issues with Service Scout and Improved documentation by @pzaino in #228
• Bump github.com/aws/aws-sdk-go from 1.51.28 to 1.51.29 by @dependabot in #227
• Improved rules control by @pzaino in #233
• Bump github.com/aws/aws-sdk-go from 1.51.29 to 1.51.30 by @dependabot in #231
• Bump github/codeql-action from 3.25.2 to 3.25.3 by @dependabot in #230
• Bump actions/dependency-review-action from 4.2.5 to 4.3.0 by @dependabot in #229
• Allowed to use IPs instead of just URLs by @pzaino in #234
• Bump actions/setup-go from 5.0.0 to 5.0.1 by @dependabot in #242
• Bump step-security/harden-runner from 2.7.0 to 2.7.1 by @dependabot in #239
• Bump github.com/aws/aws-sdk-go from 1.51.30 to 1.52.2 by @dependabot in #243
• Bump github.com/likexian/whois from 1.15.2 to 1.15.3 by @dependabot in #236
• Bump github.com/PuerkitoBio/goquery from 1.9.1 to 1.9.2 by @dependabot in #237
• Major Update by @pzaino in #245
• Removed old RBee integration tests by @pzaino in #246
• Updated gitignore by @pzaino in #247
• Introducing new DB schema release 1.3 and web pages performance analysis by @pzaino in #257
• Bump github.com/aws/aws-sdk-go from 1.52.2 to 1.52.6 by @dependabot in #256
• Bump github/codeql-action from 3.25.3 to 3.25.4 by @dependabot in #255
• Bump codacy/codacy-analysis-cli-action from 4.4.0 to 4.4.1 by @dependabot in #254
• Bump actions/checkout from 4.1.4 to 4.1.5 by @dependabot in #251
• Bump golang.org/x/crypto from 0.22.0 to 0.23.0 by @dependabot in #250
• Bump golang.org/x/net from 0.24.0 to 0.25.0 by @dependabot in #249
• Added a working config.yaml called config.default to make people life even easier by @pzaino in #258
• Improved detection, now it returns which micro signatures have matched and where by @pzaino in #261
• Bump github.com/aws/aws-sdk-go from 1.52.6 to 1.53.0 by @dependabot in #260
• Bump ossf/scorecard-action from 2.3.1 to 2.3.3 by @dependabot in #259
• Improved detection in cases where a WAF might be configured in a certain way that was preventing previous version to get the content by @pzaino in #262
• Fixed a problem with a URL for scorecards by @pzaino in #263
• Fixed some minor issues and merged scraping data into webObjects table by @pzaino in #266
• Bump github.com/aws/aws-sdk-go from 1.53.0 to 1.53.1 by @dependabot in #265
• Bump github/codeql-action from 3.25.4 to 3.25.5 by @dependabot in #264
• Bump actions/checkout from 4.1.5 to 4.1.6 by @dependabot in #270
• Bump github.com/aws/aws-sdk-go from 1.53.1 to 1.53.4 by @dependabot in #269
• Ported webObject API endpoint, improved BD performenace and added fuzzing tests for config and rulesets by @pzaino in #271
• Ported Sources correlation and URL status check in the API by @pzaino in #273
• Bump github.com/aws/aws-sdk-go from 1.53.4 to 1.53.5 by @dependabot in #272
• Improvements on the DB Schema and the links extraction by @pzaino in #274
• Introduced Status Report and fixed some minor issues for clusterized crawling by @pzaino in #275
• Added a check to avoid malicious metatags to disrupt data collection by @pzaino in #276
• Improved NetInfo collection and general code improvements by @pzaino in #280
• Bump github.com/aws/aws-sdk-go from 1.53.5 to 1.53.7 by @dependabot in #279
• Bump github/codeql-action from 3.25.5 to 3.25.6 by @dependabot in #278
• Improved Vulnerabilities detection data collection and default configuration by @pzaino in #283
• Bump step-security/harden-runner from 2.7.1 to 2.8.0 by @dependabot in #281
• Bump github.com/aws/aws-sdk-go from 1.53.7 to 1.53.8 by @dependabot in #282

Full Changelog: v0.9.2...v0.9.3

pre-release 0.9.2

What's Changed

• Improved crawling performance and documentation by @pzaino in #176
• Bump github.com/aws/aws-sdk-go from 1.51.9 to 1.51.10 by @dependabot in #175
• Improved DB automatic maintenance by @pzaino in #177
• Started to port URL Fuzzing logic and Crawling Rules support by @pzaino in #178
• Bump github.com/aws/aws-sdk-go from 1.51.10 to 1.51.11 by @dependabot in #179
• Bump golang.org/x/net from 0.22.0 to 0.24.0 by @dependabot in #185
• Bump github.com/aws/aws-sdk-go from 1.51.11 to 1.51.15 by @dependabot in #184
• Added support for detection rules, still being tested and extended by @pzaino in #192
• Bump github.com/aws/aws-sdk-go from 1.51.15 to 1.51.18 by @dependabot in #191
• Bump github.com/likexian/whois from 1.15.1 to 1.15.2 by @dependabot in #190
• Bump github/codeql-action from 3.24.9 to 3.24.10 by @dependabot in #186
• Bump github.com/antchfx/htmlquery from 1.3.0 to 1.3.1 by @dependabot in #189
• Added more configuration checks for database section by @pzaino in #194
• Bump github.com/aws/aws-sdk-go from 1.51.18 to 1.51.19 by @dependabot in #193
• Improved action rules selector algorithm and improved configuration validation by @pzaino in #196
• Bump github.com/aws/aws-sdk-go from 1.51.19 to 1.51.20 by @dependabot in #195
• Improved documentation by @pzaino in #197
• Improved Configuration validation and fixed a vulnerability GO-2024-2687 with x/net dependecy by @pzaino in #198
• Fixed a minor permission issue with the crowler DB user when performing automated DB maintenance operations by @pzaino in #200
• Bump github.com/aws/aws-sdk-go from 1.51.20 to 1.51.21 by @dependabot in #199
• Bump github.com/aws/aws-sdk-go from 1.51.21 to 1.51.22 by @dependabot in #203
• Bump github/codeql-action from 3.24.10 to 3.25.0 by @dependabot in #202
• Bump github.com/robertkrimen/otto from 0.3.0 to 0.4.0 by @dependabot in #201
• Improvements on the ruleset management code to increase performance when using large rulesets by @pzaino in #204
• Improved detection rules and documentation by @pzaino in #209
• Bump actions/upload-artifact from 4.3.1 to 4.3.2 by @dependabot in #208
• Bump github.com/aws/aws-sdk-go from 1.51.22 to 1.51.24 by @dependabot in #207
• Bump github/codeql-action from 3.25.0 to 3.25.1 by @dependabot in #206
• Completed detection rules support by @pzaino in #212
• Bump actions/checkout from 4.1.2 to 4.1.3 by @dependabot in #211
• Bump github.com/aws/aws-sdk-go from 1.51.24 to 1.51.25 by @dependabot in #210
• Improved detection rules support and added Ruleset Reference documentation by @pzaino in #213
• Improved documentation by @pzaino in #214
• Improved documentation and fixed some issue on some of the doc links by @pzaino in #215
• Improved response fields population by @pzaino in #220
• Bump github/codeql-action from 3.25.1 to 3.25.2 by @dependabot in #219
• Bump github.com/aws/aws-sdk-go from 1.51.25 to 1.51.27 by @dependabot in #218
• Bump actions/upload-artifact from 4.3.2 to 4.3.3 by @dependabot in #216
• Completed SSLInfo support port from the old Rust version to the new Go Lang and fixed some bugs discovered when the user tries to add a source without http or ftp protocols by @pzaino in #221
• Added check for certificate revoked and improved SSLInfo data collection by @pzaino in #224
• Bump actions/checkout from 4.1.3 to 4.1.4 by @dependabot in #223
• Bump github.com/aws/aws-sdk-go from 1.51.27 to 1.51.28 by @dependabot in #222

Full Changelog: v0.9.1...v0.9.2

v0.9.1

What's Changed

• Cumulative set of fixes by @pzaino in #166 and #174
• Bump actions/dependency-review-action from 4.2.4 to 4.2.5 by @dependabot in #168
• Fixed some issues with full DNS detection and improved documentation by @pzaino in #169
• fixed a minor issue in the dns discovery and improved documentation by @pzaino in #170
• Minor code improvements and documentation improvements by @pzaino in #171
• Improve permissions granting for crowler users on the DB by @pzaino in #172
• Bump github.com/aws/aws-sdk-go from 1.51.8 to 1.51.9 by @dependabot in #173
• Moved to Selenium 4.18.1

Full Changelog: v0.9...v0.9.1

pre-release 0.9

What's Changed

• Bump golang.org/x/net from 0.21.0 to 0.22.0 by @dependabot in #131
• More ruleset support code ported from the Rust version by @pzaino in #137
• Improved search API security by @pzaino in #138
• Improved ruleset schema and ported more ruleset support code from the Rust version by @pzaino in #142
• Bump github.com/aws/aws-sdk-go from 1.50.36 to 1.50.37 by @dependabot in #145
• Bump github/codeql-action from 3.24.6 to 3.24.7 by @dependabot in #144
• Bump actions/checkout from 4.1.1 to 4.1.2 by @dependabot in #143
• More ruleset support code ported from the Rust version and added support for JavaScript modules by @pzaino in #146
• Cumulative merge with latest ruleset support code update from Develop to main branch by @pzaino in #148
• Bump github.com/aws/aws-sdk-go from 1.50.37 to 1.50.38 by @dependabot in #147
• Improved Rulesets handling by @pzaino in #150
• Bump github.com/aws/aws-sdk-go from 1.50.38 to 1.51.0 by @dependabot in #149
• Fixed some issues with GitHub Issue Templates by @pzaino in #151
• Bump github.com/aws/aws-sdk-go from 1.51.0 to 1.51.2 by @dependabot in #154
• Bump github/codeql-action from 3.24.7 to 3.24.8 by @dependabot in #153
• Cumulative Merge of latest improvements from Develop branch to Main branch by @pzaino in #160
• Bump github.com/aws/aws-sdk-go from 1.51.2 to 1.51.5 by @dependabot in #158
• Bump actions/dependency-review-action from 4.1.3 to 4.2.3 by @dependabot in #157
• Bump github/codeql-action from 3.24.8 to 3.24.9 by @dependabot in #162
• Fixed some issues in the search engine by @pzaino in #163
• Bump github.com/aws/aws-sdk-go from 1.51.5 to 1.51.6 by @dependabot in #161

Full Changelog: v0.8...v0.9

pre-release 0.8

What's Changed

• Improved DB structure to guest HTML, started to convert default rules… by @pzaino in #128
• fixed some typos in a comment by @pzaino in #129
• Bump github.com/stretchr/testify from 1.8.4 to 1.9.0 by @dependabot in #127
• Bump github.com/aws/aws-sdk-go from 1.50.29 to 1.50.30 by @dependabot in #126

Full Changelog: v0.7...v0.8

pre-release 0.7

What's Changed

• Improved quality for meta tags extraction and fixed a bug in removeSource command by @pzaino in #125
• Bump github.com/PuerkitoBio/goquery from 1.9.0 to 1.9.1 by @dependabot in #123
• Bump github.com/aws/aws-sdk-go from 1.50.28 to 1.50.29 by @dependabot in #124

Full Changelog: v0.6...v0.7

pre-release 0.6

What's Changed

• Bump github.com/PuerkitoBio/goquery from 1.8.1 to 1.9.0 by @dependabot in #116
• Bump github.com/aws/aws-sdk-go from 1.50.25 to 1.50.28 by @dependabot in #120
• Minor code porting from Rust to go for the rules engine execution by @pzaino in #122
• Bump github/codeql-action from 3.24.5 to 3.24.6 by @dependabot in #121

Full Changelog: v0.5...v0.6

pre-release v0.5

What's Changed

• Bump github.com/aws/aws-sdk-go from 1.50.21 to 1.50.23 by @dependabot in #113
• Bump actions/dependency-review-action from 4.1.2 to 4.1.3 by @dependabot in #112
• Improved HTTP codes in the API, completed all network objects dork-based query support and some code refactoring for the API console @pzaino in #114

Full Changelog: v0.4...v0.5

pre-release v0.4

What's Changed

• Some fixes for the console feature by @pzaino in #110
• Bump github.com/aws/aws-sdk-go from 1.50.18 to 1.50.21 by @dependabot in #109
• Bump actions/dependency-review-action from 4.1.0 to 4.1.2 by @dependabot in #108

Full Changelog: v0.3...v0.4