quantitative: ftw + Coraza with libInjection reports incorrect numbers

[RedXanadu]

For quantitative testing mode, the numbers reported for libInjection rules do not match with the numbers produced by running the reference shell script against Apache + ModSecurity v2.

Testing 10k corpus

Reference script with Apache + ModSecurity v2:

Number	Payload
1187	“Both business and consumer confidence are extremely low at the moment as a result of the impact of the lockdowns.
Rules triggered
  [Tue Nov 05 12:16:35.051320 2024]
      ModSecurity: Warning. detected SQLi using libinjection with fingerprint 'skn&n'
      [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"]
      [id "942100"]
      [msg "SQL Injection Attack Detected via libinjection"]
      [data "Matched Data: skn&n found within ARGS:payload: \\x22Both business and consumer confidence are extremely low at the moment as a result of the impact of the lockdowns."]
      [severity "CRITICAL"]
      [ver "OWASP_CRS/4.9.0-dev"]
      ["paranoia-level/1"]
      [unique_id "ZyoMozJg4LsbCW9Wi7KhRwAAAAg"]

Number	Payload
2336	Four Things That Love Is “Not”
Rules triggered
  [Tue Nov 05 12:16:57.102054 2024]
      ModSecurity: Warning. detected SQLi using libinjection with fingerprint 'sos'
      [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"]
      [id "942100"]
      [msg "SQL Injection Attack Detected via libinjection"]
      [data "Matched Data: sos found within ARGS:payload: Four Things That Love Is \\x22Not\\x22"]
      [severity "CRITICAL"]
      [ver "OWASP_CRS/4.9.0-dev"]
      ["paranoia-level/1"]
      [unique_id "ZyoMudyN9Vue4kcwsPCgcgAAAI8"]

Number	Payload
5674	People having immunotherapy or other continuing antibody treatments for cancer.
Rules triggered
  [Tue Nov 05 12:17:54.505590 2024]
      ModSecurity: Warning. detected SQLi using libinjection with fingerprint 'nBn&n'
      [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"]
      [id "942100"]
      [msg "SQL Injection Attack Detected via libinjection"]
      [data "Matched Data: nBn&n found within ARGS:payload: People having immunotherapy or other continuing antibody treatments for cancer."]
      [severity "CRITICAL"]
      [ver "OWASP_CRS/4.9.0-dev"]
      ["paranoia-level/1"]
      [unique_id "ZyoM8iRgA_tBdp_ot4quWQAAAMU"]

Number	Payload
8236	The statement posted to their website claims that “both civil and religious authorities have been notified and are investigating the allegations.”
Rules triggered
  [Tue Nov 05 12:18:38.445437 2024]
      ModSecurity: Warning. detected SQLi using libinjection with fingerprint 'skn&n'
      [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"]
      [id "942100"]
      [msg "SQL Injection Attack Detected via libinjection"]
      [data "Matched Data: skn&n found within ARGS:payload: The statement posted to their website claims that \\x22both civil and religious authorities have been notified and are investigating the allegations.\\x22"]
      [severity "CRITICAL"]
      [ver "OWASP_CRS/4.9.0-dev"]
      ["paranoia-level/1"]
      [unique_id "ZyoNHiRgA_tBdp_ot4qxkQAAAMQ"]

---

ftw + Coraza: FP 1 (payload 1187): NO MATCHES

$ ./ftw quantitative --crs-path ~/.git/coreruleset --corpus-lang=eng --corpus-source=news --corpus-year=2020 --corpus-size=10K --paranoia-level=2 --trace --payload '“Both business and consumer confidence are extremely low at the moment as a result of the impact of the lockdowns.' #1
5:56PM TRC No overrides file specified, skipping.
5:56PM TRC ftw/output: creating output normal

5:56PM INF ⏳Running quantitative tests with 1 goroutines
5:56PM TRC Rule: 0
5:56PM TRC Payload: “Both business and consumer confidence are extremely low at the moment as a result of the impact of the lockdowns.
5:56PM TRC Directory: /home/xanadu/.git/coreruleset
5:56PM TRC Paranoia level: 2
5:56PM DBG Using paranoia level: 2
5:56PM TRC --payload is used, ignoring corpus related parameters. Payload received: "“Both business and consumer confidence are extremely low at the moment as a result of the impact of the lockdowns."
5:56PM TRC Rules: map[]
5:56PM DBG Printing Stats summary
No false positives detected with the passed corpus

ftw + Coraza: FP 2 (payload 2336): NO MATCHES

$ ./ftw quantitative --crs-path ~/.git/coreruleset --corpus-lang=eng --corpus-source=news --corpus-year=2020 --corpus-size=10K --paranoia-level=2 --trace --payload 'Four Things That Love Is “Not”' #2
5:57PM TRC No overrides file specified, skipping.
5:57PM TRC ftw/output: creating output normal

5:57PM INF ⏳Running quantitative tests with 1 goroutines
5:57PM TRC Rule: 0
5:57PM TRC Payload: Four Things That Love Is “Not”
5:57PM TRC Directory: /home/xanadu/.git/coreruleset
5:57PM TRC Paranoia level: 2
5:57PM DBG Using paranoia level: 2
5:57PM TRC --payload is used, ignoring corpus related parameters. Payload received: "Four Things That Love Is “Not”"
5:57PM TRC Rules: map[]
5:57PM DBG Printing Stats summary
No false positives detected with the passed corpus

ftw + Coraza: FP 3 (payload 5674): SUCCESSFUL MATCH

$ ./ftw quantitative --crs-path ~/.git/coreruleset --corpus-lang=eng --corpus-source=news --corpus-year=2020 --corpus-size=10K --paranoia-level=2 --trace --payload 'People having immunotherapy or other continuing antibody treatments for cancer.' #3
5:57PM TRC No overrides file specified, skipping.
5:57PM TRC ftw/output: creating output normal

5:57PM INF ⏳Running quantitative tests with 1 goroutines
5:57PM TRC Rule: 0
5:57PM TRC Payload: People having immunotherapy or other continuing antibody treatments for cancer.
5:57PM TRC Directory: /home/xanadu/.git/coreruleset
5:57PM TRC Paranoia level: 2
5:57PM DBG Using paranoia level: 2
5:57PM TRC --payload is used, ignoring corpus related parameters. Payload received: "People having immunotherapy or other continuing antibody treatments for cancer."
5:57PM TRC Rules: map[942100: chain#0: People having immunotherapy or other continuing antibody treatments for cancer.]
5:57PM TRC False positive with string: &{%!s(int=0) People having immunotherapy or other continuing antibody treatments for cancer.}
5:57PM TRC => rules matched: map[942100: chain#0: People having immunotherapy or other continuing antibody treatments for cancer.]
5:57PM DBG **> rule 942100 with payload 0 =>  chain#0: People having immunotherapy or other continuing antibody treatments for cancer.
5:57PM DBG Printing Stats summary
Run 1 payloads in 115.139679ms
Total False positive ratio: 1/1 = 1.0000
False positives per rule id:
  942100: 1 false positives

ftw + Coraza: FP 4 (payload 8236): NO MATCHES

$ ./ftw quantitative --crs-path ~/.git/coreruleset --corpus-lang=eng --corpus-source=news --corpus-year=2020 --corpus-size=10K --paranoia-level=2 --trace --payload 'The statement posted to their website claims that “both civil and religious authorities have been notified and are investigating the allegations.”' #4
5:58PM TRC No overrides file specified, skipping.
5:58PM TRC ftw/output: creating output normal

5:58PM INF ⏳Running quantitative tests with 1 goroutines
5:58PM TRC Rule: 0
5:58PM TRC Payload: The statement posted to their website claims that “both civil and religious authorities have been notified and are investigating the allegations.”
5:58PM TRC Directory: /home/xanadu/.git/coreruleset
5:58PM TRC Paranoia level: 2
5:58PM DBG Using paranoia level: 2
5:58PM TRC --payload is used, ignoring corpus related parameters. Payload received: "The statement posted to their website claims that “both civil and religious authorities have been notified and are investigating the allegations.”"
5:58PM TRC Rules: map[]
5:58PM DBG Printing Stats summary
No false positives detected with the passed corpus

---

Testing 100k corpus

ftw + Coraza:

16 MATCHES

$ ./ftw quantitative --crs-path ~/.git/coreruleset --corpus-lang=eng --corpus-source=news --corpus-year=2020 --corpus-size=100K --rule=942100
6:16PM INF ⏳Running quantitative tests with 10 goroutines
Run 100000 payloads in 55.84883211s
Total False positive ratio: 16/100000 = 0.0002
False positives per rule id:
  942100: 16 false positives

Reference script with Apache + ModSecurity v2:

31 matches

3737	Always try to explain tangible benefits for better conversions.
4493	“And he is now the governor of Georgia.
4676	"And I read somewhere that you used to shoot marbles?"
7419	“As dioceses and schools instantiate their policies in different formats, it is up to each individual diocese to work with schools, parishes, and ministries to put the principles into practice.”
10921	“Between 2013-14 and 2020-21, central funding for fire has been reduced by 30 per cent in cash terms alone.
10933	"Between March and June … sellers took their homes off the market.
11530	"Both experts and laypersons testified that the burdens of this increased travel would fall disproportionately on poor women, who are least able to absorb them."
15147	Cele said the measures were not designed to “limit” the freedoms of ordinary people but implemented to prevent the spread of the virus.
15976	Clouds begin to return Sunday, and for now we will maintain a dry forecast with a high in the low 60s, but rain should return to the state Sunday night.
16768	Cosplay or “costume play” is dressing up as your favorite television, movie or comic book character.
18385	Director RaMell Ross sadness about the generalized inability to see communities like this one from the inside,” and it is evident from this movie — which features a section titled “How do we not frame someone?”
18476	“Diversity and change is something Blankenship has brought to the Owasso program,” Calip said.
34947	In 2018, Ohio State coach Urban Meyer went with Dwayne Haskins as the Buckeyes’ starting quarterback and Burrow, a former Ohio Mr. Football winner, left for a new opportunity.
35748	Indeed, when asked why it collects such data, a Facebook spokesperson responded: “As we set out in our Data Policy, we use this data to deliver our service and personalize features and content for people.
35875	In each of the four grade categories, the first two teams receive a "+" designation and the last two a "-" designation.
40046	It is a limit transformed into a time to deepen his and his family’s faith.
41805	“It’s having a top player playing with quality and playing with dedication.
45816	Labor councillor Linda Scott said she did not want to see ratepayers' money go to waste paying contractors to run a closed pool, arguing that $1 million could cover several upgrades to pocket parks, or a new green space.
45965	Lastly, on top of having another useful "set it and forget it" gadget, Alibi's previously mentioned Mx4 Storm is an incredibly strong SMG, offering a high rate-of-fire and low recoil.
46941	‘Limit’ to go on screen in Columbus Intl.
48202	Many use laptops where the screen is positioned too low, said Jaremey.
57725	PITTSBURGH (KDKA) – Ahead of a planned protest over Port Authority barring Black Lives Matter masks, Port Authority’s CEO says the uniform policy doesn’t target any “specific message, group or ideology.”
60662	Rosenberg said Facebook is “not resourcing and financing fast fact-checking and are not consulting civil rights groups about what hate speech is.”
61873	Seven years after leaving Liverpool on loan for Sheffield United, where he impressed before exiting permanently for Huddersfield, he has since become a core part of Wolves’ return to and continued success in the Premier League.
68505	"That's why nobody knows who you are, including me" — Trump and Birx team up to attack the Yahoo reporter who asked earlier about testing pic.
74871	Their Chair, Dr Russell Rook, commented, "While the lockdown is starting to ease, the need isn't.
76972	“The Night of the Physicists” is described as “the story of Nazi Germany’s hunt for a nuclear bomb” and “a tale of the genius and guilt of lauded, respected scientists.”
91225	Viewers can now buy tuques and T-shirts and doormats inscribed with Alexis’s trademark, “!”
92369	"We can't accommodate you unless you pay us $25,000," is ludicrous and probably criminal.
95855	“When James and I founded Flipdish in 2015, we originally set out to build a takeaway marketplace but quickly realised that the restaurant industry was being negatively affected by food delivery marketplaces.
96923	While the app is now being demoed at CES, LucidPix is currently in beta with more than 250,000 beta testers.

[RedXanadu]

It has been suggested that Coraza is compiling a different version of libInjection and may even be working using different data sets to mainstream ModSecurity. This may explain why the Coraza version is hiding lots of false positives. Therefore, the new Coraza-based quantitative testing numbers will be misleading and will mask (user) problems with libInjection-based rules.