Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[firefox] - single test - multiple assertScreenshot - if one fails, the diff image show one of the not failed assertion #47

Open
andreabisello opened this issue May 27, 2016 · 0 comments
Open

[firefox] - single test - multiple assertScreenshot - if one fails, the diff image show one of the not failed assertion #47

andreabisello opened this issue May 27, 2016 · 0 comments

Comments

@andreabisello
Copy link

andreabisello commented May 27, 2016 •
edited
Loading

i have a complex single page web application.
i want to make a visual regression splitting the web page in different small screenshot (for example left menu, main menu, right menu, footer and navigation).

i make the baseline and screenshot are good.
i make a regression check and there is no difference.

now, if i modify something in the left menu, the test fails, but i obtain images of area that are not different on FIREFOX, but it works on CHROME

this is my code

    def test_home_mainMenu(self):
        self.driver.delete_all_cookies()
        self.driver.get('https://'+configuration.clientId+'.thron.com/#/login/'+configuration.token)
        time.sleep(15)
        self.assertScreenshot('.main-menu', 'main-menu')
        self.assertScreenshot('.apps-list-container', 'apps-list-container')
        self.assertScreenshot('.menu-right-container', 'menu-right-container')
        self.assertScreenshot('.notifications', 'notifications')
        self.assertScreenshot('.secondary-menu-contents', 'sx-menu-library')
        self.assertScreenshot('.secondary-menu-calendar', 'calendar-live')
        self.assertScreenshot('.main-primary', 'contents-area')

and i obtain all the correct screenshot

image

but now i make the test again making a change on sx-menu-library.png , closing the folder before assertScreenshot starts (this explain why there is the time.sleep(15))

so from
image to
image

test correcly fails,

but, as you can see, the screenshot created show a difference where there is no difference

image

and this is the log

C:\src\automatictestingtool\NeedleVisual>nosetests homepage.py

F

FAIL: test_home_mainMenu (homepage.THRONVisualRegression)

Traceback (most recent call last):
File "C:\src\automatictestingtool\NeedleVisual\homepage.py", line 15, in test_
home_mainMenu
self.assertScreenshot('.notifications', 'notifications')
File "c:\python27\lib\site-packages\needle\cases.py", line 131, in assertScree
nshot
pass
File "c:\python27\lib\contextlib.py", line 24, in exit
self.gen.next()
File "c:\python27\lib\site-packages\needle\cases.py", line 202, in compareScre
enshot
self.engine.assertSameFiles(output_file, baseline_file, threshold)
File "c:\python27\lib\site-packages\needle\engines\pil_engine.py", line 25, in
assertSameFiles
% (output_file, baseline_file, distance))
AssertionError: The new screenshot 'C:\src\automatictestingtool\NeedleVisual\scr
eenshots\notifications.png' did not match the baseline 'C:\src\automatictestingt
ool\NeedleVisual\screenshots\baseline\notifications.png' (by a distance of 198.6
1)
-------------------- >> begin captured logging << --------------------
requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1)
: thronqa-view.4me.it
requests.packages.urllib3.connectionpool: DEBUG: "POST /api/xsso/resources/ident
itymanager/login HTTP/1.1" 200 56
selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/
hub/session {"desiredCapabilities": {"platform": "ANY", "browserName": "firefox"
, "version": "", "marionette": false, "javascriptEnabled": true}}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/
hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/window/current/position {"y": 0
, "x": 0, "windowHandle": "current", "sessionId": "7ca39783-4f52-4288-8afe-44a99
c6b574c"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/
hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/window/current/size {"width": 1
024, "windowHandle": "current", "sessionId": "7ca39783-4f52-4288-8afe-44a99c6b57
4c", "height": 768}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/
hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/execute {"sessionId": "7ca39783
-4f52-4288-8afe-44a99c6b574c", "args": [], "script": "return {width: document.bo
dy.clientWidth, height: document.body.clientHeight};"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/
hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/window/current/size {"width": 1
037, "windowHandle": "current", "sessionId": "7ca39783-4f52-4288-8afe-44a99c6b57
4c", "height": 768}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: DELETE http://127.0.0.1:5883
5/hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/cookie {"sessionId": "7ca3978
3-4f52-4288-8afe-44a99c6b574c"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/
hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/url {"url": "https://thronqa.th
ron.com/#/login/b8171bd2-69c3-4f26-9e71-c05aa0726c47", "sessionId": "7ca39783-4f
52-4288-8afe-44a99c6b574c"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/
hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element {"using": "css selector
", "sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "value": ".main-menu"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h
ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element/{faf2a1af-be85-4b3d-96fb
-441d9621939f}/location {"sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "i
d": "{faf2a1af-be85-4b3d-96fb-441d9621939f}"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h
ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element/{faf2a1af-be85-4b3d-96fb
-441d9621939f}/size {"sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "id":
"{faf2a1af-be85-4b3d-96fb-441d9621939f}"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h
ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/screenshot {"sessionId": "7ca397
83-4f52-4288-8afe-44a99c6b574c"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13
PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 8192
PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13
PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 4875
PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13
PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 4875
selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/
hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element {"using": "css selector
", "sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "value": ".apps-list-con
tainer"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h
ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element/{b0806653-ccfe-4755-bc37
-bf95d7b59c96}/location {"sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "i
d": "{b0806653-ccfe-4755-bc37-bf95d7b59c96}"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h
ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element/{b0806653-ccfe-4755-bc37
-bf95d7b59c96}/size {"sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "id":
"{b0806653-ccfe-4755-bc37-bf95d7b59c96}"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h
ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/screenshot {"sessionId": "7ca397
83-4f52-4288-8afe-44a99c6b574c"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13
PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 8192
PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13
PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 136
PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13
PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 136
selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/
hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element {"using": "css selector
", "sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "value": ".menu-right-co
ntainer"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h
ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element/{c73b4fa6-fe5d-4dc7-a214
-5e45bb0eb5b8}/location {"sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "i
d": "{c73b4fa6-fe5d-4dc7-a214-5e45bb0eb5b8}"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h
ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element/{c73b4fa6-fe5d-4dc7-a214
-5e45bb0eb5b8}/size {"sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "id":
"{c73b4fa6-fe5d-4dc7-a214-5e45bb0eb5b8}"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h
ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/screenshot {"sessionId": "7ca397
83-4f52-4288-8afe-44a99c6b574c"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13
PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 8192
PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13
PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 2523
PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13
PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 2523
selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/
hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element {"using": "css selector
", "sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "value": ".notifications
"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h
ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element/{5d8563ea-3067-4dc7-8caf
-080f55194a5e}/location {"sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "i
d": "{5d8563ea-3067-4dc7-8caf-080f55194a5e}"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h
ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element/{5d8563ea-3067-4dc7-8caf
-080f55194a5e}/size {"sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "id":
"{5d8563ea-3067-4dc7-8caf-080f55194a5e}"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h
ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/screenshot {"sessionId": "7ca397
83-4f52-4288-8afe-44a99c6b574c"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13
PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 8192
PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13
PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 8948
PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13
PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 7340
--------------------- >> end captured logging << ---------------------

Ran 1 test in 19.994s

FAILED (failures=1)

C:\src\automatictestingtool\NeedleVisual>

why?

thanks
@andreabisello andreabisello changed the title single test - multiple assertScreenshot - if one fails, the diff image show one of the not failed assertion [firefox] - single test - multiple assertScreenshot - if one fails, the diff image show one of the not failed assertion Jun 1, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@andreabisello